JP7439410B2 - Image processing device, image processing method and program - Google Patents

Description

The present technology relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art Conventionally, an image processing apparatus is known that uses a visual sensor to detect the position and orientation of workpieces, which are objects stacked in bulk, and controls the picking operation of the workpieces by a robot based on the detection results.

Japanese Unexamined Patent Publication No. 2018-146347 (Patent Document 1) describes an image in which a three-dimensional search process using a workpiece model is executed on three-dimensional measurement data obtained by three-dimensional measurement to specify the position and orientation of the workpiece. A processing device is disclosed. The image processing device disclosed in Patent Document 1 superimposes and displays feature points of a workpiece model as a result of three-dimensional search processing on a height image showing a state in which the workpieces are stacked in bulk.

Japanese Patent Application Publication No. 2018-146347

However, it is not possible to sufficiently grasp the detection accuracy of the position and orientation of the workpiece only from a screen in which the feature points of the workpiece model are displayed superimposed on the height image.

The present invention has been made in view of the above problems, and an object thereof is to provide an image processing device, an image processing method, and a program that allow a user to easily confirm the detection accuracy of the position and orientation of an object.

According to an example of the present disclosure, an image processing device includes a generation section, a detection section, and an analysis section. The generation unit generates three-dimensional point group data of the viewing area based on an image obtained from an imaging device installed such that the object is included in the viewing area. The detection unit detects the position and orientation of the object based on the three-dimensional point group data and a model representing the three-dimensional shape of the object. For each of the one or more axes, the analysis unit calculates the first intersection of the model placed in the position and orientation detected by the detection unit with a plane orthogonal to the axis, and the plane indicated by the three-dimensional point cloud data. The second intersection with the plane is displayed in a superimposed manner on the display device.

According to this disclosure, the user can confirm the shift between the first intersection and the second intersection on the virtual plane orthogonal to each axis. The deviation depends on the accuracy with which the detection unit detects the position and orientation of the workpiece. Therefore, the user can easily check the detection accuracy of the position and orientation of the workpiece by the detection unit.

In the above disclosure, for each of the one or more axes, at least one of the direction of the axis and the position of a plane on the axis is variable.

According to this disclosure, the user can move the virtual plane to a desired position and check the first intersection and second intersection on the virtual plane.

In the above disclosure, the one or more axes include three mutually orthogonal axes. According to this disclosure, the user can confirm the first intersecting portion and the second intersecting portion in each of three virtual planes that are orthogonal to three axes that are orthogonal to each other. As a result, the user can confirm the accuracy with which the detection unit detects the position and orientation of the workpiece in various directions.

In the above disclosure, the image processing device includes a setting unit that sets an offset amount for the detected position and orientation, and a robot that picks a target object according to the position and orientation that is moved by the offset amount from the detected position and orientation. The apparatus further includes a determining section that determines the operation.

According to this disclosure, the user can appropriately adjust the operation of the robot by specifying the offset amount. Note that the offset amount includes at least one of a translational component and a rotational component.

In the above disclosure, the setting unit sets the offset amount according to a movement operation on the model displayed on the display device.

According to this disclosure, the user can easily specify the offset amount by performing a movement operation on the workpiece model.

In the above disclosure, the analysis unit moves the model displayed on the display device by an offset amount.

According to this disclosure, the user can visually recognize the magnitude of the offset amount by checking the amount of movement of the workpiece model displayed on the display device.

According to an example of the present disclosure, an image processing method includes first to third steps. The first step is to generate three-dimensional point group data of the visual field based on an image obtained from an imaging device installed such that the object is included in the visual field. The second step is a step of detecting the position and orientation of the object based on three-dimensional point group data and a model representing the three-dimensional shape of the object. The third step is to identify, for each of the one or more axes, the first intersection with a plane perpendicular to the axis in the model placed in the detected position and orientation, and the plane in the plane indicated by the three-dimensional point cloud data. This is a step of superimposing and displaying the second intersection with the second intersection on the display device.

According to an example of the present disclosure, a program causes a computer to perform the above method. These disclosures also make it easier for the user to confirm the detection accuracy of the position and orientation of the target object.

According to the present disclosure, a user can easily check the detection accuracy of the position and orientation of a target object.

1 is a schematic diagram showing the overall configuration of a control system according to an embodiment. FIG. 3 is a diagram showing the correspondence of coordinate systems of each member constituting the control system. FIG. 3 is a diagram showing the positional relationship between a workpiece model and an end effector model that grips the workpiece model. FIG. 3 is a diagram showing an example of a measurement surface indicated by three-dimensional point group data and a workpiece model placed at the position and orientation of the workpiece detected from the three-dimensional point group data. FIG. 7 is a diagram showing another example of a measurement surface indicated by three-dimensional point group data and a workpiece model placed at the position and orientation of the workpiece detected from the three-dimensional point group data. FIG. 6 is a diagram showing an example of an analysis screen for checking the detection result of the position and orientation of the workpiece. 2 is a schematic diagram showing an example of the hardware configuration of the image processing device shown in FIG. 1. FIG. 2 is a block diagram showing an example of a functional configuration of the image processing device shown in FIG. 1. FIG. 9 is a diagram showing an example of an analysis screen displayed by the analysis section shown in FIG. 8. FIG. FIG. 7 is a diagram illustrating an example of a screen when accepting input of a rotational component of an offset amount by a drag operation. FIG. 3 is a diagram showing a first setting example of an offset amount. FIG. 7 is a diagram illustrating an example of translational deviation between a workpiece and a workpiece model placed in a detected position/orientation. FIG. 6 is a diagram illustrating an example of rotational deviation between a workpiece model and a workpiece placed in a detected position/orientation. FIG. 7 is a diagram showing a second setting example of an offset amount. 3 is a flowchart showing the flow of processing of the image processing device in analysis mode. 3 is a flowchart showing the flow of processing of the image processing apparatus in normal mode.

Embodiments of the present invention will be described in detail with reference to the drawings. Note that the same or corresponding parts in the figures are given the same reference numerals and the description thereof will not be repeated.

§1 Application Example First, an example of a situation where the present invention is applied will be described with reference to FIGS. 1 to 6. FIG. 1 is a schematic diagram showing the overall configuration of a control system according to an embodiment. The control system SYS illustrated in FIG. 1 is installed in a production line, etc., and controls picking of objects (hereinafter referred to as "work 1") stored in a container 2. Works 1 are stacked in bulk in a container 2. Therefore, the position and orientation of the workpiece 1 within the container 2 is random.

The control system SYS illustrated in FIG. 1 includes an image processing device 100, a robot controller 200, a robot 300, and a measurement head 400. A display device 150 and an input device 160 are connected to the image processing device 100. Input device 160 includes, for example, a keyboard and a mouse.

The robot 300 picks the work 1. The robot 300 includes an end effector 30 that holds the workpiece 1, a multi-joint arm 31 for changing the position and orientation of the end effector 30, and a base 32 that supports the multi-joint arm 31. The operation of robot 300 is controlled by robot controller 200.

The end effector 30 illustrated in FIG. 1 has two claws 33, and grips the workpiece 1 using the two claws 33. Note that the claws 33 are portions that come into contact with the workpiece 1 when gripping the workpiece 1, and are also referred to as "fingers." The method of holding the workpiece 1 is not limited to gripping using two claws. For example, the end effector 30 may grip the workpiece using three or more claws, or may suction the workpiece 1 using a suction pad.

The robot controller 200 receives operation commands from the image processing device 100 and controls the multi-joint arm 31 of the robot 300. Specifically, the robot controller 200 controls the multi-joint arm 31 so that the end effector 30 performs a picking operation on the workpiece 1 .

The measurement head 400 is installed so that the container 2 and the workpiece 1 housed in the container 2 serve as objects, and images the objects. The measurement head 400 includes a projection section 401 and an imaging section 402. The projection unit 401 projects arbitrary projection pattern light onto a subject according to instructions from the image processing device 100. The projection pattern is, for example, a pattern in which the brightness changes periodically along a predetermined direction within the irradiation surface. The imaging unit 402 images the subject onto which the projection pattern light is projected.

The projection unit 401 includes, as main components, a light source such as an LED (Light Emitting Diode) or a halogen lamp, and a filter disposed on the irradiation surface side of the projection unit 401. The filter generates projection pattern light necessary for measuring a three-dimensional shape as described later, and can arbitrarily change the in-plane light transmittance according to a command from the image processing device 100. The projection unit 401 may have a configuration using a liquid crystal or a DMD (Digital Mirror Device) and a light source (such as an LED or a laser light source).

The imaging unit 402 includes, as main components, an optical system such as a lens, and an imaging element such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

The image processing device 100 detects the position and orientation of the workpiece 1 accommodated in the container 2 based on the image received from the measurement head 400. The image processing device 100 determines the picking operation of the end effector 30 based on the detected position and orientation of the workpiece 1 . The image processing device 100 generates a motion command corresponding to the determined picking motion, and outputs the generated motion command to the robot controller 200. The robot controller 200 causes the robot 300 to perform the picking operation determined by the image processing device 100 in accordance with the operation command.

Specifically, the image processing device 100 determines the position and orientation that the end effector 30 should take when gripping the workpiece 1 (hereinafter also referred to as "grasping point") according to the detected position and orientation of the workpiece 1. decide. The image processing device 100 moves from a standby position to a gripping point, grips a workpiece 1, and then determines a picking operation to move to a target position (a place position for the workpiece 1).

A method for determining a picking operation by the image processing apparatus 100 will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing the correspondence of the coordinate systems of each member constituting the control system. FIG. 3 is a diagram showing the positional relationship between a workpiece model and an end effector model that grips the workpiece model.

In FIG. 2, "base coordinate system B" is a coordinate system based on the base 32 of the robot 300. The "tool coordinate system T" is a coordinate system set on the tip surface of the multi-joint arm 31 of the robot 300 (the flange surface to which the end effector 30 is attached). The position and orientation of the flange surface defines the position and orientation of the end effector 30. “Camera coordinate system C” is a coordinate system based on the imaging unit 402 of the measurement head 400. The “work coordinate system W” is a coordinate system based on the work 1.

The position and orientation of the workpiece 1 detected based on the image received from the measurement head 400 is indicated by, for example, a coordinate transformation matrix ^C H _W that transforms the camera coordinate system C into the workpiece coordinate system W. The coordinate transformation matrix ^C H _W represents the component of the base vector of the work coordinate system W in the camera coordinate system C and the position of the origin.

On the other hand, an operation command output from the image processing device 100 to the robot controller 200 is represented by a coordinate transformation matrix ^B H _T that transforms the base coordinate system B into the tool coordinate system T, for example. The coordinate transformation matrix ^B H _T represents the components of the base vector of the tool coordinate system T in the base coordinate system B and the position of the origin.

The coordinate transformation matrices ^C H _W and ^B H _T satisfy the following equation (1).
^B H _C・^C H _W = ^B H _T・^T H _W ...Formula (1)
In equation (1), the coordinate transformation matrix ^B H _C is a matrix that transforms the base coordinate system B into the camera coordinate system C. The coordinate transformation matrix ^B H _C represents the components of the base vector of the camera coordinate system C in the base coordinate system B and the position of the origin. The coordinate transformation matrix ^T H _W is a matrix that transforms the tool coordinate system T into the workpiece coordinate system W. The coordinate transformation matrix ^T H _W represents the component of the basis vector of the workpiece coordinate system W in the tool coordinate system T and the position of the origin.

In this embodiment, the measurement head 400 is fixed at a position where the entire container 2 can be imaged. Therefore, the coordinate transformation matrix ^B H _C in Equation (1) is expressed as a fixed value and is determined by calibration performed in advance. As the calibration, for example, hand-eye calibration may be employed in which a marker is attached to the flange surface of the robot 300 and an image of the marker is captured by the imaging unit 402 of the measurement head 400 while the robot 300 is operated.

The position and orientation (gripping point) of the end effector 30 when gripping the workpiece 1 is determined by a model representing the three-dimensional shape of the workpiece 1 (hereinafter referred to as "workpiece model 1M") and a three-dimensional shape of the end effector 30. model (hereinafter referred to as "end effector model 30M").

As shown in FIG. 3, the user creates a workpiece model 1M and an end effector model in virtual space so that the relative position and orientation of the end effector 30 with respect to the workpiece 1 when the end effector 30 grips the workpiece 1 match. Place 30M. In this way, a coordinate transformation matrix ^TM H _WM indicating the relative position and orientation of the workpiece model 1M with respect to the end effector model 30M arranged in the virtual space is calculated. The calculated coordinate transformation matrix ^TM H _WM is registered in advance in the image processing device 100 as grasping point data.

The coordinate transformation matrix ^TM H _WM represents the component of the base vector and the position of the origin of the coordinate system WM of the workpiece model 1M in the coordinate system TM based on the surface on the multi-joint arm side of the end effector model 30M. The base vector and origin of the coordinate system TM are set to match the base vector and origin of the tool coordinate system T (see FIG. 2), respectively, when the end effector model 30M is attached to the flange surface of the multi-joint arm 31. Set. The base vector and origin of the workpiece model 1M are set so that, when the workpiece model 1M is superimposed on the workpiece 1, they coincide with the base vector and origin of the workpiece coordinate system W (see FIG. 2), respectively.

The image processing device 100 inputs grip point data (coordinate transformation matrix ^TM H _WM ) registered ⁱⁿ advance into the coordinate transformation matrix TH _W of Equation (1). Thereby, the image processing device 100 generates a coordinate transformation matrix ^B Calculate _T.
^B H _T = ( ^TM H _WM ) ^-1・^B H _C・^C H _W ...Formula (2)
The image processing device 100 may generate _an operation command to the robot controller 200 according to the calculated coordinate transformation matrix ^BHT .

Detection of the position and orientation of the workpiece 1 by the image processing device 100 is performed using a three-dimensional search. In the 3D search, feature points that match or approximate the workpiece model representing the 3D shape of the workpiece 1 are extracted from the 3D point cloud data generated based on the image received from the measurement head 400. This is a process of detecting the position and orientation of the workpiece 1 based on feature points. The three-dimensional point group data indicates an object surface (hereinafter referred to as a "measurement surface") existing within the visual field that is measured from an image received from the measurement head 400. The three-dimensional search is performed using a known detection algorithm (also referred to as a "recognition algorithm").

FIG. 4 is a diagram showing an example of a measurement surface indicated by three-dimensional point cloud data and a workpiece model placed at the position and orientation of the workpiece detected from the three-dimensional point cloud data. FIG. 5 is a diagram showing another example of a measurement surface indicated by three-dimensional point cloud data and a workpiece model placed at the position and orientation of the workpiece detected from the three-dimensional point cloud data.

As shown in FIGS. 4 and 5, due to a quirk of the detection algorithm, the workpiece model 71 placed in the detected position and orientation is always translated or It can rotate. In the example shown in FIG. 4, the workpiece model 71 is offset from the measurement surface 70 in the +Z direction. In the example shown in FIG. 5, the workpiece model 71 is displaced from the measurement surface 70 in the rotational direction around the Y-axis. However, the directions of the translational shift and rotational shift of the workpiece model 71 with respect to the measurement surface 70 depend on the peculiarities of the detection algorithm, and are not limited to the directions illustrated in FIGS. 4 and 5.

Conventionally, a user has not been able to sufficiently confirm detection deviations as shown in FIGS. 4 and 5. Therefore, the image processing apparatus 100 according to the present embodiment presents the user with an analysis screen for checking the detection accuracy of the position and orientation of the workpiece 1.

FIG. 6 is a diagram showing an example of an analysis screen for checking the detection result of the position and orientation of the workpiece. The analysis screen 50 illustrated in FIG. 6 is displayed on the display device 150 (see FIG. 1) by the image processing device 100.

As shown in FIG. 6, the analysis screen 50 includes areas 51, 61-62. In the area 51, a distance image represented by three-dimensional point group data measured using the measurement head 400 is displayed. The distance image is expressed as a color map, and the Z coordinate (height) of each pixel on the XY plane is indicated by color or density. The range of Z coordinates that the distance image can take is indicated by a color bar 52. Note that the X axis, Y axis, and Z axis are orthogonal to each other. The Z-axis is parallel to the optical axis of the imaging unit 402.

In the area 51, an outline 71d of an area obtained by projecting the workpiece model 71 placed in the position and orientation detected by the three-dimensional search in the Z-axis direction is displayed superimposed on the distance image. In the example shown in FIG. 6, the positions and orientations of a plurality of workpieces 1 are detected from three-dimensional point cloud data by three-dimensional search. Therefore, a plurality of outlines 71d are displayed.

Further, in the area 51, a line 53 for specifying the position of a virtual plane perpendicular to the A line 54 for specifying the position of the virtual plane (referred to as "2 virtual plane") is displayed. A line 55 for specifying the position of a virtual plane (hereinafter referred to as "third virtual plane") perpendicular to the Z-axis is displayed on the color bar 52. The positions of the lines 53, 54, and 55 are variable depending on the operation on the input device 160.

In the area 60, an intersection 71a of the work model 71 placed in the detected position and orientation with the first virtual plane and an intersection 70a of the measurement surface 70 with the first virtual plane are displayed in a superimposed manner.

In the area 61, an intersection 71b of the work model 71 placed in the detected position and orientation with the second virtual plane and an intersection 70b of the measurement surface 70 with the second virtual plane are displayed in a superimposed manner.

In the area 62, an intersection 71c of the work model 71 disposed in the detected position and orientation with the third virtual plane and an intersection 70c of the measurement surface 70 with the third virtual plane are displayed in a superimposed manner.

In this way, the image processing device 100 determines, for each of the X, Y, and Z axes, the intersection of the workpiece model 71 placed at the detected position and orientation with a plane perpendicular to the axes, and the corresponding part of the measurement surface 70. The intersection portion with the plane is displayed in a superimposed manner on the display device 150. The user can confirm the displacement of the two intersections on the plane orthogonal to each of the X, Y, and Z axes. The deviation depends on the detection accuracy of the position and orientation of the workpiece by three-dimensional search. Therefore, the user can easily confirm the detection accuracy of the position and orientation of the workpiece by three-dimensional search.

§2 Specific Example Next, a specific example of the control system according to the present embodiment will be described.

<A. Example of hardware configuration of image processing device>
The image processing device 100 is typically a computer having a general-purpose architecture, and executes the image processing according to the present embodiment by executing a program (instruction code) installed in advance. . Such programs are typically distributed in a state stored in various recording media or installed in the image processing apparatus 100 via a network or the like.

When using such a general-purpose computer, in addition to the application for executing the image processing according to this embodiment, an OS (Operating System) for executing the basic processing of the computer is installed. may have been done. In this case, the program according to the present embodiment may call necessary modules at a predetermined timing in a predetermined sequence among the program modules provided as part of the OS to execute processing. good. That is, the program itself according to this embodiment does not include the above-mentioned modules, and the processing may be executed in cooperation with the OS. The program according to this embodiment may be in a form that does not include some of these modules.

Furthermore, the program according to this embodiment may be provided by being incorporated into a part of another program. Even in that case, the program itself does not include modules included in other programs to be combined as described above, and processing is executed in cooperation with the other programs. That is, the program according to this embodiment may be incorporated into such another program. Note that part or all of the functions provided by executing the program may be implemented as a dedicated hardware circuit.

FIG. 7 is a schematic diagram showing an example of the hardware configuration of the image processing apparatus shown in FIG. As shown in FIG. 7, the image processing device 100 includes a CPU (Central Processing Unit) 110 as an arithmetic processing unit, a main memory 112 and a hard disk 114 as storage units, a measurement head interface 116, and an input interface 118. , a display controller 120, a communication interface 124, and a data reader/writer 126. These units are connected to each other via a bus 128 so as to be able to communicate data.

The CPU 110 deploys programs (codes) installed on the hard disk 114 into the main memory 112 and executes them in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory), and holds images acquired by the measurement head 400 in addition to programs read from the hard disk 114. do. Furthermore, the hard disk 114 stores various types of data as will be described later. Note that in addition to or in place of the hard disk 114, a semiconductor storage device such as a flash memory may be employed.

Measurement head interface 116 mediates data transmission between CPU 110 and measurement head 400. That is, the measurement head interface 116 is connected to the measurement head 400. Measurement head interface 116 provides projection commands and imaging commands to measurement head 400 in accordance with internal commands generated by CPU 110. Measurement head interface 116 includes an image buffer 116a for temporarily storing images from measurement head 400. When a predetermined number of images are accumulated in the image buffer 116a, the measurement head interface 116 transfers the accumulated images to the main memory 112.

Input interface 118 mediates data transmission between CPU 110 and input device 160. That is, the input interface 118 receives input information input by the user into the input device 160.

The display controller 120 is connected to the display device 150 and controls the display of the display device 150 so as to notify the user of processing results in the CPU 110 and the like.

Communication interface 124 mediates data transmission between CPU 110 and external devices such as robot controller 200. The communication interface 124 typically includes Ethernet (registered trademark), USB (Universal Serial Bus), or the like.

Data reader/writer 126 mediates data transmission between CPU 110 and memory card 106, which is a recording medium. That is, the memory card 106 stores and distributes programs to be executed by the image processing apparatus 100, and the data reader/writer 126 reads the programs from the memory card 106. Furthermore, the data reader/writer 126 writes the image captured by the measurement head 400 and/or the processing results in the image processing device 100 to the memory card 106 in response to an internal command from the CPU 110 . Note that the memory card 106 may be a general-purpose semiconductor storage device such as an SD (Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory). Consists of etc.

<B. Functional configuration example of image processing device>
FIG. 8 is a block diagram showing an example of the functional configuration of the image processing apparatus shown in FIG. 1. As shown in FIG. 8, the image processing apparatus 100 includes a head control section 10, a storage section 11, an image processing section 12, and a UI (User Interface) processing section 13. The head control unit 10 is realized by a CPU 110 and a measurement head interface 116 shown in FIG. The storage unit 11 is realized by the main memory 112 and hard disk 114 shown in FIG. The image processing section 12 is realized by the CPU 110 and the communication interface 124 shown in FIG. The UI processing unit 13 is realized by the CPU 110, input interface 118, and display controller 120 shown in FIG.

The image processing device 100 has a normal mode and an analysis mode as operating modes. When the normal mode is selected, the head control section 10 and the image processing section 12 operate. When the analysis mode is selected, the head control section 10, image processing section 12, and UI processing section 13 operate. The analysis mode is selected, for example, when starting up a production line or during regular maintenance.

The head control unit 10 controls the operation of the measurement head 400. The head control unit 10 outputs a projection command to the measurement head 400 to instruct projection of a predetermined projection pattern of light. The head control unit 10 outputs an imaging command to the measurement head 400 while the projection pattern light is being projected. The head control unit 10 stores image data 11a acquired from the measurement head 400 in the storage unit 11.

The storage unit 11 stores image data 11a obtained by imaging with the measurement head 400. Further, the storage unit 11 stores in advance CAD (Computer Aided Design) data 11b indicating the three-dimensional shape of the workpiece 1 and gripping point data 11c indicating the position and orientation of the end effector 30 when gripping the workpiece 1. Ru. Furthermore, the storage unit 11 also stores offset data 11d indicating a correction amount (hereinafter referred to as "offset amount") for the position and orientation detected by the three-dimensional search. The offset data 11d is generated by the UI processing unit 13. Details of the offset data 11d will be described later.

The image processing unit 12 processes image data 11a obtained by imaging with the measurement head 400. As shown in FIG. 8, the image processing section 12 includes a 3D image generation section 14, a detection section 15, an operation command determination section 16, and a communication section 17.

The 3D image generation unit 14 performs three-dimensional measurement in the visual field of the measurement head 400 based on the image data 11a, and generates three-dimensional point group data. The three-dimensional point group data indicates the three-dimensional coordinates of each point on the object surface (measurement surface) existing in the field of view of the measurement head 400.

The three-dimensional point group data is, for example, three-dimensional coordinates (x ₁ , y ₁ , z ₁ ), ..., (x _M , y ₁ , z _M ), ( x ₁ , y ₂ , z _M+1 ), ..., (x _M , y ₂ , z _2M ), ..., (x _M , y _N , z _NM ). The points (x ₁ , y ₁ ), . . . , (x _M , y _N ) represent M×N points arranged in a matrix on the XY plane orthogonal to the optical axis of the imaging unit 402. Note that when the imaging unit 402 generates the image data 11a by perspective projection, the point cloud represented by the three-dimensional point cloud data generated based on the image data 11a usually forms a matrix when projected onto the XY plane. are not arranged in the same way. In such a case, the 3D image generation unit 14 appropriately interpolates the coordinate values of the three-dimensional point group data so that it corresponds to the M×N point group arranged in a matrix when projected onto the XY plane. , the point cloud indicated by the three-dimensional point cloud data may be rearranged.

As the three-dimensional measurement process, for example, a triangulation method or a coaxial method may be adopted. The triangular distance measurement method is a method in which the optical axes of imaging and light projection are separated by a base line length, and parallax is converted into distance. Triangulation methods include active methods and passive methods. Active methods include structured illumination methods, phase shift methods, and spatial code methods. The coaxial method is a method in which the optical axes of the imaging means and the distance measuring means are set to be the same. The distance measuring means includes a focusing method. Furthermore, a TOF (Time of Flight) method is also included as a method similar to coaxial. The 3D image generation unit 14 may execute the three-dimensional measurement process using any of these methods. For example, three-dimensional measurement processing is performed using a phase shift method.

The phase shift method is a method that uses an image (sinusoidal projection image) that is captured while projecting a projection pattern of light whose density changes in a sinusoidal manner onto a subject. A plurality of projection patterns having different gradation cycles and phases are prepared, and the projection section 401 of the measurement head 400 sequentially irradiates the viewing area of the imaging section 402 with the plurality of projection pattern lights. The 3D image generation unit 14 reads out from the storage unit 11 a group of image data 11a (a sine wave projection image group) that are respectively captured when a plurality of projection patterns are projected. Then, the 3D image generation unit 14 calculates three-dimensional coordinates based on changes in brightness (brightness or lightness) of corresponding portions between these image data 11a.

The detection unit 15 detects the position and orientation of the workpiece 1 by three-dimensional search. Specifically, the detection unit 15 compares the three-dimensional point group data with a plurality of template data created in advance, and extracts data similar to the template from the three-dimensional point group data. The template data is created in advance based on the CAD data 11b. Typically, a work model indicated by CAD data 11b and a plurality of virtual viewpoints are arranged in a virtual space, and a template is provided that indicates the coordinates of a point on the surface of the work model when viewed from each of the plurality of virtual viewpoints. Data is created. The detection unit 15 detects the position and orientation of the workpiece 1 based on data similar to the template extracted from the three-dimensional point group data. The position and orientation of the workpiece 1 detected by the detection unit 15 is indicated by the coordinate system of the imaging unit 402 of the measurement head 400 (camera coordinate system C (see FIG. 2)). Note that the detection unit 15 may detect the position and orientation of the workpiece 1 using a known detection algorithm.

Note that when detecting the positions and orientations of a plurality of works 1, the detection unit 15 selects one of the plurality of works 1 as a picking target according to a predetermined selection criterion.

The operation command determination unit 16 determines the operation of the end effector 30 according to the position and orientation of the workpiece 1 to be picked. Specifically, the motion command determination unit 16 uses the coordinate transformation matrix ^C H _W indicating the position and orientation detected by the detection unit 15, the coordinate transformation matrix ^B H _C calculated by prior calibration, and the grip point data. By substituting the coordinate transformation matrix ^TM H _WM shown by 11c into the above equation (2), the coordinate transformation matrix ^B H _T is calculated. The image processing device 100 generates _an operation command to the robot controller 200 according to the calculated coordinate transformation matrix ^BHT .

However, if the offset data 11d is stored in the storage unit 11, the motion command determination unit 16 corrects the coordinate transformation matrix ^C H _W indicating the position and orientation detected by the detection unit 15 by the offset amount indicated by the offset data 11d. do. The motion command determination unit 16 substitutes the corrected coordinate transformation matrix ^C H _W into the above equation (2).

The communication unit 17 is capable of communicating with the robot controller 200 and transmits the operation command generated by the operation command determination unit 16 to the robot controller 200.

The UI processing unit 13 executes processing for providing the user with information regarding the operations of the image processing device 100 and the operations of the robot 300. As shown in FIG. 8, the UI processing section 13 includes an analysis section 18 and a setting section 19.

The analysis unit 18 acquires the most recent three-dimensional point group data generated by the 3D image generation unit 14 and the detection result (position and orientation of the work 1) of the detection unit 15 for the three-dimensional point group data. The analysis unit 18 executes analysis processing on the acquired three-dimensional point cloud data and detection results, and displays an analysis screen showing the analysis results on the display device 150. Details of the analysis screen will be described later.

The setting unit 19 sets an offset amount for the position and orientation detected by the three-dimensional search. As will be described later, the analysis screen displayed on the display device 150 by the analysis unit 18 is provided with an input field for setting the offset amount. The setting unit 19 sets the offset amount according to the input to the input field. The setting unit 19 generates offset data 11d indicating the set offset amount, and stores the generated offset data 11d in the storage unit 11.

<C. Analysis screen>
FIG. 9 is a diagram showing an example of an analysis screen displayed on a display device by the analysis section shown in FIG. 8. The analysis screen 50a illustrated in FIG. 9 includes areas 51, 60 to 62, similar to the analysis screen 50 illustrated in FIG. The analysis screen 50a further includes an input field 63 for selecting one workpiece model placed in the detected position/orientation, input fields 64 and 65 for inputting the offset amount, and input fields 64 and 65 for setting the offset amount. OK button 68.

The analysis unit 18 displays a distance image indicated by the three-dimensional point group data acquired from the 3D image generation unit 14 in the area 51. Further, the analysis unit 18 superimposes and displays on the distance image an outline 71d of a region obtained by projecting the workpiece model 71 placed in the position and orientation of the workpiece 1 indicated by the detection result obtained from the detection unit 15 onto the XY plane. Specifically, the analysis unit 18 transforms the coordinates of the CAD data 11b to match the position and orientation indicated by the detection result. The work model 71 is represented by coordinate-transformed CAD data 11b.

The analysis unit 18 determines a first virtual plane perpendicular to the X-axis based on the X coordinate=x0 of the line 53 on the distance image. Specifically, the analysis unit 18 determines the plane indicated by x=x0 as the first virtual plane.

The analysis unit 18 displays the intersection portion 71a of the workpiece model 71 with the first virtual plane in the area 60. As described above, the workpiece model 71 is indicated by the CAD data 11b whose coordinates have been transformed so as to match the position and orientation indicated by the detection results. The analysis unit 18 displays the intersection line between the surface element (triangular mesh, etc.) indicated by the coordinate-converted CAD data 11b and the first virtual plane as an intersection portion 71a.

Furthermore, the analysis unit 18 displays, in the region 60, the intersection portion 70a of the measurement surface 70 with the first virtual plane, which is indicated by the three-dimensional point group data. That is, the analysis unit 18 displays the intersecting portion 70a and the intersecting portion 71a in a superimposed manner.

When the three-dimensional point group data indicates the coordinates of M×N points on the measurement surface (x ₁ , y ₁ , z ₁ ), ..., (x _M , y _N , z _NM ), the line 53 The X coordinate may be set so that it can take only one of x ₁ , . . . , x _M at a predetermined pitch Δx interval. In this case, the analysis unit 18 extracts the coordinates of a point having the same X coordinate (=x0) as the first virtual plane from the three-dimensional point group data, and places the intersection 70a in the area 60 based on the extracted coordinates. Display.

The X coordinate of the line 53 may be set to take not only x ₁ , . . . , x _M but also other values. In this case, the analysis unit 18 extracts N pairs of two points sandwiching the first virtual plane from the three-dimensional point group data. A pair consists of two points whose distance along the X-axis direction is equal to or less than a predetermined pitch Δx. For example, if the X coordinate x0 of the first _virtual plane is x _{k-1 <} x0 < x _k , the N pairs are _the point ₁ ) and a point (x _k , y _i , z _(i-1)M+k ) (i is an integer from 1 to N). For each of the N pairs, the analysis unit 18 performs an interpolation calculation using the coordinates of the two points constituting the pair to find a point on the intersection line between the measurement surface indicated by the three-dimensional point group data and the first virtual plane. Find the coordinates of a point. For example, the analysis unit 18 uses the coordinates (x _k-1 , y ₁ , z _k-1 ) and the coordinates (x _k , y ₁ , z _k ) to locate a point on the intersection line between the measurement surface and the first virtual plane. The coordinates (x0, y ₁ , {(x _k -x0)z _k-1 + (x0 - x _k-1 )z _k }/(x _k -x _k-1 )) of the point are calculated.

Every time the position of the line 53 is changed, the analysis unit 18 updates the intersecting parts 70a and 71a displayed in the area 60 according to the changed position of the line 53. Thereby, by changing the position of the line 53, the user can check the relative positional relationship between the intersecting parts 70a and 71a on the first virtual plane at a desired position.

The analysis unit 18 determines a second virtual plane perpendicular to the Y axis based on the Y coordinate=y0 of the line 54 on the distance image. Specifically, the analysis unit 18 determines the plane indicated by y=y0 as the second virtual plane.

The analysis unit 18 displays the intersection portion 71b of the workpiece model 71 with the second virtual plane in the area 61. The analysis unit 18 displays the intersection line between the second virtual plane and the surface element indicated by the CAD data 11b whose coordinates have been transformed so as to match the position and orientation indicated by the detection result, as an intersection portion 71b.

Furthermore, the analysis unit 18 displays, in the area 61, the intersection portion 70b of the measurement surface 70 with the second virtual plane, which is indicated by the three-dimensional point group data. That is, the analysis unit 18 displays the intersecting portion 70b and the intersecting portion 71b in a superimposed manner.

When the three-dimensional point group data indicates the coordinates (x ₁ , y ₁ , z ₁ ), ..., (x _M , y _N , z _NM ) of M×N points on the measurement surface, the line 54 The Y coordinate may be set so that it can take only one of y ₁ , . . . , y _N at a predetermined pitch Δy interval. In this case, the analysis unit 18 extracts the coordinates of a point having the same Y coordinate (=y0) as the second virtual plane from the three-dimensional point group data, and places the intersection 70b in the area 61 based on the extracted coordinates. Display.

The Y coordinate of the line 54 may be set to take not only y ₁ , . . . , y _N but also other values. In this case, the analysis unit 18 extracts M pairs of two points sandwiching the second virtual plane from the three-dimensional point group data. A pair consists of two points whose distance along the Y-axis direction is equal to or less than a predetermined pitch Δy. For example, if the Y coordinate y0 of the second virtual plane is y _k-1 < y0 < y _k , the M pairs are the points (x _j , y _k-1 , z _(k-1)M+j ) and a point (x _j , y _k , z _kM+j ) (j is an integer from 1 to M). For each of the M pairs, the analysis unit 18 performs an interpolation calculation using the coordinates of the two points constituting the pair to find a point on the intersection line between the measurement surface indicated by the three-dimensional point group data and the first virtual plane. Find the coordinates of a point. For example, the analysis unit 18 uses the coordinates (x ₁ , y _k-1 , z _(k-1)M+1 ) and the coordinates (x ₁ , y _k , z _kM+1 ) to 2. Coordinates _{of a} point on the line _of intersection _with the virtual _{plane k} −y _k-1 )).

Every time the position of the line 54 is changed, the analysis unit 18 updates the intersecting parts 70b and 71b displayed in the area 61 according to the changed position of the line 54. Thereby, by changing the position of the line 54, the user can check the relative positional relationship between the intersecting parts 70b and 71b on the second virtual plane at a desired position.

The analysis unit 18 determines a third virtual plane perpendicular to the Z axis based on the Z coordinate=z0 of the line 55 on the color bar 52. Specifically, the analysis unit 18 determines the plane indicated by z=z0 as the third virtual plane.

The analysis unit 18 displays the intersection portion 71c of the workpiece model 71 with the third virtual plane in the area 62. The analysis unit 18 displays the intersection line between the surface element indicated by the CAD data 11b whose coordinates have been transformed so as to match the position and orientation indicated by the detection result and the third virtual plane as an intersection portion 71c.

Furthermore, the analysis unit 18 displays, in the area 62, an intersection portion 70c of the measurement surface 70 with the third virtual plane, which is indicated by the three-dimensional point group data. That is, the analysis unit 18 displays the intersecting portion 70c and the intersecting portion 71c in a superimposed manner.

When the three-dimensional point group data indicates the coordinates (x ₁ , y ₁ , z ₁ ), ..., (x _M , y _N , z _NM ) of M×N points on the measurement surface, the analysis unit 18 extracts a pair of two points sandwiching the third virtual plane from the three-dimensional point group data. A pair consists of two points whose distance along the X-axis direction is equal to or less than a predetermined pitch Δx, or two points whose distance along the Y-axis direction is equal to or less than a predetermined pitch Δy. For each pair, the analysis unit 18 calculates the coordinates of a point on the intersection line between the measurement surface and the third virtual plane, which are indicated by the three-dimensional point group data, by performing an interpolation calculation using the coordinates of the two points constituting the pair. demand. For example, the analysis unit 18 calculates the coordinates (x ₁ , y ₁ , z ₁ ), (x ₂ , y ₁ , z ₂ ) of two points constituting one pair (where z ₁ and z ₂ are ₁ < z 0 < z ₂ or z ₂ < z 0 < z ₁ ), and calculate the coordinates of the point on the intersection line of the measurement surface and the third virtual plane ({(|z ₂ - z0|) x ₁ + ( |z0−z ₁ |)x ₂ }/(|z ₂ −z ₁ |), y ₁ , z0) is calculated. Similarly, the analysis unit 18 calculates the coordinates (x ₁ , y ₁ , z ₁ ), (x ₁ , y ₂ , z _M+1 ) of two points constituting one pair (where z ₁ and z _{M +1} is _the coordinate _{( x 1} , _{ (|z _M+1 −z0|)y ₁ +(|z0−z ₁ |)y ₂ }/(|z _M+1 −z ₁ |), z0) is calculated.

Every time the position of the line 55 is changed, the analysis unit 18 updates the intersecting portions 70c and 71c displayed in the area 62 according to the changed position of the line 55. Thereby, by changing the position of the line 55, the user can confirm the relative positional relationship between the intersecting portions 70b and 71b on the third virtual plane at a desired position.

The analysis unit 18 assigns a unique identification number to each of the positions and orientations of the one or more workpieces 1 indicated by the detection results obtained from the detection unit 15.

In the input field 63, an identification number corresponding to one work model 71 used for setting the offset amount is input. The analysis unit 18 highlights the outline 71d and intersecting portions 71a to 71c corresponding to the workpiece model 71 placed in the position and orientation of the identification number input in the input field 63 in the areas 51 and 60 to 62, respectively. In the analysis screen 50a illustrated in FIG. 9, a contour 71d_s and intersections 71a_s to 71c_s corresponding to the workpiece model 71 placed in the position and orientation of the identification number input in the input field 63 are displayed as thick lines.

In the input field 64, translational components of the offset amount (translational components of each of the X-axis, Y-axis, and Z-axis) are input. When the translational component of the offset amount is input into the input field 64, the analysis unit 18 converts the intersection portions 71a_s to 71c_s of the workpiece model 71_s placed in the position and orientation of the identification number input into the input field 63 by the translational component. move it. This allows the user to visually recognize the magnitude of the offset amount.

In the input field 65, rotational components (translational components of each of the X-axis, Y-axis, and Z-axis) of the offset amount are input. The input field 65 includes a radio button group 66 for setting the rotation center. The radio button group 66 includes a radio button for setting the click position to the center of rotation, a radio button for setting the center of gravity of the work model 71 to the center of rotation, and a radio button for setting the center of gravity of the work model 71 to the center of rotation, and a radio button for setting the center of gravity of the work model 71 to the center of rotation, and a radio button for setting the click position to the center of rotation, and a radio button for setting the center of gravity of the work model 71 to the center of rotation, and a radio button for setting the center of gravity of the work model 71 to the center of rotation. Includes a radio button for setting the origin as the center of rotation.

Furthermore, the input field 65 includes an input field 67 for inputting a rotation angle. When a rotation angle is input in the input field 67, the analysis unit 18 moves the intersecting portions 71a_s to 71c_s of the workpiece model 71_s placed in the position and orientation of the identification number input in the input field 63 by the rotation angle. This allows the user to visually recognize the magnitude of the offset amount. At this time, the analysis unit 18 sets the position corresponding to the checked radio button in the radio button group 66 as the center of rotation.

When the OK button 68 is operated, the setting unit 19 sets the offset amount according to the inputs to the input fields 64 and 65. The setting unit 19 generates offset data 11d indicating the set offset amount. The offset data 11d is represented by a coordinate transformation matrix. The analysis screen 50a shown in FIG. 9 includes both input fields 64 and 65. Thereby, the setting unit 19 can set both the rotational component and the translational component of the offset amount. However, one of the input fields 64 and 65 may be omitted. In this case, the setting unit 19 only needs to set one of the rotational component and the translational component of the offset amount.

The setting unit 19 may accept a movement operation (drag operation) for the intersection portions 71a_s to 71c_s of the work model 71_s placed in the position and orientation of the identification number input in the input field 63. The setting unit 19 may change the numerical values in the input fields 64 and 65 according to the drag operation. That is, the setting unit 19 may set the offset amount according to the movement operation for the intersection portions 71a_s to 71c_s of the workpiece model 71_s.

For example, the setting unit 19 may receive an input of a translational component of the offset amount in at least one of the Y-axis direction and the Z-axis direction by a translational movement operation on the intersection portion 71a_s displayed in the area 60. The setting unit 19 may receive an input of a translational component of the offset amount in at least one of the X-axis direction and the Z-axis direction by the translational movement with respect to the intersection portion 71b_s displayed in the area 61. The setting unit 19 may receive an input of a translational component of the offset amount in at least one of the X-axis direction and the Y-axis direction by the translational movement with respect to the intersection portion 71c_s displayed in the area 62.

Further, by performing a rotational movement operation on the intersection portion 71a_s displayed in the area 60, the setting unit 19 may receive an input of a rotational component of an offset amount about an axis parallel to the X-axis. By performing a rotational movement operation on the intersection portion 71b_s displayed in the area 61, the setting unit 19 may receive an input of a rotational component of an offset amount about an axis parallel to the Y-axis. By performing a rotational movement operation on the intersection portion 71c_s displayed in the area 62, the setting unit 19 may receive an input of a rotational component of an offset amount about an axis parallel to the Z-axis.

FIG. 10 is a diagram showing an example of a screen when receiving input of a rotational component of an offset amount by a drag operation. FIG. 10 shows an example of a screen when accepting input of a rotational component of an offset amount about an axis parallel to the Y-axis. As shown in FIG. 10, the setting unit 19 displays a mark 72 indicating the center of rotation. The position of the mark 72 corresponds to the checked radio button in the radio button group 66. The setting unit 19 receives an input of a rotation angle about an axis parallel to the Y-axis in response to a drag operation (drag operation in the direction of the arrow 73) on the intersection portion 71b_s.

<D. First setting example of offset amount>
As described above using FIGS. 4 and 5, due to a quirk of the detection algorithm, the workpiece model 71 placed at the detected position and orientation is always oriented in a fixed direction with respect to the measurement surface 70 indicated by the three-dimensional point cloud data. Can be translated or rotated. By checking the analysis screen 50a illustrated in FIG. 8, the user can check the deviation between the workpiece model 71 placed in the detected position and orientation and the measurement surface 70. In response to confirmation of such a shift, the user can take action, for example, by reviewing the detection algorithm. However, detection algorithms are generally complex, and users cannot easily take such measures. Therefore, the user only needs to input an offset amount to eliminate such a shift.

FIG. 11 is a diagram showing a first setting example of the offset amount. In FIG. 11, (a) shows the state before setting the offset amount, and (b) shows the state after setting the offset amount. In FIG. 11, the workpiece model 71 and the measurement surface 70 arranged in the detected position and orientation are shown in the lower part, and the workpiece 1 and the end effector 30 when gripping the workpiece 1 are shown in the upper part.

FIG. 11 shows an example in which a workpiece model 71 placed in a detected position and orientation is translated in the −Z direction with respect to the measurement surface 70 due to a quirk of the detection algorithm. In this case, as shown in the upper part of (a), the gripping position of the end effector 30 with respect to the workpiece 1 is lower than the relative position of the end effector model 30M with respect to the workpiece model 1M (see FIG. 3), which is indicated by the gripping point data 11c. -Move in the Z direction. As a result, the possibility that the claw 33 collides with another workpiece 1 increases, for example.

Therefore, the user inputs an offset amount so that the deviation between the workpiece model 71 and the measurement surface 70 becomes smaller. Specifically, the user inputs a translational component in the +Z direction. As a result, as shown in the upper part of (b), the gripping position of the end effector 30 with respect to the workpiece 1 approaches the relative position of the end effector model 30M with respect to the workpiece model 1M (see FIG. 3), which is indicated by the gripping point data 11c. . As a result, for example, the possibility that the claw 33 will collide with another workpiece 1 can be reduced.

<E. Second setting example of offset amount>
As described above, the motion command determination unit 16 generates the motion command using the above equation (2). In equation (2), the coordinate transformation matrix ^B H _C includes a component due to a calibration error. The coordinate transformation matrix ^C H _W contains errors caused by errors between the actual dimensions of the work 1 and the dimensions of the work model indicated by the CAD data 11b (including manufacturing tolerances, errors due to aging of the manufacturing equipment for the work 1, etc.). Contains ingredients. Furthermore, even if the motion command is generated according to the coordinate transformation matrix ^B H _T calculated using the above equation (2), the motion command and the robot 300 may differ due to aging of the robot 300 and the influence of the temperature around the robot 300. Errors may occur in actual operation. Due to these causes, even if the deviation between the workpiece model 71 and the measurement surface 70 is small, the gripping position of the end effector 30 with respect to the workpiece 1 is the same as that of the end effector model 30M with respect to the workpiece model 1M indicated by the gripping point data 11c. The relative position (see FIG. 3) may not match.

FIG. 12 is a diagram illustrating an example of a translational shift between a workpiece model and a workpiece placed in the detected position and orientation. In the example shown in FIG. 12, the actual position of the workpiece 1 is shifted from the position of the workpiece model 71 in the −Z direction. Therefore, the workpiece 1 may not be gripped by the end effector 30.

FIG. 13 is a diagram illustrating an example of rotational deviation between a workpiece model and a workpiece placed in the detected position and orientation. When the workpiece 1 has an elongated shape, even if the angle between the longitudinal direction of the workpiece 1 and the XY plane is slightly different, the shape of the workpiece 1 when viewed from the imaging section 402 of the measurement head 400 will change. small. Therefore, as shown in FIG. 13, the actual angle between the longitudinal direction of the workpiece 1 and the XY plane may deviate from the angle between the longitudinal direction of the workpiece model 71 and the XY plane. Therefore, the workpiece 1 may not be gripped by the end effector 30.

Even in such a case, the user can input the offset amount on the analysis screen 50a shown in FIG. The user only has to check the state in which the end effector 30 grips the workpiece 1 and input the offset amount according to the state.

FIG. 14 is a diagram showing a second setting example of the offset amount. Similar to FIG. 11, (a) shows the state before setting the offset amount, and (b) shows the state after setting the offset amount. In the lower part, the workpiece model 71 and the measurement surface 70 arranged in the detected position and orientation are shown, and in the upper part, the relative positional relationship between the workpiece 1 and the end effector 30 is shown.

FIG. 14 shows an example in which the deviation between the workpiece model 71 placed in the detected position and orientation and the measurement surface 70 is small. However, as shown in the upper part of FIG. (see) in the +Z direction. Therefore, there is a possibility that the workpiece 1 cannot be gripped.

In such a case, the user may input an offset amount so that the end effector 30 can grip the workpiece 1. Specifically, the user inputs a translation component in the −Z direction. At this time, the amount of deviation between the workpiece model 71 and the measurement surface 70 corresponds to the amount of offset. Therefore, the user can input an appropriate offset amount while referring to the measurement surface 70 displayed on the analysis screen 50a. By setting an appropriate offset amount, as shown in the upper part of (b), the gripping position of the end effector 30 with respect to the workpiece 1 can be adjusted relative to the end effector model 30M with respect to the workpiece model 1M indicated by the gripping point data 11c. position (see Figure 3). As a result, the end effector 30 can accurately grip the workpiece 1.

<F. Processing flow of the image processing device in analysis mode>
FIG. 15 is a flowchart showing the process flow of the image processing apparatus in the analysis mode. First, the CPU 110 generates three-dimensional point group data indicating the object surface (measurement surface) in the visual field based on the image data 11a acquired from the measurement head 400 (step S1). The CPU 110 detects the position and orientation of the work 1 based on the three-dimensional point group data and a model indicating the three-dimensional shape of the work 1 (work model indicated by the CAD data 11b) (step S2).

For each of the X-axis, Y-axis, and Z-axis, the CPU 110 calculates the intersection of the workpiece model 71 placed at the detected position and orientation with a plane orthogonal to the axis, and the measurement surface indicated by the three-dimensional point group data. The intersection portion with the plane is displayed in a superimposed manner on the display device 150 (step S3). Specifically, an analysis screen 50a as shown in FIG. 8 is displayed on the display device 150. The CPU 110 sets an offset amount for the detected position and orientation according to the input to the analysis screen 50a (step S4). After step S4, the image processing device 100 ends the analysis mode processing.

<G. Processing flow of the image processing device in normal mode>
FIG. 16 is a flowchart showing the flow of processing of the image processing apparatus in normal mode. First, the CPU 110 generates three-dimensional point group data indicating the object surface (measurement surface) in the visual field based on the image data 11a acquired from the measurement head 400 (step S11). The CPU 110 detects the position and orientation of the work 1 based on the three-dimensional point group data and a model indicating the three-dimensional shape of the work 1 (work model indicated by the CAD data 11b) (step S12).

The CPU 110 corrects the detected position and orientation of the workpiece 1 by the offset amount set in the analysis mode (step S13). The CPU 110 determines the operation of the robot 300 for picking the workpiece 1 according to the corrected position and orientation (step S14). The CPU 110 generates a motion command corresponding to the determined motion, and outputs the generated motion command to the robot controller 200 (step S15). After step S15, the image processing apparatus 100 ends the normal mode processing.

<H. Modified example>
In the above description, the areas 60 to 62 of the analysis screens 50 and 50a display the intersections of virtual planes perpendicular to the X, Y, and Z axes, and the workpiece model 71 and the measurement surface 70, respectively. However, the one or more axes that define the virtual plane are not limited to the X, Y, and Z axes, and may be, for example, three mutually perpendicular axes that are inclined at a predetermined angle from the X, Y, and Z axes. Furthermore, the direction of the axis that defines the virtual plane may be variable according to a user's operation. This allows the user to appropriately change the axis in the direction he or she wants to check.

<I. Action/Effect>
As described above, the image processing device 100 according to the present embodiment includes the 3D image generation section 14, the detection section 15, and the analysis section 18. The 3D image generation unit 14 generates three-dimensional point group data of the visual field based on an image obtained from the imaging unit 402 installed so that the work 1 is included in the visual field. The detection unit 15 detects the position and orientation of the workpiece 1 based on three-dimensional point group data and a workpiece model indicating the three-dimensional shape of the workpiece 1. The analysis unit 18 calculates, for each of the one or more axes, the intersection of the work model placed at the position and orientation detected by the detection unit 15 with a virtual plane perpendicular to the axis, and three-dimensional point group data. The intersecting portion of the measurement surface with the virtual plane is displayed in a superimposed manner on the display device 150.

According to the above configuration, the user can check the displacement of two intersections on the virtual plane orthogonal to each axis. The deviation depends on the detection accuracy of the position and orientation of the workpiece by three-dimensional search. Therefore, the user can easily confirm the detection accuracy of the position and orientation of the workpiece by three-dimensional search.

For each of the one or more axes, at least one of the direction of the axis and the position of the virtual plane on the axis is variable. Thereby, the user can move the virtual plane to a desired position and check the intersection of the virtual plane, work model 71, and measurement surface 70.

The one or more axes include three mutually orthogonal axes. This allows the user to simultaneously confirm the intersections of the work model 71 and the measurement surface 70 with three virtual planes that are orthogonal to three axes that are orthogonal to each other.

The image processing device 100 further includes a setting section 19 and an operation command determining section 16. The setting unit 19 sets an offset amount for the detected position and orientation. The motion command determination unit 16 determines the motion of the robot 300 that picks the workpiece 1 according to the position and orientation that has been moved by the offset amount from the detected position and orientation. Thereby, the user can adjust the robot's operation as appropriate by specifying the offset amount. Note that the offset amount includes at least one of a translational component and a rotational component.

The setting unit 19 sets an offset amount according to a movement operation for the workpiece model displayed on the display device 150. Thereby, the user can easily specify the offset amount by performing a movement operation on the workpiece model.

The analysis unit 18 moves the work model displayed on the display device 150 by the offset amount. Thereby, the user can visually recognize the magnitude of the offset amount by checking the amount of movement of the workpiece model displayed on the display device 150.

<J. Additional notes>
As described above, the present embodiment and the modified examples include the following disclosures.

(Configuration 1)
An image processing device (100),
a generation unit (110, 14) that generates three-dimensional point cloud data of the visual field based on an image obtained from an imaging device (400, 402) installed such that the object (1) is included in the visual field; and,
a detection unit (110, 15) that detects the position and orientation of the object (1) based on the three-dimensional point group data and a model indicating the three-dimensional shape of the object (1);
For each of the one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the position and orientation detected by the detection unit (110, 15) and the three-dimensional point group data. An image processing device (100) comprising an analysis unit (110, 18) that causes a display device (150) to display a second intersecting portion of the shown plane with the plane in a superimposed manner.

(Configuration 2)
The image processing device (100) according to configuration 1, wherein for each of the one or more axes, at least one of the direction of the axis and the position of the plane on the axis is variable.

(Configuration 3)
The image processing device (100) according to configuration 1 or 2, wherein the one or more axes include three axes that are orthogonal to each other.

(Configuration 4)
a setting unit (110, 19) that sets an offset amount for the detected position and orientation;
Any one of configurations 1 to 3, further comprising a determining unit (110, 16) that determines an operation of the robot that picks the target object according to a position and orientation that is moved by the offset amount from the detected position and orientation. An image processing device (100) according to claim 1.

(Configuration 5)
The image processing device (100) according to configuration 4, wherein the setting unit (110, 19) sets the offset amount according to a movement operation for the model displayed on the display device (150).

(Configuration 6)
The image processing device (100) according to configuration 4, wherein the analysis unit (110, 19) moves the model displayed on the display device (150) by the offset amount.

(Configuration 7)
The image processing device (100) according to any one of configurations 4 to 6, wherein the offset amount includes at least one of a translational component and a rotational component.

(Configuration 8)
An image processing method, comprising:
generating three-dimensional point cloud data of the visual field based on images obtained from imaging devices (400, 402) installed such that the object (1) is included in the visual field;
detecting the position and orientation of the object (1) based on the three-dimensional point group data and a model indicating the three-dimensional shape of the object;
For each of the one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the detected position and orientation, and a first intersection with the plane in the plane indicated by the three-dimensional point group data. An image processing method comprising the step of superimposing and displaying the two intersections on a display device (150).

(Configuration 9)
A program that causes a computer to execute an image processing method,
The image processing method includes:
generating three-dimensional point cloud data of the visual field based on images obtained from imaging devices (400, 402) installed such that the object (1) is included in the visual field;
detecting the position and orientation of the object (1) based on the three-dimensional point group data and a model representing the three-dimensional shape of the object (1);
For each of the one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the detected position and orientation, and a first intersection with the plane in the plane indicated by the three-dimensional point group data. A program comprising: displaying the two intersections in a superimposed manner on a display device (150).

Although the embodiments of the present invention have been described, the embodiments disclosed this time should be considered to be illustrative in all respects and not restrictive. The scope of the present invention is indicated by the claims, and it is intended that all changes within the meaning and range equivalent to the claims are included.

1 workpiece, 1M, 71 work model, 2 container, 10 head control unit, 11 storage unit, 11a image data, 11b CAD data, 11c grip point data, 11d offset data, 12 image processing unit, 13 UI processing unit, 14 3D Image generation section, 15 Detection section, 16 Operation command determination section, 17 Communication section, 18 Analysis section, 19 Setting section, 30 End effector, 30M End effector model, 31 Multi-joint arm, 32 Base, 33 Claw, 50, 50a Analysis Screen, 51, 60, 61, 62 Area, 52 Color bar, 53, 54, 55 Line, 63, 64, 65, 67 Input field, 66 Radio button group, 68 OK button, 70 Measurement surface, 70a, 70b, 70c , 71a, 71b, 71c intersection, 71d contour, 72 mark, 73 arrow, 100 image processing device, 106 memory card, 110 CPU, 112 main memory, 114 hard disk, 116 measurement head interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader/writer, 128 bus, 150 display device, 160 input device, 200 robot controller, 300 robot, 400 measurement head, 401 projection unit, 402 imaging unit, SYS control system.

Claims (9)

An image processing device,
a generation unit that generates three-dimensional point group data of the visual field based on an image obtained from an imaging device installed such that the target object is included in the visual field;
a detection unit that detects the position and orientation of the object based on the three-dimensional point group data and a model indicating the three-dimensional shape of the object;
For each of one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the position and orientation detected by the detection unit, and an analysis unit that displays a second intersection with the plane in a superimposed manner on a display device ;
An image processing apparatus, comprising : a setting section that sets an offset amount for the detected position and orientation . The image processing device according to claim 1, wherein for each of the one or more axes, at least one of the direction of the axis and the position of the plane on the axis is variable. The image processing device according to claim 1 or 2, wherein the one or more axes include three axes orthogonal to each other. The image processing apparatus according to claim 1, further comprising a determining unit that determines an operation of the robot that picks the target object according to a position and orientation that is moved by the offset amount from the detected position and orientation. The image processing device according to any one of claims 1 to 4 , wherein the setting unit sets the offset amount according to a movement operation on the model displayed on the display device. The image processing device according to claim 1 , wherein the analysis unit moves the model displayed on the display device by the offset amount . The image processing device according to claim 1 , wherein the offset amount includes at least one of a translational component and a rotational component. An image processing method, comprising:
generating three-dimensional point cloud data of the viewing area based on an image obtained from an imaging device installed such that the object is included in the viewing area;
detecting the position and orientation of the object based on the three-dimensional point group data and a model representing the three-dimensional shape of the object;
For each of the one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the detected position and orientation, and a first intersection with the plane in the plane indicated by the three-dimensional point group data. displaying the two intersections in a superimposed manner on a display device ;
An image processing method, comprising : setting an offset amount for the detected position and orientation . A program that causes a computer to execute an image processing method,
The image processing method includes:
generating three-dimensional point cloud data of the viewing area based on an image obtained from an imaging device installed such that the object is included in the viewing area;
detecting the position and orientation of the object based on the three-dimensional point group data and a model representing the three-dimensional shape of the object;
For each of the one or more axes, a first intersection with a plane orthogonal to the axis in the model placed in the detected position and orientation, and a first intersection with the plane in the plane indicated by the three-dimensional point group data. displaying the two intersections in a superimposed manner on a display device ;
A program comprising: setting an offset amount for the detected position and orientation .

Images