JP2021024052A - Image processing device, image processing method and program - Google Patents

Abstract

To provide an image processing device which allows a user to easily confirm the detection accuracy of a position attitude of an object.SOLUTION: An image processing device comprises a generation unit, a detection unit and an analysis unit. The generation unit generates three-dimensional point cloud data of a visual area on the basis of an image obtained from an imaging device installed such that an object is included in the visual area. The detection unit detects the position attitude of the object on the basis of the three-dimensional point cloud data and a model indicating the three-dimensional shape of the object. The analysis unit displays on a display device a first intersection portion with a plane orthogonal to an axis in the model arranged in the position attitude detected by the detection unit for each of one or more axes and a second intersection portion with a plane on a surface indicated by the three-dimensional point cloud data in a superimposed manner.SELECTED DRAWING: Figure 6

Description

The present technology relates to image processing devices, image processing methods and programs.

Conventionally, there is known an image processing device that uses a visual sensor to detect the position and orientation of workpieces that are piled up separately and controls the picking operation of the workpiece by a robot based on the detection result.

Japanese Patent Laying-Open No. 2018-146347 (Patent Document 1) describes an image in which a three-dimensional search process using a work model is executed on three-dimensional measurement data acquired by three-dimensional measurement to specify the position and orientation of the work. The processing apparatus is disclosed. The image processing apparatus disclosed in Patent Document 1 superimposes and displays the feature points of the work model as a result of the three-dimensional search processing on the height image showing the loosely stacked state of the works.

Japanese Unexamined Patent Publication No. 2018-146347

However, the detection accuracy of the position and orientation of the work cannot be sufficiently grasped only by the screen in which the feature points of the work model are superimposed and displayed on the height image.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device, an image processing method, and a program that make it easy for a user to confirm the detection accuracy of the position and orientation of an object.

According to an example of the present disclosure, the image processing apparatus includes a generation unit, a detection unit, and an analysis unit. The generation unit generates three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup device installed so that the object is included in the visual field region. The detection unit detects the position and orientation of the object based on the three-dimensional point cloud data and the model showing the three-dimensional shape of the object. For each of the one or more axes, the analysis unit is located at the first intersection with the plane orthogonal to the axis in the model arranged in the position and orientation detected by the detection unit, and on the surface indicated by the three-dimensional point cloud data. The second intersection with the plane is superimposed and displayed on the display device.

According to this disclosure, the user can confirm the deviation between the first intersection portion and the second intersection portion on the virtual plane orthogonal to each axis. The deviation depends on the detection accuracy of the position and orientation of the work by the detection unit. Therefore, the user can easily confirm the detection accuracy of the position and orientation of the work by the detection unit.

In the above disclosure, for each of the one or more axes, at least one of the direction of the axis and the position of the plane on the axis is variable.

According to this disclosure, the user can move the virtual plane to a desired position and confirm the first intersection portion and the second intersection portion on the virtual plane.

In the above disclosure, one or more axes include three axes that are orthogonal to each other. According to this disclosure, the user can confirm the first intersection portion and the second intersection portion in each of the three virtual planes orthogonal to the three axes orthogonal to each other. As a result, the user can confirm the detection accuracy of the position and orientation of the work by the detection unit in various directions.

In the above disclosure, the image processing device is a setting unit that sets an offset amount with respect to the detected position / orientation, and a robot that picks an object according to the position / orientation moved by the offset amount from the detected position / orientation. It further includes a determination unit that determines the operation.

According to this disclosure, the user can appropriately adjust the operation of the robot by specifying the offset amount. The offset amount includes at least one of a translational component and a rotation component.

In the above disclosure, the setting unit sets the offset amount according to the movement operation with respect to the model displayed on the display device.

According to this disclosure, the user can easily specify the offset amount by performing a movement operation on the work model.

In the above disclosure, the analysis unit moves the model displayed on the display device by an offset amount.

According to this disclosure, the user can visually recognize the magnitude of the offset amount by checking the movement amount of the work model displayed on the display device.

According to an example of the present disclosure, the image processing method includes first to third steps. The first step is a step of generating three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup device installed so that the object is included in the visual field region. The second step is a step of detecting the position and orientation of the object based on the three-dimensional point cloud data and the model showing the three-dimensional shape of the object. The third step is, for each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model placed in the detected position and orientation, and the plane on the plane indicated by the 3D point cloud data. This is a step of superimposing and displaying the second intersection portion with and on the display device.

According to an example of the present disclosure, the program causes a computer to perform the above method. These disclosures also make it easier for the user to confirm the detection accuracy of the position and orientation of the object.

According to the present disclosure, the user can easily confirm the detection accuracy of the position and orientation of the object.

It is the schematic which shows the whole structure of the control system which concerns on embodiment. It is a figure which shows the correspondence | correspondence of the coordinate system of each member which constitutes a control system. It is a figure which shows the positional relationship between a work model and an end effector model which holds the work model. It is a figure which shows an example of the measurement surface shown by 3D point cloud data, and the work model arranged in the position and orientation of the work detected from the 3D point cloud data. It is a figure which shows another example of the measurement surface shown by 3D point cloud data, and the work model arranged in the position and orientation of the work detected from the 3D point cloud data. It is a figure which shows an example of the analysis screen for confirming the detection result of the position and orientation of a work. It is the schematic which shows an example of the hardware configuration of the image processing apparatus shown in FIG. It is a block diagram which shows an example of the functional structure of the image processing apparatus shown in FIG. It is a figure which shows an example of the analysis screen displayed by the analysis unit shown in FIG. It is a figure which shows the screen example at the time of accepting the input of the rotation component of an offset amount by a drag operation. It is a figure which shows the 1st setting example of the offset amount. It is a figure which shows an example of the translational deviation between the work model arranged in the detected position and posture, and the work. It is a figure which shows an example of the rotation deviation between a work model arranged in the detected position and posture, and a work. It is a figure which shows the 2nd setting example of the offset amount. It is a flowchart which shows the processing flow of the image processing apparatus in the analysis mode. It is a flowchart which shows the processing flow of the image processing apparatus in a normal mode.

Embodiments of the present invention will be described in detail with reference to the drawings. The same or corresponding parts in the drawings are designated by the same reference numerals and the description thereof will not be repeated.

§1 Application example First, an example of a situation in which the present invention is applied will be described with reference to FIGS. 1 to 6. FIG. 1 is a schematic view showing an overall configuration of a control system according to an embodiment. The control system SYS illustrated in FIG. 1 is incorporated in a production line or the like and controls picking of an object (hereinafter, referred to as “work 1”) housed in the container 2. The work 1 is separately stacked in the container 2. Therefore, the position and orientation of the work 1 in the container 2 are random.

The control system SYS illustrated in FIG. 1 includes an image processing device 100, a robot controller 200, a robot 300, and a measurement head 400. A display device 150 and an input device 160 are connected to the image processing device 100. The input device 160 includes, for example, a keyboard and a mouse.

The robot 300 picks the work 1. The robot 300 includes an end effector 30 that holds the work 1, an articulated arm 31 for changing the position and orientation of the end effector 30, and a base 32 that supports the articulated arm 31. The operation of the robot 300 is controlled by the robot controller 200.

The end effector 30 illustrated in FIG. 1 has two claws 33, and the work 1 is gripped by using the two claws 33. The claw 33 is a portion that comes into contact with the work 1 when gripping the work 1, and is also referred to as a “finger”. The method of holding the work 1 is not limited to gripping with two claws. For example, the end effector 30 may grip the work with three or more claws, or may suck the work 1 with a suction pad.

The robot controller 200 controls the articulated arm 31 of the robot 300 in response to an operation command from the image processing device 100. Specifically, the robot controller 200 controls the articulated arm 31 so that the end effector 30 performs a picking operation on the work 1.

The measurement head 400 is installed so that the container 2 and the work 1 housed in the container 2 are the subjects, and images the subject. The measuring head 400 includes a projection unit 401 and an imaging unit 402. The projection unit 401 projects an arbitrary projection pattern light onto the subject according to an instruction from the image processing device 100. The projection pattern is, for example, a pattern in which the brightness periodically changes along a predetermined direction in the irradiation surface. The imaging unit 402 images a subject in a state where the projection pattern light is projected.

The projection unit 401 includes, as main components, a light source such as an LED (Light Emitting Diode) or a halogen lamp, and a filter arranged on the irradiation surface side of the projection unit 401. The filter generates projection pattern light necessary for measuring a three-dimensional shape as described later, and can arbitrarily change the in-plane light transmittance in accordance with a command from the image processing device 100. The projection unit 401 may be configured by using a liquid crystal or a DMD (Digital Mirror Device) and a light source (LED, laser light source, etc.).

The image pickup unit 402 includes, for example, an optical system such as a lens and an image pickup element such as a CCD (Coupled Charged Device) or a CMOS (Complementary Metal Oxide Semiconductor) sensor as main components.

The image processing device 100 detects the position and orientation of the work 1 housed in the container 2 based on the image received from the measurement head 400. The image processing device 100 determines the picking operation of the end effector 30 based on the detected position and orientation of the work 1. The image processing device 100 generates an operation command corresponding to the determined picking operation, and outputs the generated operation command to the robot controller 200. The robot controller 200 causes the robot 300 to perform a picking operation determined by the image processing device 100 according to an operation command.

Specifically, the image processing device 100 determines the position and orientation (hereinafter, also referred to as “grasping point”) to be taken by the end effector 30 when gripping the work 1 according to the detected position and orientation of the work 1. decide. The image processing device 100 determines a picking operation that moves from the standby position to the gripping point, grips the work 1, and then moves to the target position (place position of the work 1).

A method of determining a picking operation by the image processing apparatus 100 will be described with reference to FIGS. 2 and 3. FIG. 2 is a diagram showing the correspondence between the coordinate systems of the members constituting the control system. FIG. 3 is a diagram showing the positional relationship between the work model and the end effector model that grips the work model.

In FIG. 2, the “base coordinate system B” is a coordinate system based on the base 32 of the robot 300. The "tool coordinate system T" is a coordinate system set on the tip surface (flange surface to which the end effector 30 is attached) of the articulated arm 31 of the robot 300. The position and orientation of the flange surface defines the position and orientation of the end effector 30. The "camera coordinate system C" is a coordinate system based on the image pickup unit 402 of the measurement head 400. The "work coordinate system W" is a coordinate system based on the work 1.

The position and orientation of the work 1 detected based on the image received from the measurement head 400 is indicated by, for example, a coordinate transformation matrix ^C H _{W that converts the camera coordinate system C into the work coordinate system W.} The coordinate transformation matrix ^C H _W represents the components of the basis vector of the work coordinate system W and the position of the origin in the camera coordinate system C.

On the other hand, it is the operation command outputted from the image processing apparatus 100 to the robot controller 200, for example, indicated by the coordinate transformation matrix ^B H _T to convert the base coordinate system B to the tool coordinate system T. The coordinate transformation matrix ^B H _T represents the components of the basis vector of the tool coordinate system T and the position of the origin in the base coordinate system B.

The coordinate transformation matrices ^C H _W and ^B H _T satisfy the following equation (1).
^B H _C・^C H _W = ^B H _T・^T H _W・ ・ ・ Equation (1)
In the equation (1), the coordinate transformation matrix ^B H _C is a matrix that transforms the base coordinate system B into the camera coordinate system C. The coordinate transformation matrix ^B H _C represents the components of the basis vector of the camera coordinate system C and the position of the origin in the base coordinate system B. The coordinate transformation matrix ^T H _W is a matrix that transforms the tool coordinate system T into the work coordinate system W. The coordinate transformation matrix ^T H _W represents the component of the basis vector of the work coordinate system W and the position of the origin in the tool coordinate system T.

In the present embodiment, the measurement head 400 is fixed at a position where the entire container 2 can be imaged. Therefore, the coordinate transformation matrix ^B H _C of the formula (1) is represented by a fixed value, determined by the calibration to be performed in advance. As the calibration, for example, a hand-eye calibration in which a marker is attached to the flange surface of the robot 300 and the marker is imaged by the imaging unit 402 of the measuring head 400 while operating the robot 300 can be adopted.

The positional posture (grip point) of the end effector 30 when gripping the work 1 indicates a model showing the three-dimensional shape of the work 1 (hereinafter, referred to as “work model 1M”) and the three-dimensional shape of the end effector 30. It is registered in advance using a model (hereinafter referred to as "end effector model 30M").

As shown in FIG. 3, the user can use the work model 1M and the end effector model in the virtual space so as to match the relative position and orientation of the end effector 30 with respect to the work 1 when the work 1 is gripped by the end effector 30. 30M and so on. ^{In this way, the coordinate transformation matrix TM} H _WM indicating the relative position and orientation of the work model 1M with respect to the end effector model 30M arranged in the virtual space is calculated. The calculated coordinate transformation matrix ^TM H _WM is registered in advance in the image processing device 100 as grip point data.

The coordinate transformation matrix ^TM H _WM represents the components of the basis vector and the position of the origin of the coordinate system WM of the work model 1M in the coordinate system TM with respect to the surface on the articulated arm side of the end effector model 30M. The basis vector and origin of the coordinate system TM should match the basis vector and origin of the tool coordinate system T (see FIG. 2), respectively, assuming that the end effector model 30M is attached to the flange surface of the articulated arm 31. Set. The basis vector and origin of the work model 1M are set so as to coincide with the basis vector and origin of the work coordinate system W (see FIG. 2) when the work model 1M is superposed on the work 1.

The image processing apparatus 100 inputs the gripping point data (coordinate transformation matrix ^TM H _WM ^{) registered in advance in the coordinate transformation matrix T} H _{W of the equation (1).} Thus, the image processing apparatus 100 according to the following equation (2), the coordinate transformation matrix ^B H representing the position and orientation of the end effector 30 at the time of holding the workpiece 1 that is detected based on the image received from the measurement head 400 Calculate _T.
^B H _T = ( ^TM H _WM ) ^-1・^B H _C・^C H _W・ ・ ・ Equation (2)
The image processing apparatus 100, in accordance with the calculated coordinate conversion matrix ^B H _T has may generate an operation command to the robot controller 200.

The detection of the position and orientation of the work 1 by the image processing device 100 is executed by using a three-dimensional search. In the three-dimensional search, feature points that match or approximate the work model showing the three-dimensional shape of the work 1 are extracted and extracted from the three-dimensional point cloud data generated based on the image received from the measurement head 400. This is a process of detecting the position and orientation of the work 1 based on the feature points. The three-dimensional point cloud data indicates an object surface (hereinafter, referred to as “measurement surface”) existing in the visual field region, which is measured from an image received from the measurement head 400. The three-dimensional search is performed using a known detection algorithm (also referred to as a "recognition algorithm").

FIG. 4 is a diagram showing an example of a measurement surface shown by the three-dimensional point cloud data and a work model arranged in the position and orientation of the work detected from the three-dimensional point cloud data. FIG. 5 is a diagram showing another example of the measurement surface shown by the three-dimensional point cloud data and the work model arranged in the position and orientation of the work detected from the three-dimensional point cloud data.

As shown in FIGS. 4 and 5, due to the habit of the detection algorithm, the work model 71 placed in the detected position / orientation is always translated or translated in a fixed direction with respect to the measurement surface 70 indicated by the three-dimensional point cloud data. Can rotate. In the example shown in FIG. 4, the work model 71 is displaced in the + Z direction with respect to the measurement surface 70. In the example shown in FIG. 5, the work model 71 is deviated from the measurement surface 70 in the rotation direction about the Y axis. However, the directions of translational deviation and rotational deviation of the work model 71 with respect to the measurement surface 70 depend on the habit of the detection algorithm and are not limited to the directions illustrated in FIGS. 4 and 5.

Conventionally, the user has not been able to sufficiently confirm the detection deviation as shown in FIGS. 4 and 5. Therefore, the image processing device 100 according to the present embodiment presents to the user an analysis screen for confirming the detection accuracy of the position and orientation of the work 1.

FIG. 6 is a diagram showing an example of an analysis screen for confirming the detection result of the position and orientation of the work. The analysis screen 50 illustrated in FIG. 6 is displayed on the display device 150 (see FIG. 1) by the image processing device 100.

As shown in FIG. 6, the analysis screen 50 includes regions 51, 61-62. In the area 51, a distance image indicated by three-dimensional point cloud data measured using the measurement head 400 is displayed. The distance image is represented by a color map, and the Z coordinate (height) is indicated by color or density for each pixel on the XY plane. The range of Z coordinates that the distance image can take is indicated by the color bar 52. The X-axis, Y-axis, and Z-axis are orthogonal to each other. The Z-axis is parallel to the optical axis of the imaging unit 402.

In the region 51, the contour 71d of the region in which the work model 71 arranged in the position and orientation detected by the three-dimensional search is projected in the Z-axis direction is superimposed and displayed on the distance image. In the example shown in FIG. 6, the positions and orientations of the plurality of works 1 are detected from the three-dimensional point cloud data by the three-dimensional search. Therefore, a plurality of contours 71d are displayed.

Further, in the region 51, a line 53 for designating the position of the virtual plane orthogonal to the X axis (hereinafter, referred to as “first virtual plane”) and the virtual plane orthogonal to the Y axis (hereinafter, “third virtual plane”). A line 54 for designating the position of (referred to as "2 virtual plane") is displayed. On the color bar 52, a line 55 for designating the position of the virtual plane (hereinafter, referred to as “third virtual plane”) orthogonal to the Z axis is displayed. The positions of lines 53, 54, 55 are variable depending on the operation with respect to the input device 160.

In the region 60, the intersection portion 71a with the first virtual plane in the work model 71 arranged in the detected position and orientation and the intersection portion 70a with the first virtual plane on the measurement surface 70 are superimposed and displayed.

In the region 61, the intersection portion 71b with the second virtual plane in the work model 71 arranged in the detected position and orientation and the intersection portion 70b with the second virtual plane on the measurement surface 70 are superimposed and displayed.

In the region 62, the intersection 71c with the third virtual plane in the work model 71 arranged in the detected position and orientation and the intersection 70c with the third virtual plane on the measurement surface 70 are superimposed and displayed.

As described above, the image processing apparatus 100 has the intersection of the X, Y, and Z axes with the plane orthogonal to the axis in the work model 71 arranged in the detected position and orientation, and the measurement surface 70. The intersection with the plane is superimposed and displayed on the display device 150. The user can confirm the deviation of the two intersections on the plane orthogonal to each of the X, Y, and Z axes. The deviation depends on the detection accuracy of the position and orientation of the work by the three-dimensional search. Therefore, the user can easily confirm the detection accuracy of the position and orientation of the work by the three-dimensional search.

§2 Specific example Next, a specific example of the control system according to the present embodiment will be described.

<A. Image processing device hardware configuration example>
The image processing device 100 is typically a computer having a general-purpose architecture, and executes image processing according to the present embodiment by executing a pre-installed program (instruction code). .. Such a program is typically distributed in a state of being stored in various recording media or the like, or installed in the image processing apparatus 100 via a network or the like.

When using such a general-purpose computer, an OS (Operating System) for executing the basic processing of the computer is installed in addition to the application for executing the image processing according to the present embodiment. It may have been done. In this case, the program according to the present embodiment may call the necessary modules in a predetermined array at a predetermined timing among the program modules provided as a part of the OS to execute the process. Good. That is, the program itself according to the present embodiment does not include the module as described above, and the process may be executed in cooperation with the OS. The program according to the present embodiment may not include such a part of the modules.

Further, the program according to the present embodiment may be provided by being incorporated into a part of another program. Even in that case, the program itself does not include the modules included in the other programs to be combined as described above, and the processing is executed in cooperation with the other programs. That is, the program according to the present embodiment may be a form incorporated in such another program. Note that some or all of the functions provided by executing the program may be implemented as a dedicated hardware circuit.

FIG. 7 is a schematic view showing an example of the hardware configuration of the image processing apparatus shown in FIG. As shown in FIG. 7, the image processing device 100 includes a CPU (Central Processing Unit) 110 which is an arithmetic processing unit, a main memory 112 and a hard disk 114 as storage units, a measurement head interface 116, and an input interface 118. , A display controller 120, a communication interface 124, and a data reader / writer 126. Each of these parts is connected to each other via a bus 128 so as to be capable of data communication.

The CPU 110 expands the programs (codes) installed on the hard disk 114 into the main memory 112 and executes them in a predetermined order to perform various operations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory), and holds an image acquired by the measurement head 400 in addition to a program read from the hard disk 114. To do. Further, the hard disk 114 stores various data and the like as will be described later. In addition to the hard disk 114, or instead of the hard disk 114, a semiconductor storage device such as a flash memory may be adopted.

The measurement head interface 116 mediates data transmission between the CPU 110 and the measurement head 400. That is, the measurement head interface 116 is connected to the measurement head 400. The measurement head interface 116 gives a projection command and an imaging command to the measurement head 400 according to an internal command generated by the CPU 110. The measurement head interface 116 includes an image buffer 116a for temporarily storing an image from the measurement head 400. When a predetermined number of frames of images are accumulated in the image buffer 116a, the measurement head interface 116 transfers the accumulated images to the main memory 112.

The input interface 118 mediates data transmission between the CPU 110 and the input device 160. That is, the input interface 118 receives the input information input by the user to the input device 160.

The display controller 120 is connected to the display device 150 and controls the display of the display device 150 so as to notify the user of the processing result of the CPU 110 and the like.

The communication interface 124 mediates data transmission between the CPU 110 and an external device such as the robot controller 200. The communication interface 124 typically comprises Ethernet (registered trademark), USB (Universal Serial Bus), or the like.

The data reader / writer 126 mediates data transmission between the CPU 110 and the memory card 106, which is a recording medium. That is, the memory card 106 is distributed in a state in which a program or the like executed by the image processing device 100 is stored, and the data reader / writer 126 reads the program from the memory card 106. Further, the data reader / writer 126 writes the image captured by the measurement head 400 and / or the processing result in the image processing device 100 to the memory card 106 in response to the internal command of the CPU 110. The memory card 106 is a general-purpose semiconductor storage device such as SD (Secure Digital), a magnetic storage medium such as a flexible disk, or an optical storage medium such as a CD-ROM (Compact Disk Read Only Memory). Etc.

<B. Function configuration example of image processing device>
FIG. 8 is a block diagram showing an example of the functional configuration of the image processing apparatus shown in FIG. As shown in FIG. 8, the image processing device 100 includes a head control unit 10, a storage unit 11, an image processing unit 12, and a UI (User Interface) processing unit 13. The head control unit 10 is realized by the CPU 110 and the measurement head interface 116 shown in FIG. 7. The storage unit 11 is realized by the main memory 112 and the hard disk 114 shown in FIG. 7. The image processing unit 12 is realized by the CPU 110 and the communication interface 124 shown in FIG. The UI processing unit 13 is realized by the CPU 110, the input interface 118, and the display controller 120 shown in FIG. 7.

The image processing device 100 has a normal mode and an analysis mode as operation modes. When the normal mode is selected, the head control unit 10 and the image processing unit 12 operate. When the analysis mode is selected, the head control unit 10, the image processing unit 12, and the UI processing unit 13 operate. The analysis mode is selected, for example, at the start-up of the production line and at regular maintenance.

The head control unit 10 controls the operation of the measurement head 400. The head control unit 10 outputs a projection command instructing the projection of a predetermined projection pattern light to the measurement head 400. The head control unit 10 outputs an imaging command to the measurement head 400 in a state where the projection pattern light is projected. The head control unit 10 stores the image data 11a acquired from the measurement head 400 in the storage unit 11.

The storage unit 11 stores the image data 11a obtained by imaging the measurement head 400. Further, the storage unit 11 stores in advance CAD (Computer Aided Design) data 11b indicating the three-dimensional shape of the work 1 and gripping point data 11c indicating the position and orientation of the end effector 30 when gripping the work 1. To. Further, the storage unit 11 also stores offset data 11d indicating a correction amount (hereinafter, referred to as “offset amount”) for the position / orientation detected by the three-dimensional search. The offset data 11d is generated by the UI processing unit 13. The details of the offset data 11d will be described later.

The image processing unit 12 processes the image data 11a obtained by imaging the measurement head 400. As shown in FIG. 8, the image processing unit 12 includes a 3D image generation unit 14, a detection unit 15, an operation command determination unit 16, and a communication unit 17.

The 3D image generation unit 14 performs three-dimensional measurement in the visual field region of the measurement head 400 based on the image data 11a, and generates three-dimensional point cloud data. The three-dimensional point cloud data indicates the three-dimensional coordinates of each point on the object surface (measurement surface) existing in the visual field region of the measurement head 400.

The three-dimensional point cloud data is, for example, the three-dimensional coordinates (x ₁ , y ₁ , z ₁ ), ..., (x _M , y ₁ , z _M ), (x M, y 1, z M) of M × N points on the measurement surface. x ₁ , y ₂ , z _{M + 1} ), ···, (x _M , y ₂ , z _2M ), ···, (x _M , y _N , z _NM ) are shown. The points (x ₁ , y ₁ ), ..., (X _M , y _N ) represent M × N points arranged in a matrix on the XY plane orthogonal to the optical axis of the imaging unit 402. When the image pickup unit 402 generates the image data 11a by perspective projection, the point cloud indicated by the three-dimensional point cloud data generated based on the image data 11a is usually a matrix when projected on the XY plane. Not arranged in a shape. In such a case, the 3D image generation unit 14 appropriately interpolates the coordinate values of the three-dimensional point cloud data so as to correspond to the M × N point clouds arranged in a matrix when projected on the XY plane. , The point cloud indicated by the three-dimensional point cloud data may be rearranged.

As the three-dimensional measurement process, for example, a triangular ranging method or a coaxial method can be adopted. The triangular ranging method is a method in which the optical axes of imaging and projection are separated by the length of the baseline, and the parallax is converted into a distance. The triangular ranging method includes an active method and a passive method. Active methods include structured illumination methods, phase shift methods, and spatial coding methods. The coaxial method is a method in which the optical axes of the imaging and the distance measuring means are set to be the same. The focusing method is included as the distance measuring means. In addition, a TOF (Time of Flight) method is also included as a method close to coaxial. The 3D image generation unit 14 may execute the three-dimensional measurement process by using any of these methods. For example, a three-dimensional measurement process is performed using a phase shift method.

The phase shift method is a method using an image (sinusoidal projection image) captured in a state where a projection pattern light whose shading is changed in a sinusoidal shape is projected onto a subject. A plurality of projection patterns having different shade change periods and phases are prepared, and the projection unit 401 of the measurement head 400 sequentially irradiates the visual field region of the imaging unit 402 with the plurality of projection pattern lights. The 3D image generation unit 14 reads out a group of image data 11a (sine wave projection image group) to be imaged when a plurality of projection patterns are projected from the storage unit 11. Then, the 3D image generation unit 14 calculates the three-dimensional coordinates based on the change in the brightness (brightness and brightness) of the corresponding portion between the image data 11a.

The detection unit 15 detects the position and orientation of the work 1 by a three-dimensional search. Specifically, the detection unit 15 collates a plurality of template data created in advance with the three-dimensional point cloud data, and extracts data similar to the template from the three-dimensional point cloud data. The template data is created in advance based on the CAD data 11b. Typically, a template in which a work model shown by CAD data 11b and a plurality of virtual viewpoints are arranged in a virtual space, and the coordinates of points on the surface of the work model when viewed from each of the plurality of virtual viewpoints are shown. The data is created. The detection unit 15 detects the position and orientation of the work 1 based on data similar to the template extracted from the three-dimensional point cloud data. The position and orientation of the work 1 detected by the detection unit 15 are indicated by the coordinate system (camera coordinate system C (see FIG. 2)) of the image pickup unit 402 of the measurement head 400. The detection unit 15 may detect the position and orientation of the work 1 by using a known detection algorithm.

When the detection unit 15 detects the position and orientation of the plurality of works 1, one of the plurality of works 1 is selected as a picking target according to a predetermined selection criterion.

The operation command determination unit 16 determines the operation of the end effector 30 according to the position and orientation of the work 1 to be picked. Specifically, the operation command determination unit 16 includes a coordinate transformation matrix ^C H _W indicating the position / orientation detected by the detection unit 15, a coordinate transformation matrix ^B H _C calculated by prior calibration, and grip point data. By substituting the coordinate transformation matrix ^TM H _WM represented by 11c into the above equation (2), the coordinate transformation matrix ^B H _T is calculated. The image processing apparatus 100, in accordance with the calculated coordinate conversion matrix ^B H _T has to generate an operation command to the robot controller 200.

However, when the offset data 11d is stored in the storage unit 11, the operation command determination unit 16 ^{corrects the coordinate transformation matrix C} H _W indicating the position / orientation detected by the detection unit 15 by the offset amount indicated by the offset data 11d. To do. The operation command determination unit 16 ^{substitutes the corrected coordinate transformation matrix C} H _W into the above equation (2).

The communication unit 17 can communicate with the robot controller 200, and transmits the operation command generated by the operation command determination unit 16 to the robot controller 200.

The UI processing unit 13 executes a process of providing the user with information regarding the operation of the image processing device 100 and the operation of the robot 300. As shown in FIG. 8, the UI processing unit 13 includes an analysis unit 18 and a setting unit 19.

The analysis unit 18 acquires the latest three-dimensional point cloud data generated by the 3D image generation unit 14 and the detection result (position and orientation of the work 1) of the detection unit 15 for the three-dimensional point cloud data. The analysis unit 18 executes analysis processing on the acquired three-dimensional point cloud data and the detection result, and displays an analysis screen showing the analysis result on the display device 150. The details of the analysis screen will be described later.

The setting unit 19 sets the offset amount with respect to the position / orientation detected by the three-dimensional search. As will be described later, the analysis screen displayed on the display device 150 by the analysis unit 18 is provided with an input field for setting the offset amount. The setting unit 19 sets the offset amount according to the input to the input field. The setting unit 19 generates offset data 11d indicating the set offset amount, and stores the generated offset data 11d in the storage unit 11.

<C. Analysis screen>
FIG. 9 is a diagram showing an example of an analysis screen displayed on the display device by the analysis unit shown in FIG. The analysis screen 50a illustrated in FIG. 9 includes regions 51, 60 to 62, similar to the analysis screen 50 shown in FIG. The analysis screen 50a further sets an input field 63 for selecting one work model arranged in the detected position and orientation, input fields 64 and 65 for inputting an offset amount, and an offset amount. Includes OK button 68 and.

The analysis unit 18 displays the distance image represented by the three-dimensional point cloud data acquired from the 3D image generation unit 14 in the area 51. Further, the analysis unit 18 superimposes and displays the contour 71d of the region where the work model 71 arranged in the position and orientation of the work 1 indicated by the detection result acquired from the detection unit 15 is projected on the XY plane on the distance image. Specifically, the analysis unit 18 converts the coordinates of the CAD data 11b so as to match the position and orientation indicated by the detection result. The work model 71 is represented by the coordinate-transformed CAD data 11b.

The analysis unit 18 determines a first virtual plane orthogonal to the X axis based on the X coordinate = x0 of the line 53 on the distance image. Specifically, the analysis unit 18 determines the plane represented by x = x0 as the first virtual plane.

The analysis unit 18 displays the intersection 71a with the first virtual plane in the work model 71 in the area 60. As described above, the work model 71 is indicated by the CAD data 11b whose coordinates have been transformed so as to match the position and orientation indicated in the detection result. The analysis unit 18 displays the line of intersection between the surface element (triangular mesh or the like) indicated by the coordinate-transformed CAD data 11b and the first virtual plane as the intersection portion 71a.

Further, the analysis unit 18 displays the intersection 70a with the first virtual plane on the measurement surface 70 represented by the three-dimensional point cloud data in the region 60. That is, the analysis unit 18 superimposes and displays the intersection portion 70a and the intersection portion 71a.

When the three-dimensional point cloud data shows the coordinates (x ₁ , y ₁ , z ₁ ), ..., (X _M , y _N , z _NM ) of M × N points on the measurement surface, the line 53 The X coordinate may be set so that only _one of x 1, ..., X _M of a predetermined pitch Δx interval can be taken. In this case, the analysis unit 18 extracts the coordinates of the points having the same X coordinates (= x0) as the first virtual plane from the three-dimensional point cloud data, and based on the extracted coordinates, sets the intersection portion 70a into the region 60. Display it.

The X coordinate of the line 53 _{may be set so that not only x 1} , ..., X _M but also other values can be taken. In this case, the analysis unit 18 extracts N pairs consisting of two points sandwiching the first virtual plane from the three-dimensional point cloud data. The pair consists of two points whose distance along the X-axis direction is equal to or less than a predetermined pitch Δx. For example, when the X coordinate x0 of the first virtual plane is x _k-1 <x ₀ <x k, the N pairs are points (x _k-1 , y _i , z _{(i-1) M + k-. It} is a pair (i is an integer of 1 to N) consisting of 1) and a point (x _k , y _i , z _{(i-1) M + k).} For each of the N pairs, the analysis unit 18 performs an interpolation calculation using the coordinates of the two points constituting the pair on the intersection line between the measurement surface indicated by the three-dimensional point cloud data and the first virtual plane. Find the coordinates of the point. For example, the analysis unit 18 uses the coordinates (x _k-1 , y ₁ , z _k-1 ) and the coordinates (x _k , y ₁ , z _k ) on the intersection line between the measurement surface and the first virtual plane. point of coordinates _{(x0, y 1, {(} x k -x0) z k-1 + (x0-x k-1) z k} / (x k -x k-1)) for calculating a.

Each time the position of the line 53 is changed, the analysis unit 18 updates the intersecting portions 70a and 71a displayed in the area 60 according to the changed position of the line 53. As a result, the user can confirm the relative positional relationship between the intersecting portions 70a and 71a on the first virtual plane at the desired position by changing the position of the line 53.

The analysis unit 18 determines a second virtual plane orthogonal to the Y axis based on the Y coordinate = y0 of the line 54 on the distance image. Specifically, the analysis unit 18 determines the plane represented by y = y0 as the second virtual plane.

The analysis unit 18 displays the intersection 71b with the second virtual plane in the work model 71 in the area 61. The analysis unit 18 displays the line of intersection between the surface element and the second virtual plane indicated by the CAD data 11b coordinate-transformed so as to match the position and orientation indicated by the detection result as the intersection portion 71b.

Further, the analysis unit 18 displays the intersection 70b with the second virtual plane on the measurement surface 70 indicated by the three-dimensional point cloud data in the region 61. That is, the analysis unit 18 superimposes and displays the intersection portion 70b and the intersection portion 71b.

When the three-dimensional point cloud data shows the coordinates (x ₁ , y ₁ , z ₁ ), ..., (X _M , y _N , z _NM ) of M × N points on the measurement surface, the line 54 The Y coordinate may be set so that only _one of y 1, ..., Y _N at a predetermined pitch Δy interval can be taken. In this case, the analysis unit 18 extracts the coordinates of the points having the same Y coordinates (= y0) as the second virtual plane from the three-dimensional point cloud data, and based on the extracted coordinates, sets the intersecting portion 70b into the region 61. Display it.

The Y coordinate of the line 54 _{may be set so that not only y 1} , ..., Y _N but also other values can be taken. In this case, the analysis unit 18 extracts M pairs consisting of two points sandwiching the second virtual plane from the three-dimensional point cloud data. The pair consists of two points whose distance along the Y-axis direction is equal to or less than a predetermined pitch Δy. For example, when the Y coordinate y0 of the second virtual plane is y _k-1 <y0 <y _k , the M pairs are points (x _j , y _k-1 , z _{(k-1) M + j} ). It is a pair consisting of a point (x _j , y _k , z _{kM + j} ) (j is an integer of 1 to M). For each of the M pairs, the analysis unit 18 performs an interpolation calculation using the coordinates of the two points constituting the pair on the intersection of the measurement surface represented by the three-dimensional point cloud data and the first virtual plane. Find the coordinates of the point. For example, the analysis unit 18 uses the coordinates (x ₁ , y _k-1 , z _{(k-1) M + 1} ) and the coordinates (x ₁ , y _k , z _{kM + 1} ) to make the measurement surface and the first. 2 Coordinates of points on the intersection with the virtual plane (x ₁ , y0, {(y _{k −} y0) z _{(k-1) M + 1} + (y0 − y _k-1 ) z _{kM + 1} } / (y) _k -y _k-1)) to calculate the.

Each time the position of the line 54 is changed, the analysis unit 18 updates the intersecting portions 70b and 71b displayed in the area 61 according to the changed position of the line 54. As a result, the user can confirm the relative positional relationship between the intersecting portions 70b and 71b on the second virtual plane at the desired position by changing the position of the line 54.

The analysis unit 18 determines a third virtual plane orthogonal to the Z axis based on the Z coordinate = z0 of the line 55 on the color bar 52. Specifically, the analysis unit 18 determines the plane represented by z = z0 as the third virtual plane.

The analysis unit 18 displays the intersection 71c with the third virtual plane in the work model 71 in the area 62. The analysis unit 18 displays the line of intersection between the surface element and the third virtual plane indicated by the CAD data 11b coordinate-transformed so as to match the position and orientation indicated by the detection result as the intersection portion 71c.

Further, the analysis unit 18 displays the intersection 70c with the third virtual plane on the measurement surface 70 indicated by the three-dimensional point cloud data in the region 62. That is, the analysis unit 18 superimposes and displays the intersection portion 70c and the intersection portion 71c.

When the three-dimensional point cloud data shows the coordinates (x ₁ , y ₁ , z ₁ ), ..., (X _M , y _N , z _NM ) of M × N points on the measurement surface, the analysis unit 18 Extracts a pair consisting of two points sandwiching the third virtual plane from the three-dimensional point cloud data. The pair consists of two points where the distance along the X-axis direction is equal to or less than the predetermined pitch Δx, or two points where the distance along the Y-axis direction is equal to or less than the predetermined pitch Δy. For each pair, the analysis unit 18 performs an interpolation calculation using the coordinates of the two points constituting the pair to obtain the coordinates of the points on the intersection line between the measurement surface and the third virtual plane indicated by the three-dimensional point cloud data. Ask. For example, the analysis unit 18 has coordinates (x ₁ , y ₁ , z ₁ ), (x ₂ , y ₁ , z ₂ ) (where z ₁ and z ₂ are z) of two points constituting one pair. ₁ <z0 <z ₂ or z ₂ <z0 with <satisfy z _1), the measuring surface and the third point on the intersection line between the virtual plane coordinates _{({(| z 2 -z0 |} ) x 1 + ( | Z0-z ₁ |) x ₂ } / (| z _2- z ₁ |), y ₁ , z0) is calculated. Similarly, the analysis unit 18 has coordinates (x ₁ , y ₁ , z ₁ ), (x ₁ , y ₂ , z _{M + 1} ) (where z ₁ and z _{M + 1) of two points constituting one pair. +1} uses z ₁ <z 0 <z _{M + 1} or z _{M + 1} <z 0 <z ₁ to satisfy the coordinates of points on the intersection of the measurement surface and the third virtual plane (x ₁ , { _{(| z M + 1 -z0 |} ) y 1 + (| z0-z 1 |) y 2} / (| z M + 1 -z 1 |), z0) calculates a.

Each time the position of the line 55 is changed, the analysis unit 18 updates the intersecting portions 70c and 71c displayed in the area 62 according to the changed position of the line 55. As a result, the user can confirm the relative positional relationship of the intersecting portions 70b and 71b on the third virtual plane at the desired position by changing the position of the line 55.

The analysis unit 18 assigns a unique identification number to each of the positions and orientations of one or more work 1s indicated by the detection results acquired from the detection unit 15.

In the input field 63, an identification number corresponding to one work model 71 used for setting the offset amount is input. The analysis unit 18 highlights the contours 71d and the intersecting portions 71a to 71c corresponding to the work model 71 arranged in the position and orientation of the identification number input in the input field 63 in the areas 51 and 60 to 62, respectively. In the analysis screen 50a illustrated in FIG. 9, the contour 71d_s and the intersecting portions 71a_s to 71c_s corresponding to the work model 71 arranged in the position and orientation of the identification number input in the input field 63 are displayed by thick lines.

A translational component of the offset amount (translational component of each of the X-axis, Y-axis, and Z-axis) is input to the input field 64. When the translational component of the offset amount is input to the input field 64, the analysis unit 18 uses only the translational component of the intersecting portions 71a_s to 71c_s of the work model 71_s arranged at the position and orientation of the identification number input to the input field 63. Move. As a result, the user can visually recognize the magnitude of the offset amount.

In the input field 65, the rotation component of the offset amount (translational component of each of the X-axis, Y-axis, and Z-axis) is input. The input field 65 includes a radio button group 66 for setting the center of rotation. The radio button group 66 includes a radio button for setting the click position at the center of rotation, a radio button for setting the center of gravity of the work model 71 at the center of rotation, and CAD data 11b after coordinate conversion indicating the work model 71. Includes a radio button to set the origin to the center of rotation.

Further, the input field 65 includes an input field 67 for inputting a rotation angle. When the rotation angle is input to the input field 67, the analysis unit 18 moves the intersecting portions 71a_s to 71c_s of the work model 71_s arranged at the position and orientation of the identification number input in the input field 63 by the rotation angle. As a result, the user can visually recognize the magnitude of the offset amount. At this time, the analysis unit 18 sets the position corresponding to the radio button checked in the radio button group 66 as the center of rotation.

When the OK button 68 is operated, the setting unit 19 sets the offset amount according to the input to the input fields 64 and 65. The setting unit 19 generates offset data 11d indicating the set offset amount. The offset data 11d is represented by a coordinate transformation matrix. The analysis screen 50a shown in FIG. 9 includes both input fields 64 and 65. As a result, the setting unit 19 can set both the rotation component and the translation component of the offset amount. However, one of the input fields 64 and 65 may be omitted. In this case, the setting unit 19 only needs to set one of the rotation component and the translation component of the offset amount.

The setting unit 19 may accept a movement operation (drag operation) for the intersecting portions 71a_s to 71c_s of the work model 71_s arranged at the position and orientation of the identification number input in the input field 63. The setting unit 19 may change the numerical values in the input fields 64 and 65 according to the drag operation. That is, the setting unit 19 may set the offset amount according to the movement operation for the intersecting portions 71a_s to 71c_s of the work model 71_s.

For example, the setting unit 19 may accept the input of at least one translation component in the Y-axis direction and the Z-axis direction of the offset amount by the translation movement operation with respect to the intersection portion 71a_s displayed in the area 60. Due to the translational movement with respect to the intersecting portion 71b_s displayed in the region 61, the setting unit 19 may accept the input of at least one translational component of the offset amount in the X-axis direction and the Z-axis direction. Due to the translational movement with respect to the intersecting portion 71c_s displayed in the region 62, the setting unit 19 may accept the input of at least one translational component in the X-axis direction and the Y-axis direction of the offset amount.

Further, the setting unit 19 may accept the input of the rotation component of the offset amount about the axis parallel to the X axis by the rotation movement operation with respect to the intersection portion 71a_s displayed in the area 60. By the rotation movement operation with respect to the intersection portion 71b_s displayed in the area 61, the setting unit 19 may accept the input of the rotation component of the offset amount about the axis parallel to the Y axis. By the rotation movement operation with respect to the intersection portion 71c_s displayed in the area 62, the setting unit 19 may accept the input of the rotation component of the offset amount about the axis parallel to the Z axis.

FIG. 10 is a diagram showing an example of a screen when receiving an input of a rotation component of an offset amount by a drag operation. FIG. 10 shows an example of a screen when receiving an input of a rotation component having an offset amount centered on an axis parallel to the Y axis. As shown in FIG. 10, the setting unit 19 displays a mark 72 indicating the center of rotation. The position of the mark 72 corresponds to the radio button checked in the radio button group 66. The setting unit 19 receives an input of a rotation angle centered on an axis parallel to the Y axis in response to a drag operation (drag operation in the direction of the arrow 73) with respect to the intersecting portion 71b_s.

<D. First setting example of offset amount>
As described above with reference to FIGS. 4 and 5, the work model 71 arranged in the detected position / orientation due to the habit of the detection algorithm is always in a fixed direction with respect to the measurement surface 70 indicated by the three-dimensional point cloud data. Can be translated or rotated. By confirming the analysis screen 50a illustrated in FIG. 8, the user can confirm the deviation between the work model 71 arranged in the detected position and orientation and the measurement surface 70. Depending on the confirmation of such a deviation, the user can take measures such as reviewing the detection algorithm. However, the detection algorithm is generally complicated, and the user cannot easily take such a measure. Therefore, the user may input an offset amount for eliminating such a deviation.

FIG. 11 is a diagram showing a first setting example of the offset amount. In FIG. 11, (a) shows a state before setting the offset amount, and (b) shows a state after setting the offset amount. In FIG. 11, the lower part shows the work model 71 and the measurement surface 70 arranged in the detected position and orientation, and the upper part shows the work 1 and the end effector 30 when gripping the work 1.

FIG. 11 shows an example in which the work model 71 arranged in the detected position and orientation is translated in the −Z direction with respect to the measurement surface 70 due to the habit of the detection algorithm. In this case, as shown in the upper part of (a), the gripping position of the end effector 30 with respect to the work 1 is larger than the relative position of the end effector model 30M with respect to the work model 1M shown in the gripping point data 11c (see FIG. 3). Move in the −Z direction. As a result, for example, the possibility that the claw 33 collides with the other work 1 increases.

Therefore, the user inputs the offset amount so that the deviation between the work model 71 and the measurement surface 70 becomes small. Specifically, the user inputs a translational component in the + Z direction. As a result, as shown in the upper part of (b), the gripping position of the end effector 30 with respect to the work 1 approaches the relative position (see FIG. 3) of the end effector model 30M with respect to the work model 1M shown in the gripping point data 11c. .. As a result, for example, the possibility that the claw 33 collides with another work 1 can be reduced.

<E. Second setting example of offset amount>
As described above, the operation command determination unit 16 generates an operation command using the above equation (2). In the equation (2), the coordinate transformation matrix ^B H _C contains a component due to a calibration error. The coordinate transformation matrix ^C H _W is caused by an error between the actual dimensions of the work 1 and the dimensions of the work model shown in the CAD data 11b (including manufacturing tolerances, errors due to aging of the manufacturing equipment of the work 1). Contains ingredients. Furthermore, even when generating an operation command according to the coordinate transformation matrix ^B H _T calculated using the above equation (2), aging of the robot 300, due to the influence of ambient temperature of the robot 300, the operation command and the robot 300 There may be an error with the actual operation. Due to these causes, even when the deviation between the work model 71 and the measurement surface 70 is small, the gripping position of the end effector 30 with respect to the work 1 is the gripping position of the end effector model 30M with respect to the work model 1M shown by the gripping point data 11c. It may not match the relative position (see FIG. 3).

FIG. 12 is a diagram showing an example of translational deviation between the work model and the work arranged in the detected position and orientation. In the example shown in FIG. 12, the actual position of the work 1 is deviated from the position of the work model 71 in the −Z direction. Therefore, the work 1 may not be gripped by the end effector 30.

FIG. 13 is a diagram showing an example of rotational deviation between the work model arranged in the detected position and orientation and the work. When the work 1 has an elongated shape, even if the angle formed by the longitudinal direction of the work 1 and the XY plane is slightly different, the shape of the work 1 changes when viewed from the imaging unit 402 of the measuring head 400. small. Therefore, as shown in FIG. 13, the actual angle formed by the longitudinal direction of the work 1 and the XY plane may deviate from the angle formed by the longitudinal direction of the work model 71 and the XY plane. Therefore, the work 1 may not be gripped by the end effector 30.

Even in such a case, the user can input the offset amount on the analysis screen 50a in FIG. The user may confirm the state when the end effector 30 grips the work 1 and input the offset amount according to the state.

FIG. 14 is a diagram showing a second setting example of the offset amount. Similar to FIG. 11, (a) shows the state before setting the offset amount, and (b) shows the state after setting the offset amount. The lower part shows the work model 71 and the measurement surface 70 arranged in the detected position and orientation, and the upper part shows the relative positional relationship between the work 1 and the end effector 30.

FIG. 14 shows an example in which the deviation between the work model 71 arranged in the detected position and orientation and the measurement surface 70 is small. However, as shown in the upper part of (a), due to the above cause, the gripping position of the end effector 30 with respect to the work 1 is the relative position of the end effector model 30M with respect to the work model 1M shown in the gripping point data 11c (FIG. 3). (See) is moving in the + Z direction. Therefore, there is a possibility that the work 1 cannot be gripped.

In such a case, the user may input the offset amount so that the end effector 30 can grip the work 1. Specifically, the user inputs a translational component in the −Z direction. At this time, the amount of deviation between the work model 71 and the measurement surface 70 corresponds to the offset amount. Therefore, the user can input an appropriate offset amount while referring to the measurement surface 70 displayed on the analysis screen 50a. By setting an appropriate offset amount, as shown in the upper part of (b), the gripping position of the end effector 30 with respect to the work 1 is relative to the end effector model 30M with respect to the work model 1M shown in the gripping point data 11c. Approach the position (see FIG. 3). As a result, the end effector 30 can accurately grip the work 1.

<F. Processing flow of the image processing device in the analysis mode>
FIG. 15 is a flowchart showing a processing flow of the image processing apparatus in the analysis mode. First, the CPU 110 generates three-dimensional point cloud data indicating an object surface (measurement surface) in the visual field region based on the image data 11a acquired from the measurement head 400 (step S1). The CPU 110 detects the position and orientation of the work 1 based on the three-dimensional point cloud data and the model showing the three-dimensional shape of the work 1 (the work model shown by the CAD data 11b) (step S2).

The CPU 110 has, for each of the X-axis, the Y-axis, and the Z-axis, the intersection with the plane orthogonal to the axis in the work model 71 arranged in the detected position and orientation, and the measurement surface indicated by the three-dimensional point cloud data. The intersection with the plane in the above is superimposed and displayed on the display device 150 (step S3). Specifically, the analysis screen 50a as shown in FIG. 8 is displayed on the display device 150. The CPU 110 sets the offset amount with respect to the detected position / orientation according to the input to the analysis screen 50a (step S4). After step S4, the image processing apparatus 100 ends the processing in the analysis mode.

<G. Processing flow of image processing device in normal mode>
FIG. 16 is a flowchart showing a processing flow of the image processing apparatus in the normal mode. First, the CPU 110 generates three-dimensional point cloud data indicating an object surface (measurement surface) in the visual field region based on the image data 11a acquired from the measurement head 400 (step S11). The CPU 110 detects the position and orientation of the work 1 based on the three-dimensional point cloud data and the model showing the three-dimensional shape of the work 1 (the work model shown by the CAD data 11b) (step S12).

The CPU 110 corrects the detected position and orientation of the work 1 by the offset amount set in the analysis mode (step S13). The CPU 110 determines the operation of the robot 300 that picks the work 1 according to the corrected position / orientation (step S14). The CPU 110 generates an operation command corresponding to the determined operation, and outputs the generated operation command to the robot controller 200 (step S15). After step S15, the image processing apparatus 100 ends the processing in the normal mode.

<H. Modification example>
In the above description, in the regions 60 to 62 of the analysis screens 50 and 50a, the intersections of the virtual plane orthogonal to the X, Y, and Z axes and the work model 71 and the measurement surface 70 are displayed, respectively. However, the one or more axes that define the virtual plane are not limited to the X, Y, and Z axes, and may be, for example, three axes that are orthogonal to each other and are inclined by a predetermined angle from the X, Y, and Z axes. Further, the direction of the axis that defines the virtual plane may be variable by the operation of the user. As a result, the user can appropriately change the axis in the desired direction.

<I. Action / effect>
As described above, the image processing apparatus 100 according to the present embodiment includes a 3D image generation unit 14, a detection unit 15, and an analysis unit 18. The 3D image generation unit 14 generates the three-dimensional point cloud data of the visual field region based on the image obtained from the image pickup unit 402 installed so that the work 1 is included in the visual field region. The detection unit 15 detects the position and orientation of the work 1 based on the three-dimensional point cloud data and the work model showing the three-dimensional shape of the work 1. The analysis unit 18 is indicated by three-dimensional point cloud data and an intersection with a virtual plane orthogonal to the axis in the work model arranged in the position and orientation detected by the detection unit 15 for each of the one or more axes. The intersection with the virtual plane on the measurement surface is superimposed and displayed on the display device 150.

According to the above configuration, the user can confirm the deviation of the two intersections on the virtual plane orthogonal to each axis. The deviation depends on the detection accuracy of the position and orientation of the work by the three-dimensional search. Therefore, the user can easily confirm the detection accuracy of the position and orientation of the work by the three-dimensional search.

For each of the one or more axes, at least one of the direction of the axis and the position of the virtual plane on the axis is variable. As a result, the user can move the virtual plane to a desired position and confirm the intersection of the virtual plane with the work model 71 and the measurement surface 70.

One or more axes include three axes that are orthogonal to each other. As a result, the user can simultaneously confirm the intersections of the three virtual planes orthogonal to the three axes orthogonal to each other and the work model 71 and the measurement surface 70.

The image processing device 100 further includes a setting unit 19 and an operation command determination unit 16. The setting unit 19 sets the offset amount with respect to the detected position / orientation. The operation command determination unit 16 determines the operation of the robot 300 that picks the work 1 according to the position / orientation moved from the detected position / orientation by an offset amount. As a result, the user can appropriately adjust the operation of the robot by specifying the offset amount. The offset amount includes at least one of a translational component and a rotation component.

The setting unit 19 sets the offset amount according to the movement operation for the work model displayed on the display device 150. As a result, the user can easily specify the offset amount by performing the movement operation on the work model.

The analysis unit 18 moves the work model displayed on the display device 150 by an offset amount. As a result, the user can visually recognize the magnitude of the offset amount by checking the movement amount of the work model displayed on the display device 150.

<J. Addendum>
As described above, the present embodiment and the modified examples include the following disclosures.

(Structure 1)
Image processing device (100)
Generation unit (110, 14) that generates three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup apparatus (400, 402) installed so that the object (1) is included in the visual field region. When,
A detection unit (110, 15) that detects the position and orientation of the object (1) based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object (1).
For each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model arranged in the position and orientation detected by the detection unit (110, 15) and the three-dimensional point group data. An image processing device (100) including an analysis unit (110, 18) for superimposing and displaying a second intersection portion with the plane on the surface shown on the display device (150).

(Structure 2)
The image processing apparatus (100) according to configuration 1, wherein at least one of the direction of the axis and the position of the plane on the axis is variable for each of the one or more axes.

(Structure 3)
The image processing apparatus (100) according to the configuration 1 or 2, wherein the one or more axes include three axes orthogonal to each other.

(Structure 4)
Setting units (110, 19) for setting the offset amount with respect to the detected position / orientation, and
Any of the configurations 1 to 3 further comprising a determination unit (110, 16) for determining the operation of the robot picking the object according to the position / orientation moved from the detected position / orientation by the offset amount. The image processing apparatus (100) described in the above.

(Structure 5)
The image processing device (100) according to configuration 4, wherein the setting unit (110, 19) sets the offset amount according to a movement operation with respect to the model displayed on the display device (150).

(Structure 6)
The image processing device (100) according to configuration 4, wherein the analysis unit (110, 19) moves the model displayed on the display device (150) by the offset amount.

(Structure 7)
The image processing apparatus (100) according to any one of configurations 4 to 6, wherein the offset amount includes at least one of a translational component and a rotation component.

(Structure 8)
It is an image processing method
A step of generating three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup apparatus (400, 402) installed so that the object (1) is included in the visual field region.
A step of detecting the position and orientation of the object (1) based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object, and
For each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model arranged in the detected position and orientation, and the plane on the plane indicated by the three-dimensional point cloud data. An image processing method comprising a step of superimposing and displaying two intersecting portions on a display device (150).

(Structure 9)
A program that causes a computer to execute an image processing method.
The image processing method is
A step of generating three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup apparatus (400, 402) installed so that the object (1) is included in the visual field region.
A step of detecting the position and orientation of the object (1) based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object (1).
For each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model arranged in the detected position and orientation, and the plane on the plane indicated by the three-dimensional point cloud data. A program including a step of superimposing and displaying two intersecting portions on a display device (150).

Although embodiments of the present invention have been described, it should be considered that the embodiments disclosed this time are exemplary in all respects and not restrictive. The scope of the present invention is indicated by the scope of claims, and is intended to include all modifications within the meaning and scope equivalent to the scope of claims.

1 work, 1M, 71 work model, 2 containers, 10 head control unit, 11 storage unit, 11a image data, 11b CAD data, 11c grip point data, 11d offset data, 12 image processing unit, 13 UI processing unit, 14 3D Image generation unit, 15 detection unit, 16 operation command determination unit, 17 communication unit, 18 analysis unit, 19 setting unit, 30 end effector, 30M end effector model, 31 articulated arm, 32 base, 33 claws, 50, 50a analysis Screen, 51, 60, 61, 62 areas, 52 color bars, 53, 54, 55 lines, 63, 64, 65, 67 input fields, 66 radio buttons, 68 OK buttons, 70 measurement surfaces, 70a, 70b, 70c , 71a, 71b, 71c intersection, 71d contour, 72 mark, 73 arrow, 100 image processor, 106 memory card, 110 CPU, 112 main memory, 114 hard disk, 116 measurement head interface, 116a image buffer, 118 input interface, 120 display controller, 124 communication interface, 126 data reader / writer, 128 bus, 150 display device, 160 input device, 200 robot controller, 300 robot, 400 measuring head, 401 projection unit, 402 imaging unit, SYS control system.

Claims (9)

It is an image processing device
A generator that generates three-dimensional point cloud data in the visual field region based on an image obtained from an image pickup device installed so that the object is included in the visual field region.
A detection unit that detects the position and orientation of the object based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object.
For each of the one or more axes, the first intersecting portion with the plane orthogonal to the axis in the model arranged in the position and orientation detected by the detection unit, and the plane indicated by the three-dimensional point group data. An image processing device including an analysis unit that superimposes and displays a second intersection with a plane on a display device. The image processing apparatus according to claim 1, wherein at least one of the direction of the axis and the position of the plane on the axis is variable for each of the one or more axes. The image processing apparatus according to claim 1 or 2, wherein the one or more axes include three axes orthogonal to each other. A setting unit that sets the offset amount with respect to the detected position / orientation, and
The present invention according to any one of claims 1 to 3, further comprising a determination unit that determines the operation of the robot that picks the object according to the position / orientation moved from the detected position / orientation by the offset amount. The image processing apparatus described. The image processing device according to claim 4, wherein the setting unit sets the offset amount according to a movement operation with respect to the model displayed on the display device. The image processing device according to claim 4, wherein the analysis unit moves the model displayed on the display device by the offset amount. The image processing apparatus according to any one of claims 4 to 6, wherein the offset amount includes at least one of a translational component and a rotation component. It is an image processing method
A step of generating three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup device installed so that the object is included in the visual field region.
A step of detecting the position and orientation of the object based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object, and
For each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model arranged in the detected position and orientation, and the plane on the plane indicated by the three-dimensional point cloud data. An image processing method including a step of superimposing and displaying two intersecting portions on a display device. A program that causes a computer to execute an image processing method.
The image processing method is
A step of generating three-dimensional point cloud data of the visual field region based on an image obtained from an image pickup device installed so that the object is included in the visual field region.
A step of detecting the position and orientation of the object based on the three-dimensional point cloud data and a model showing the three-dimensional shape of the object, and
For each of the one or more axes, the first intersection with the plane orthogonal to the axis in the model arranged in the detected position and orientation, and the plane on the plane indicated by the three-dimensional point cloud data. A program including a step of superimposing and displaying two intersections on a display device.