JP2018017610A - Three-dimensional measuring device, robot, robot controlling device, and robot system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional measuring device capable of suppressing an error caused by optical characteristics of an object and performing three-dimensional measurement with high accuracy even when the object is a glossy object or a translucent object.SOLUTION: A three-dimensional measuring device functions as follows. A predetermined figure is projected onto a first position and a second position in an imaging range including an object, where images can be captured by a first imaging part and a second imaging part; based on a first captured image of the imaging range including the predetermined figure projected onto the first position captured by the first imaging part, a second captured image of the imaging range captured by the second imaging part, a third captured image of the imaging range including the predetermined figure projected onto the second position, the image captured by the first imaging part, and a fourth captured image of the imaging range captured by the second imaging part, first feature points detected from the first captured image are associated with second feature points detected from the second captured image; and, subsequently, three-dimensional measurement is performed based on the images captured by the first imaging part and the second imaging part.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a three-dimensional measuring device, a robot, a robot control device, and a robot system.

  Research and development of a technique for calculating the position and orientation of an object based on the model of the object and causing a robot to perform a predetermined operation based on the calculated position and orientation are being performed.

  In this regard, pattern light is projected onto the measurement object by the pattern light projecting means, and the measurement target object on which the pattern light is projected is imaged from different directions by the plurality of image capturing means. Calibration is performed by the calibration means, and each 3D point cloud is measured by the 3D point cloud measurement means and measured by the 3D point cloud measurement means based on the image captured by each imaging means. Projecting each 3D point cloud to a predetermined coordinate system based on the calibration result obtained by the calibration means, and comparing the reliability of each 3D point cloud to integrate the measurement results of each 3D point cloud An original measuring device is known (see Patent Document 1).

JP 2015-21862 A

  In such a three-dimensional measurement apparatus, when the target object is a glossy object or a translucent object, a three-dimensional measurement is performed depending on the optical characteristics of the target object by using a phase shift method using three or more imaging means. It has been suppressed that the accuracy of the measurement result of the point cloud is lowered. However, increasing the number of image pickup means increases the cost related to the use of the three-dimensional measuring apparatus, and in addition, it is necessary to secure a space necessary for installing three or more image pickup means. .

In order to solve at least one of the above problems, one embodiment of the present invention is an imaging range including an object, and the first position and the second position in the imaging range that can be imaged by the first imaging unit and the second imaging unit. A predetermined figure is projected on the position by the projection unit, the first imaging image in which the imaging range including the predetermined figure projected on the first position is captured by the first imaging unit, and the imaging range is the second A second captured image captured by the image capturing unit; a third captured image captured by the first image capturing unit including the imaging range including the predetermined figure projected on the second position; After associating the first feature point detected from the first captured image with the second feature point detected from the second captured image based on the fourth captured image captured by the two imaging units The first imaging unit and the second imaging unit Performing three-dimensional measurement based on the image captured by a three-dimensional measuring apparatus.
With this configuration, the three-dimensional measuring apparatus projects a predetermined figure to the first position and the second position in the imaging range including the target object and imageable by the first imaging unit and the second imaging unit by the projection unit. A first captured image in which an imaging range including a predetermined graphic projected on the first position is captured by the first imaging unit, a second captured image in which the imaging range is captured by the second imaging unit, and a second First imaging based on a third captured image in which an imaging range including a predetermined graphic projected at a position is captured by the first imaging unit and a fourth captured image in which the imaging range is captured by the second imaging unit. After associating the first feature point detected from the image with the second feature point detected from the second captured image, three-dimensional measurement based on the images captured by the first imaging unit and the second imaging unit is performed. Do. Thereby, even if a target object is a glossy object or a semi-transparent object, the three-dimensional measurement apparatus can suppress an error due to the optical characteristics of the target object and can accurately perform the three-dimensional measurement.

According to another aspect of the present invention, in the three-dimensional measurement apparatus, a first vector that represents a difference between a position of the predetermined graphic in the first captured image and a position of the predetermined graphic in the third captured image; Based on at least one of the position of the predetermined graphic in the second captured image and the second vector representing the difference between the position of the predetermined graphic in the fourth captured image and the second feature point A configuration for associating feature points may be used.
With this configuration, the three-dimensional measurement apparatus allows the first vector representing the difference between the position of the predetermined graphic in the first captured image and the position of the predetermined graphic in the third captured image, and the position of the predetermined graphic in the second captured image. The first feature point and the second feature point are associated with each other based on at least one of the position of the predetermined figure in the fourth captured image and the second vector representing the difference. Thereby, the three-dimensional measurement apparatus can suppress the error due to the optical characteristic of the object based on at least one of the first vector and the second vector, and can accurately perform the three-dimensional measurement.

According to another aspect of the present invention, in the three-dimensional measurement apparatus, the first position and the second position are respectively determined from a position where the first imaging unit is provided and a position where the second imaging unit is provided. Based on correspondence information stored in advance, which is correspondence information indicating a correspondence relationship between a change in distance to a position and a change in value based on at least one of the first vector and the second vector. A configuration in which a feature point is associated with the second feature point may be used.
With this configuration, the three-dimensional measurement apparatus has a change in distance from the position where the first imaging unit is provided and the position where the second imaging unit is provided to the first position and the second position, and the first The first feature point and the second feature point are associated with each other based on correspondence information that is a correspondence information that indicates a correspondence relationship between a value change based on at least one of the vector and the second vector and is stored in advance. Thereby, the three-dimensional measuring apparatus can suppress an error due to the optical characteristics of the object based on the correspondence information, and can accurately perform the three-dimensional measurement.

According to another aspect of the present invention, in the three-dimensional measurement device, a three-dimensional point group representing a three-dimensional shape of the object is generated based on the associated first feature point and the second feature point. A configuration may be used.
With this configuration, the three-dimensional measurement apparatus generates a three-dimensional point group representing the three-dimensional shape of the target object based on the associated first feature point and second feature point. Thereby, the three-dimensional measuring apparatus can detect the position and orientation of the object with high accuracy based on the generated three-dimensional point group.

Moreover, the structure which calculates the position and attitude | position of the said target object based on the produced | generated said three-dimensional point group may be used for the other aspect of this invention in a three-dimensional measuring device.
With this configuration, the three-dimensional measurement apparatus calculates the position and orientation of the object based on the generated three-dimensional point group. Thereby, the three-dimensional measuring apparatus can output the position and orientation of the target object calculated with high accuracy to another apparatus.

Another aspect of the present invention is a robot that performs a predetermined operation based on the position and orientation of the object calculated by the three-dimensional measurement apparatus described above.
With this configuration, the robot performs a predetermined operation based on the result of the three-dimensional measurement device performing the three-dimensional measurement. Thereby, the robot can perform a predetermined operation with high accuracy.

Another aspect of the present invention is a robot control apparatus that causes the robot described above to perform the predetermined operation.
With this configuration, the robot control device causes the robot to perform a predetermined operation based on the result of the three-dimensional measurement performed by the three-dimensional measurement device. Thereby, the robot controller can cause the robot to perform a predetermined operation with high accuracy.

Another aspect of the present invention is a robot system including the projection unit, the first imaging unit, the second imaging unit, the robot described above, and the robot control device described above. is there.
With this configuration, the robot system causes the robot to perform a predetermined operation based on the result of the three-dimensional measurement performed by the three-dimensional measurement device. Accordingly, the robot system can cause the robot to perform a predetermined operation with high accuracy.

As described above, in the three-dimensional measurement apparatus, the predetermined figure is projected by the projection unit at the first position and the second position in the imaging range that includes the target object and can be imaged by the first imaging unit and the second imaging unit. A first captured image in which an imaging range including a predetermined graphic projected on the first position is captured by the first imaging unit, a second captured image in which the imaging range is captured by the second imaging unit, and a second position The first captured image is based on a third captured image obtained by imaging the imaging range including the predetermined figure projected on the first imaging unit and a fourth captured image obtained by imaging the imaging range by the second imaging unit. After associating the first feature point detected from the second feature point detected from the second captured image, three-dimensional measurement is performed based on the images captured by the first imaging unit and the second imaging unit . Thereby, even if a target object is a glossy object or a semi-transparent object, the three-dimensional measurement apparatus can suppress an error due to the optical characteristics of the target object and can accurately perform the three-dimensional measurement.
Further, the robot performs a predetermined operation based on the result of the three-dimensional measurement device performing the three-dimensional measurement. Thereby, the robot can perform a predetermined operation with high accuracy.
The robot control device and the robot system cause the robot to perform a predetermined operation based on the result of the 3D measurement performed by the 3D measurement device. Thereby, the robot control device and the robot system can cause the robot to perform a predetermined operation with high accuracy.

It is a figure which shows an example of a structure of the three-dimensional measurement system 1 which concerns on embodiment. It is a figure which shows an example of a predetermined figure. 3 is a diagram illustrating an example of a hardware configuration of a three-dimensional measurement apparatus 30. FIG. 3 is a diagram illustrating an example of a functional configuration of a three-dimensional measurement apparatus 30. FIG. It is a flowchart which shows an example of the flow of a process in which the three-dimensional measuring apparatus 30 performs three-dimensional measurement. It is a figure which illustrates the 1st captured image before the 1st feature point is detected, and the 2nd captured image before the 2nd feature point is detected. It is a figure which illustrates the 1st picked-up image after the 1st feature point is detected, and the 2nd picked-up image after the 2nd feature point is detected. It is a flowchart which shows an example of the flow of a process in which the three-dimensional measuring device 30 generates correspondence information. When the target height is Z = 0, the predetermined position of the predetermined figure projected in the projection range PA matches the first position determined in the projection range PA in the projection range PA. It is a figure which shows an example of a mode that it exists. When the target height is Z = 2, the predetermined position of the predetermined figure projected in the projection range PA matches the first position predetermined in the projection range PA in the projection range PA. It is a figure which shows an example of a mode that it exists. In the case where the target height is Z = 4, the predetermined position of the predetermined figure projected in the projection range PA matches the first position predetermined in the projection range PA in the projection range PA. It is a figure which shows an example of a mode that it exists. It is a figure which shows an example of correspondence information. It is a figure which shows an example of a structure of the robot system 2 which concerns on the modification of embodiment.

<Embodiment>
Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Configuration of 3D measurement system>
First, the configuration of the three-dimensional measurement system 1 will be described.
FIG. 1 is a diagram illustrating an example of a configuration of a three-dimensional measurement system 1 according to the embodiment. The three-dimensional measurement system 1 includes a projection unit 5, an imaging unit 10, and a three-dimensional measurement device 30.

  The projection unit 5 is a projector including, for example, a liquid crystal light valve and a projection lens for projecting a projection image, a liquid crystal driving unit, and an ultrahigh pressure mercury lamp, a metal halide lamp, and the like as a light source. The projection part 5 is installed in the position which can project a projection image in the projection range PA which is the range containing the target object O mounted on the upper surface of the worktable TB shown in FIG.

  The work table TB is a table such as a table, for example. The work table TB may be another table as long as it is a table on which the object O can be placed instead of the table.

  The object O is, for example, an industrial part or member such as a plate, screw, or bolt that is assembled to a product. In FIG. 1, the object O is represented as a rectangular parallelepiped object for simplification of the drawing. The object O may be other objects such as daily necessities and living bodies instead of industrial parts and members. Further, the shape of the object O may be another shape instead of the rectangular parallelepiped shape.

  The object O is an object having optical characteristics that cause unintentional reflection or transmission of light projected by the projection unit 5, for example. Hereinafter, as an example, the case where the object O is a semi-transparent object on at least a part of the surface on which the light is projected will be described. Note that the object O may be an object having gloss on at least a part of the surface on which the light is projected, instead of a translucent object. It may be an opaque object that does not have gloss on the part, and the entire surface may be an opaque object.

  The projection unit 5 is connected to the three-dimensional measuring device 30 via a cable so as to be communicable. Wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB (Universal Serial Bus), for example. The projection unit 5 may be configured to be connected to the three-dimensional measurement apparatus 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The imaging unit 10 is a stereo camera that includes a first imaging unit 11 and a second imaging unit 12. The imaging unit 10 can capture both the first imaging unit 11 and the second imaging unit 12 in the imaging range CA that includes the object O placed on the upper surface of the work table TB illustrated in FIG. It is installed in a proper position. Below, the case where the whole projection range PA is contained inside the imaging range CA as an example is demonstrated. Instead of this, the imaging range CA may be configured to partially overlap the projection range PA, or may be configured to be entirely included inside the projection range PA.

The first imaging unit 11 is a camera including, for example, a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is an imaging element that converts collected light into an electrical signal.
The second imaging unit 12 is, for example, a camera that includes a CCD, a CMOS, or the like that is an imaging device that converts collected light into an electrical signal.

  The first imaging unit 11 and the second imaging unit 12 are communicably connected to the three-dimensional measurement apparatus 30 via a cable. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. In addition, one or both of the first imaging unit 11 and the second imaging unit 12 is configured to be connected to the three-dimensional measurement apparatus 30 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark). May be.

  The three-dimensional measuring apparatus 30 is, for example, a workstation, a desktop PC (Personal Computer), a notebook PC, a tablet PC, a multi-function mobile phone terminal (smartphone), an electronic book reader with a communication function, a PDA (Personal Digital Assistant), or the like. Information processing apparatus.

  The three-dimensional measuring apparatus 30 causes the projection unit 5 to project a projection image onto the projection range PA. The three-dimensional measuring apparatus 30 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA including the projection range PA on which the projection image is projected. The three-dimensional measurement apparatus 30 acquires images captured by the first imaging unit 11 and the second imaging unit 12 in stereo from the first imaging unit 11 and the second imaging unit 12, respectively. The three-dimensional measuring device 30 performs three-dimensional measurement based on the acquired image. Specifically, the three-dimensional measuring device 30 generates a three-dimensional point group representing the three-dimensional shape of the object O based on the image as the three-dimensional measurement based on the acquired image. More specifically, the three-dimensional measuring device 30 detects a three-dimensional position (three-dimensional coordinates) of each point on the surface of the object O included in the image based on the image. The three-dimensional measuring device 30 generates a three-dimensional point group that is a point group representing the three-dimensional shape of the surface based on the detected three-dimensional position. That is, each point constituting the three-dimensional point group indicates a three-dimensional position. The three-dimensional position may be a three-dimensional position in the world coordinate system WC, a three-dimensional position in the robot coordinate system RC, or a three-dimensional position in another three-dimensional coordinate system. Hereinafter, as an example, a case will be described in which the three-dimensional measurement device 30 detects a three-dimensional position in the world coordinate system WC of each point on the surface of the object O included in the image based on the image. In addition, the method by which the three-dimensional measuring device 30 generates a three-dimensional point group from the image may be a known method or a method developed from now on. The three-dimensional measuring device 30 calculates the position and orientation of the object O based on the generated three-dimensional point group.

<Outline of processing performed by 3D measuring device>
Hereinafter, an outline of processing performed by the three-dimensional measurement apparatus 30 will be described. In the following, a case where the upper surface of the work table TB is parallel to the XY plane in the world coordinate system WC will be described as an example. Note that the upper surface may be configured to be non-parallel to the XY plane.

  The three-dimensional measuring apparatus 30 causes the projection unit 5 to project a projection image including a predetermined graphic within the projection range PA (that is, within the imaging range CA). At this time, the three-dimensional measuring device 30 is configured so that the predetermined position of the predetermined figure projected in the projection range PA matches the first position determined in the projection range PA within the projection range PA. The projection image is projected on the projection unit 5 to the projection range PA. The predetermined position of the predetermined figure is, for example, the position of the centroid of the predetermined figure. It should be noted that the predetermined position of the predetermined graphic may be another position associated with the predetermined graphic instead. Here, the predetermined figure will be described with reference to FIG.

  FIG. 2 is a diagram illustrating an example of a predetermined figure. The predetermined figure is a random dot pattern in this example. The random dot pattern is an image in which a plurality of dots, which are clusters of pixels having a predetermined shape, are regularly arranged, and an image in which a luminance value of 0 or 255 is randomly assigned to each dot. In the example shown in FIG. 2, the dot shape is a square shape. The dot shape may be a rectangular shape instead of a square shape, a circular shape, an elliptical shape, or another shape. Further, the predetermined figure may be any other figure instead of the random dot pattern as long as the luminance values of all the pixels are not constant (figure having a uniform luminance value). For example, the figure may be an image obtained by capturing a scene or a person. Below, the case where the whole projection image is occupied by the predetermined figure as an example is demonstrated. That is, hereinafter, a case where the projection image is a predetermined graphic will be described. Note that the projection image may be an image including a predetermined graphic in a part of the projection image.

  In the first state, the three-dimensional measurement apparatus 30 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA. The first state is a state in which the position of the predetermined figure projected within the projection range PA and the first position are coincident within the projection range PA. The three-dimensional measuring apparatus 30 acquires a captured image captured by the first imaging unit 11 by the stereo imaging from the first imaging unit 11 as a first captured image. In addition, the three-dimensional measurement apparatus 30 acquires a captured image captured by the second imaging unit 12 by the stereo imaging from the second imaging unit 12 as a second captured image.

  The three-dimensional measuring device 30 projects the predetermined figure projected in the projection range PA so that the predetermined position in the projection range PA coincides with the predetermined second position in the projection range PA. The image is projected on the projection unit 5 to the projection range PA. The second position is a position different from the first position in the projection range PA. However, the height of the second position in the world coordinate system WC must be substantially the same as the height of the first position in the world coordinate system WC. The height of the first position in the world coordinate system WC is the position in the Z-axis direction of the world coordinate system WC of the first position, and the height of the second position in the world coordinate system WC is the world position of the second position. It is a position in the Z-axis direction in the coordinate system WC. When the upper surface of the work table TB is not parallel to the XY plane in the world coordinate system WC, the difference between the height in the world coordinate system WC at the first position and the height in the world coordinate system WC at the second position is within a predetermined range. By bringing the first position and the second position close to each other so as to be within the range (approximately considered to be the same), the three-dimensional measurement apparatus 30 can perform the same processing as described below.

  In the second state, the three-dimensional measurement apparatus 30 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA. The second state is a state in which the position of the predetermined figure projected within the projection range PA and the second position are coincident within the projection range PA. The three-dimensional measuring apparatus 30 acquires a captured image captured by the first imaging unit 11 by the stereo imaging from the first imaging unit 11 as a third captured image. In addition, the three-dimensional measurement apparatus 30 acquires a captured image captured by the second imaging unit 12 by the stereo imaging from the second imaging unit 12 as a fourth captured image.

  The three-dimensional measurement apparatus 30 detects feature points from the acquired first captured image and second captured image. A feature point is a characteristic point in a captured image. The three-dimensional measuring apparatus 30 may be configured to detect feature points from the first captured image and the second captured image using existing algorithms such as SURF (Speeded Up Robust Features) and SIFT (Scale-Invariant Feature Transform). The feature point may be detected from the first captured image and the second captured image by a method to be developed. Specifically, the three-dimensional measurement apparatus 30 detects a first feature point that is one or more feature points in the first captured image based on the acquired first captured image. Further, the three-dimensional measurement device 30 detects a second feature point that is one or more feature points in the second captured image based on the acquired second captured image.

  The three-dimensional measurement apparatus 30 associates the detected first feature point with the detected second feature point based on the first captured image to the fourth captured image. Then, the three-dimensional measurement device 30 performs three-dimensional measurement based on the associated first feature point and second feature point, and the first captured image and the second captured image. Thereby, the three-dimensional measurement apparatus 30 calculates the position and orientation of the object O.

  That is, the three-dimensional measuring apparatus 30 has a predetermined figure at the first position and the second position in the imaging range CA that includes the object O and can be imaged by the first imaging unit 11 and the second imaging unit 12. A first captured image in which an imaging range CA including a predetermined figure projected by the projection unit 5 and projected on the first position is captured by the first imaging unit 11, and the imaging range CA is captured by the second imaging unit 12. The second imaged image, the third imaged image obtained by imaging the imaging range CA including the predetermined graphic projected at the second position by the first imaging unit 11, and the imaging range CA are imaged by the second imaging unit 12. After associating the first feature point detected from the first captured image with the second feature point detected from the second captured image based on the fourth captured image, the first imaging unit 11 and the second 1st imaging imaged by 2 imaging parts 12 Performing three-dimensional measurement based on the image and the second image. Thereby, even if the target object O is a glossy object or a semitransparent object, the three-dimensional measurement apparatus 30 can suppress an error due to the optical characteristics of the target object O and can accurately perform the three-dimensional measurement. . Below, the process in which the three-dimensional measurement apparatus 30 performs three-dimensional measurement is demonstrated in detail.

<Hardware configuration of 3D measuring device>
Hereinafter, the hardware configuration of the three-dimensional measurement apparatus 30 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of a hardware configuration of the three-dimensional measurement apparatus 30.

  The three-dimensional measurement apparatus 30 includes, for example, a CPU (Central Processing Unit) 31, a storage unit 32, an input reception unit 33, a communication unit 34, and a display unit 35. These components are connected to each other via a bus Bus so that they can communicate with each other. In addition, the three-dimensional measurement device 30 communicates with each of the projection unit 5 and the imaging unit 10 via the communication unit 34.

The CPU 31 executes various programs stored in the storage unit 32.
The storage unit 32 includes, for example, a hard disk drive (HDD), a solid state drive (SSD), an electrically erasable programmable read-only memory (EEPROM), a read-only memory (ROM), and a random access memory (RAM). Note that the storage unit 32 may be an external storage device connected by a digital input / output port such as a USB instead of the one built in the three-dimensional measurement device 30. The storage unit 32 stores various information processed by the three-dimensional measurement apparatus 30, various images, various programs, and the like.

The input receiving unit 33 is, for example, a touch panel configured integrally with the display unit 35. The input receiving unit 33 may be a keyboard, a mouse, a touch pad, or other input device.
The communication unit 34 includes, for example, a digital input / output port such as USB, an Ethernet (registered trademark) port, and the like.
The display unit 35 is, for example, a liquid crystal display panel or an organic EL (ElectroLuminescence) display panel.

<Functional configuration of 3D measuring device>
Hereinafter, the functional configuration of the three-dimensional measurement apparatus 30 will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of a functional configuration of the three-dimensional measurement apparatus 30.

  The three-dimensional measurement apparatus 30 includes a storage unit 32 and a control unit 36.

  The control unit 36 controls the entire three-dimensional measurement device 30. The control unit 36 includes a projection control unit 361, an imaging control unit 363, an image acquisition unit 365, a feature point detection unit 367, a correspondence processing unit 369, a three-dimensional point group generation unit 375, and a position and orientation calculation unit. 377. These functional units included in the control unit 36 are realized, for example, when the CPU 31 executes various programs stored in the storage unit 32. Further, some or all of the functional units may be hardware functional units such as LSI (Large Scale Integration) and ASIC (Application Specific Integrated Circuit).

The projection control unit 361 causes the projection unit 5 to project a projection image onto the projection range PA.
The imaging control unit 363 causes the imaging unit 10 to perform stereo imaging of the imaging range CA. Specifically, the imaging control unit 363 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA.

  The image acquisition unit 365 acquires the captured image captured by the imaging unit 10 from the imaging unit 10. Specifically, the image acquisition unit 365 acquires the captured image captured by the first imaging unit 11 from the first imaging unit 11. Further, the image acquisition unit 365 acquires the captured image captured by the second imaging unit 12 from the second imaging unit 12.

The feature point detection unit 367 detects one or more feature points in the captured image based on the captured image acquired by the image acquisition unit 365.
The association processing unit 369 includes one or more feature points detected by the feature point detection unit 367 based on a certain captured image and one or more feature points detected by the feature point detection unit 367 based on another captured image different from the captured image. A process of associating with feature points of

The three-dimensional point cloud generation unit 375 has a result of processing by the association processing unit 369, a captured image acquired by the image acquisition unit 365 from the first imaging unit 11, and an image acquisition unit 365 acquired from the second imaging unit 12. Based on the captured images, a three-dimensional point group of the object O included in these captured images is generated.
The position / orientation calculation unit 377 calculates the position and orientation of the object O in the world coordinate system WC based on the three-dimensional point group generated by the three-dimensional point group generation unit 375.

<Process for 3D measurement by 3D measurement device>
Hereinafter, with reference to FIG. 5, the process in which the three-dimensional measurement apparatus 30 performs three-dimensional measurement will be described. FIG. 5 is a flowchart illustrating an example of a process flow in which the three-dimensional measurement apparatus 30 performs three-dimensional measurement.

  The projection control unit 361 causes the projection unit 5 to project a projection image including a predetermined graphic onto the projection range PA (Step S110). Specifically, the projection control unit 361 causes the predetermined position of the predetermined graphic projected in the projection range PA to coincide with the first position predetermined in the projection range PA within the projection range PA. The projection image is projected on the projection unit 5 to the projection range PA.

  Next, the imaging control unit 363 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA (step S120). Next, the image acquisition unit 365 acquires the captured image captured by the first imaging unit 11 as the first captured image from the first imaging unit 11, and the captured image captured by the second imaging unit 12 as the second captured image. Obtained from the second imaging unit 12 (step S130).

  Next, the projection control unit 361 causes the projection unit 5 to project a projection image including a predetermined graphic onto the projection range PA (step S140). Specifically, the projection control unit 361 causes the predetermined position of the predetermined figure projected in the projection range PA to coincide with the second position predetermined in the projection range PA in the projection range PA. The projection image is projected on the projection unit 5 to the projection range PA.

  The difference between the projection image projected on the projection range PA in step S110 and the projection image projected on the projection range PA in step S140 is the position of the predetermined figure in the projection image. Here, a case where the predetermined graphic is included in a part of the projection image and the shape of the predetermined graphic is a triangular graphic will be described as an example. When the projection image projected on the projection range PA in step S110 and the projection image projected on the projection range PA in step S140 are compared, the predetermined figure is within the projection range PA from the first position in the projection range PA. It looks like it has moved to the second position. An example of a method for realizing this is to project one of two projection images having different predetermined graphic positions in the image onto the projection range PA in step S110 and project the other to the projection unit 5 in step S140. Is projected onto the projection range PA. The three-dimensional measurement device 30 may be configured to realize the difference by other methods such as a method of changing the direction of the optical axis of the projection unit 5.

  After the process of step S140 is performed, the imaging control unit 363 causes the first imaging unit 11 and the second imaging unit 12 to perform stereo imaging of the imaging range CA (step S150). Next, the image acquisition unit 365 acquires the captured image captured by the first imaging unit 11 as the third captured image from the first imaging unit 11, and the captured image captured by the second imaging unit 12 as the fourth captured image. Obtained from the second imaging unit 12 (step S160).

  Next, the feature point detection unit 367 detects a first feature point that is one or more feature points in the first captured image based on the first captured image acquired by the image acquisition unit 365 in step S130. Based on the second captured image acquired by the image acquisition unit 365 in S130, a second feature point that is one or more feature points in the second captured image is detected (step S170). Here, the first feature point detected from the first captured image and the second feature point detected from the second captured image will be described with reference to FIGS. 6 and 7.

  FIG. 6 is a diagram illustrating a first captured image before the first feature point is detected and a second captured image before the second feature point is detected. The first captured image P1 illustrated in FIG. 6 is an example of the first captured image before the first feature point is detected. Each of the points in the first captured image P1 indicates a dot having a luminance value of “255”. In addition, the portion other than the point in the first captured image P1 is a portion occupied by dots having a luminance value of “0”. Further, the second captured image P2 illustrated in FIG. 6 is an example of the second captured image before the second feature point is detected. Each of the points in the second captured image P2 indicates a dot having a luminance value of “255”. In addition, the part other than the point in the second captured image P2 is a part occupied by dots having a luminance value of “0”. Here, in the example shown in FIG. 6, in order to avoid the figure from becoming complicated, the number of dots having a luminance value of “255” in each of the first captured image P1 and the second captured image P2 is set. The number of dots is smaller than the number of “0” dots.

  As shown in FIG. 7, the feature point detection unit 367 detects the first feature point from the first captured image P1, and detects the second feature point from the second captured image P2. FIG. 7 is a diagram illustrating a first captured image after the first feature point is detected and a second captured image after the second feature point is detected. The first captured image P11 illustrated in FIG. 7 is an example of the first captured image after the first feature point is detected. Each of the hatched points in the first captured image P11 is a first feature point detected by the feature point detection unit 367 from the first captured image P1. The second captured image P21 illustrated in FIG. 7 is an example of the second captured image after the second feature point is detected. Each of the hatched points in the second captured image P21 is a second feature point detected by the feature point detection unit 367 from the second captured image P2.

  After the process of step S170 is performed, the association processing unit 369 performs a first association process (step S180). Here, the first association process in step S180 will be described. In the first association processing, the association processing unit 369 is, for example, one or more second feature points that are candidates for the second feature point corresponding to the first feature point and satisfy the epipolar geometry for each first feature point. Detect feature point candidates. Hereinafter, for convenience of explanation, the second feature point candidates will be referred to as second feature point candidates. The method by which the association processing unit 369 detects the second feature point candidate may be a known method or a method to be developed in the future. The association processing unit 369 associates the first feature point with one or more second feature point candidates corresponding to the first feature point for each of the first feature points.

  In addition, when there are two or more second feature point candidates corresponding to a certain first feature point, the association processing unit 369 uses a world based on the first feature point and each of the two or more second feature point candidates. A position in the Z-axis direction in the coordinate system WC is calculated. The method for calculating the position is, for example, a triangulation method, but is not limited thereto, and other methods may be used. Then, the association processing unit 369 includes a first feature point, a second feature point candidate among the two or more second feature point candidates, the first feature point, and the second feature point candidate. The position in the Z-axis direction calculated based on this is associated.

  After the process of step S180 is performed, the association processing unit 369 starts from the position on the first captured image of the predetermined graphic included in the first captured image based on the first captured image and the third captured image. The first vector representing the trajectory up to the position on the third captured image of the predetermined graphic included in the third captured image is calculated as an optical flow (step S190). Here, the process of step S190 will be described.

  As described above, the position (that is, the first position) of the predetermined graphic projected on the projection range PA in step S110 and the projection within the projection range PA of the predetermined graphic projected on the projection range PA in step S140. The position (that is, the second position) is a different position. Therefore, the first captured image in which the imaging range CA is captured by the first imaging unit 11 in the first state is compared with the third captured image in which the imaging range CA is captured by the first imaging unit 11 in the second state. In this case, the predetermined graphic included in these captured images appears to have moved from the first position in the projection range PA included in the captured image to the second position in the projection range PA included in the captured image.

  Therefore, the association processing unit 369 detects the position in the imaging unit coordinate system CC at a predetermined position of the predetermined image included in the first captured image as the first detection position. In addition, the association processing unit 369 detects a third detection position that is a position in the imaging unit coordinate system CC at a predetermined position of a predetermined image included in the third captured image. The correspondence processing unit 369 represents a first difference representing a difference between the first detection position in the imaging unit coordinate system CC and the third detection position in the imaging unit coordinate system CC based on the detected first detection position and third detection position. The vector is calculated as an optical flow.

  The magnitude of such an optical flow is the first position in the projection range PA when the positions of the first imaging unit 11 and the second imaging unit 12 in the world coordinate system WC are fixed as shown in FIG. It changes according to the change of the height in the world coordinate system WC. By using the relationship between the magnitude of the optical flow and the height, the association processing unit 369 causes the second feature point corresponding to the first feature point for each first feature point in step S210 to be described later. Can be identified from two or more second feature point candidates corresponding to the first feature point. Hereinafter, for convenience of description, the height will be referred to as a target height.

  Here, the magnitude of the optical flow is represented by the norm of the optical flow in this example. Instead of this, the size of the optical flow may be represented by another value based on the optical flow.

  In step S190, the association processing unit 369 may be configured to calculate an optical flow based on the second captured image and the fourth captured image. In this case, the association processing unit 369 detects a position in the imaging unit coordinate system CC at a predetermined position of the predetermined image included in the second captured image as the second detection position. In addition, the association processing unit 369 detects a fourth detection position that is a position in the imaging unit coordinate system CC of a predetermined position of the predetermined image included in the fourth captured image. The association processing unit 369 calculates, as an optical flow, a second vector representing a difference between the second detection position and the fourth detection position in the imaging unit coordinate system CC based on the detected second detection position and fourth detection position. To do.

  Further, in step S190, the association processing unit 369 may calculate an optical flow based on the first captured image, the second captured image, the third captured image, and the fourth captured image. Good. In this case, the association processing unit 369 calculates the first vector and the second vector. Then, the association processing unit 369 calculates an average of the first vector and the second vector as an optical flow.

  After the process of step S190 is performed, the association processing unit 369 reads the correspondence information stored in advance in the storage unit 32 from the storage unit 32 (step S200). The correspondence information is information in which the target height is associated with the size of the optical flow corresponding to each target height. For example, when it is desired to specify the size of the optical flow corresponding to a certain target height, the association processing unit 369 can specify the size of the optical flow corresponding to the target height based on the correspondence information. The process in which the three-dimensional measuring device 30 generates correspondence information will be described later.

  Next, the correspondence processing unit 369 performs the second correspondence processing for each of the first feature points based on the optical flow calculated in step S190 and the correspondence information read from the storage unit 32 in step S200. This is performed (step S210). In the second association processing for a certain first feature point, the second feature point corresponding to the first feature point is identified from the second feature point candidates associated with the first feature point and identified. This is a process of associating the second feature point with the first feature point. However, in the second association processing, the association processing unit 369 corresponds the second feature point candidate to the first feature point when there is one second feature point candidate associated with a certain first feature point. The second feature point is specified. Here, the process of step S210 will be described.

  As described above, when the positions of the first imaging unit 11 and the second imaging unit 12 in the world coordinate system WC are fixed, the heights of the first position and the second position in the world coordinate system WC within the projection range PA are determined. It changes according to change. That is, when the correspondence information is derived in advance, the association processing unit 369 determines the predetermined range within the projection range PA included in the first captured image based on the correspondence information and the optical flow calculated in step S190. It shows that the height in the world coordinate system WC of the surface on which the figure is located can be specified. The height is a position in the Z-axis direction in the world coordinate system WC. Therefore, the association processing unit 369, based on the optical flow calculated in step S190 and the correspondence information read from the storage unit 32 in step S200, the world coordinate system of the surface (for example, the upper surface of the object O). Specify the height in WC. The association processing unit 369 performs the following processing for each first feature point included in the plane included in the first captured image. In the process for a certain first feature point, the association processing unit 369 triangulates the three-dimensional position based on the first feature point and each of the second feature point candidates associated with the first feature point. Calculated by the method. The association processing unit 369 specifies a three-dimensional position in the calculated three-dimensional position where the height in the world coordinate system WC substantially matches (closest) to the height of the surface in the world coordinate system WC. The association processing unit 369 specifies the second feature point candidate used to calculate the specified three-dimensional position as the second feature point corresponding to the first feature point used to calculate the three-dimensional position. To do. The association processing unit 369 associates the identified first feature point with the second feature point. The association processing unit 369 performs such second association processing for each first feature point. Thereby, the correspondence processing unit 369 can associate the first feature point and the second feature point with high accuracy.

  After the process of step S210 is performed, the three-dimensional point group generation unit 375 generates a first feature point and a second feature point that are associated with each other as a result of the second association process in step S210. A three-dimensional point group of the object O included in one captured image is generated (step S220). The method for generating the three-dimensional point group may be a known method or a method developed from now on. Note that the processing in step S220 is an example of three-dimensional measurement. Next, the position / orientation calculation unit 377 calculates the position and orientation of the object O in the world coordinate system WC based on the three-dimensional point group of the object O generated in step S220 (step S230), and performs processing. finish. The method by which the position / orientation calculation unit 377 calculates the position and orientation of the object O based on the three-dimensional point group may be a known method or a method that will be developed in the future.

  As described above, the three-dimensional measuring device 30 is predetermined at the first position and the second position in the imaging range CA that includes the object O and can be imaged by the first imaging unit 11 and the second imaging unit 12. A first captured image in which an imaging range CA including a predetermined graphic projected on the first position is projected by the projection unit 5 and captured by the first imaging unit 11, and the imaging range CA is captured by the second imaging unit 12. The second captured image, the third captured image in which the first imaging unit 11 captures the imaging range CA including the predetermined graphic projected on the second position, and the second imaging unit 12 captures the imaging range CA. After associating the first feature point detected from the first captured image with the second feature point detected from the second captured image based on the captured fourth captured image, the first imaging unit 11 And the first image picked up by the second image pickup unit 12 Performing three-dimensional measurement based on the image and the second image. Thereby, even if the target object O is a glossy object or a semitransparent object, the three-dimensional measurement apparatus 30 can suppress an error due to the optical characteristics of the target object O and can accurately perform the three-dimensional measurement. .

<Process in which 3D measuring device generates correspondence information>
Hereinafter, with reference to FIGS. 8 to 11, a process in which the three-dimensional measurement apparatus 30 generates correspondence information will be described. FIG. 8 is a flowchart illustrating an example of a process flow in which the three-dimensional measurement apparatus 30 generates correspondence information.

  The association processing unit 369 reads the target height information stored in advance in the storage unit 32 from the storage unit 32. The corresponding height information is information indicating a plurality of target heights selected by the user. In the following, for convenience of explanation, the target height is represented by the value of Z. That is, Z = 0 represents that the target height is a reference height. Z = n represents that the target height is higher than the reference height by n times the predetermined height. Here, n may be any number as long as it is a real number excluding 0.

  In the following, a case where five target heights Z = 0, 2, 4, 6, and 8 are selected by the user will be described as an example. That is, the target height information indicates each of the five target heights Z = 0, 2, 4, 6, and 8.

  The association processing unit 369 repeatedly performs the processing of step S320 to step S410 for each target height indicated by the target height information read from the storage unit 32 (step S310). That is, the association processing unit 369 repeatedly performs the process for each of the five target heights of Z = 0, 2, 4, 6, and 8.

  The correspondence processing unit 369 waits until the user matches the height in the world coordinate system WC on the upper surface of the work table TB with the target height based on the target height selected in step S310 (step S320). For example, when the association processing unit 369 receives an operation indicating that the operation of matching the height of the upper surface of the work table TB in the world coordinate system WC with the target height is completed from the user, the projection control unit 361 performs step S330. Transition to. When the work table TB includes a mechanism unit that changes the height of the upper surface of the work table TB in the world coordinate system WC, the association processing unit 369 controls the mechanism unit so that the work is performed. It may be configured to be performed. In this case, the projection control unit 361 transitions to step S330 when the association processing unit 369 ends the process.

  In step S330, the projection control unit 361 matches the predetermined position of the predetermined figure projected in the projection range PA with the first position determined in the projection range PA within the projection range PA. In this manner, the projection image is projected onto the projection range 5 to the projection range PA (step S330).

  Next, the imaging control unit 363 causes at least one of the first imaging unit 11 and the second imaging unit 12 to image the imaging range CA (step S340). Hereinafter, as an example, a case where the imaging control unit 363 causes the first imaging unit 11 to image the imaging range CA in step S340 will be described. Next, the image acquisition unit 365 acquires the captured image captured by the first imaging unit 11 from the first imaging unit 11 (step S350).

  Next, the projection control unit 361 performs projection so that the predetermined position of the predetermined figure projected in the projection range PA matches the second position predetermined in the projection range PA within the projection range PA. The image is projected on the projection unit 5 to the projection range PA (step S370).

  Next, the imaging control unit 363 causes the same imaging unit (in this example, the first imaging unit 11) as the imaging unit that captured the imaging range CA in step S340 to image the imaging range CA (step S380). Next, the image acquisition unit 365 acquires the captured image captured by the first imaging unit 11 from the first imaging unit 11 (step S390).

  Next, the association processing unit 369 is included in the captured image acquired by the image acquisition unit 365 in step S390 from the position on the captured image of the predetermined graphic included in the captured image acquired by the image acquisition unit 365 in step S350. A vector representing the trajectory of the predetermined figure to the position on the second captured image is calculated as an optical flow (step S410). Then, the association processing unit 369 associates the target height selected in Step S310 with the calculated optical flow.

  Here, with reference to FIG. 9 to FIG. 12, processing in steps S <b> 310 to S <b> 410 will be described. FIG. 9 shows that when the target height is Z = 0, the predetermined position of the predetermined figure projected within the projection range PA is the first position determined within the projection range PA and within the projection range PA. It is a figure which shows an example of a mode that it corresponds in. FIG. 10 shows that when the target height is Z = 2, the predetermined position of the predetermined figure projected within the projection range PA is the first position determined within the projection range PA and within the projection range PA. It is a figure which shows an example of a mode that it corresponds in. FIG. 11 shows that when the target height is Z = 4, the predetermined position of the predetermined figure projected within the projection range PA is the first position determined within the projection range PA and within the projection range PA. It is a figure which shows an example of a mode that it corresponds in. 9 to 11, the predetermined figure is represented by an arrow AR in order to clearly show the predetermined figure included in the projection image.

  For example, the three-dimensional measuring apparatus 30 performs the first processing of Step S320 to Step S410 in a state where the height of the upper surface of the work table TB in the world coordinate system WC matches the height shown in FIG. The three-dimensional measuring apparatus 30 performs the second processing from step S320 to step S410 in a state where the height of the upper surface of the work table TB in the world coordinate system WC matches the height shown in FIG. Then, the three-dimensional measurement apparatus 30 performs the third processing in steps S320 to S410 in a state where the height of the upper surface of the work table TB in the world coordinate system WC matches the height shown in FIG. By comparing each of FIGS. 9 to 11, as the height of the upper surface of the work table TB in the world coordinate system WC increases, each of the first imaging unit 11 and the second imaging unit 12 and the first position (and The distance to the second position) is increased. As a result, as the height of the upper surface of the work table TB in the world coordinate system WC increases, from the position on the captured image of the predetermined graphic included in the captured image acquired by the image acquisition unit 365 in step S350, the image is displayed in step S390. The distance to the position on the 2nd captured image of the predetermined figure contained in the captured image which the acquisition part 365 acquired becomes long. As a result, the higher the height, the larger the optical flow.

  After the repetitive processing of Steps S310 to S410 is performed, the association processing unit 369, for example, based on each of the optical flows calculated by the repetitive processing, changes in the size of the optical flow and each optical flow. A function representing the relationship with the change in the target height associated with is generated (derived) by linear interpolation as correspondence information (step S420). Note that the association processing unit 369 may be configured to generate other information such as a table representing the relationship as correspondence information instead of the function. Further, the association processing unit 369 may be configured to generate correspondence information using another statistical method instead of linear interpolation. Here, the function will be described with reference to FIG.

  FIG. 12 is a diagram illustrating an example of correspondence information. The vertical axis of the graph shown in FIG. 12 indicates the magnitude of the optical flow. The horizontal axis of the graph indicates the value of the target height Z. The points plotted in the graph are points representing the size of the optical flow corresponding to each of the target heights that can be selected in step S310. Based on these points, the association processing unit 369 performs linear interpolation to obtain a function that represents the relationship between the change in the magnitude of the optical flow and the change in the target height associated with each optical flow. Generate as correspondence information. A line connecting each of the points plotted on the graph is a line representing the function.

After generating the correspondence information, the association processing unit 369 stores the generated correspondence information in the storage unit 32. Then, the association processing unit 369 ends the process.
As described above, the three-dimensional measurement apparatus 30 generates correspondence information through the processes in steps S310 to S420. Thereby, the three-dimensional measurement apparatus 30 can perform the process of the flowchart shown in FIG. 5 with high accuracy.

<Modification of Embodiment>
Hereinafter, a modification of the embodiment will be described with reference to FIG. In addition, in the modification of embodiment, the same code | symbol is attached | subjected with respect to the structure part similar to embodiment, and description is abbreviate | omitted.

  FIG. 13 is a diagram illustrating an example of a configuration of a robot system 2 according to a modification of the embodiment. The robot system 2 includes an imaging unit 10, a robot 20, and a robot control device 50. Further, the robot control device 50 includes a three-dimensional measurement device 30. More specifically, the robot control device 50 includes functional units included in the three-dimensional measurement device 30.

  The robot 20 is a single-arm robot including an arm A and a support base B that supports the arm A. The single arm robot is a robot having one arm (arm) like the arm A in this example. The robot 20 may be a multi-arm robot instead of the single-arm robot. The multi-arm robot is a robot having two or more arms (for example, two or more arms A). Of the multi-arm robot, a robot having two arms is also referred to as a double-arm robot. That is, the robot 20 may be a double-arm robot having two arms or a multi-arm robot having three or more arms (for example, three or more arms A). The robot 20 may be another robot such as a SCARA robot or a cylindrical robot.

The arm A includes an end effector E and a manipulator M.
In this example, the end effector E is an end effector including a finger portion that can grip an object. The end effector E may be an end effector capable of lifting an object by air suction, a magnetic force, a jig or the like, or another end effector, instead of the end effector including the finger portion.

  The end effector E is communicably connected to the robot controller 50 via a cable. Thereby, the end effector E performs an operation based on the control signal acquired from the robot control device 50. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, the end effector E may be configured to be connected to the robot control device 50 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  The manipulator M includes six joints. Each of the six joints includes an actuator (not shown). That is, the arm A including the manipulator M is a 6-axis vertical articulated arm. The arm A performs an operation with six degrees of freedom by a coordinated operation by the support base B, the end effector E, the manipulator M, and the actuators of each of the six joints included in the manipulator M. The arm A may be configured to operate with a degree of freedom of 5 axes or less, or may be configured to operate with a degree of freedom of 7 axes or more.

  Each of the six actuators (provided at the joints) included in the manipulator M is communicably connected to the robot controller 50 via a cable. Accordingly, the actuator operates the manipulator M based on the control signal acquired from the robot control device 50. Note that wired communication via a cable is performed according to standards such as Ethernet (registered trademark) and USB, for example. Further, a part or all of the six actuators included in the manipulator M may be connected to the robot control device 50 by wireless communication performed according to a communication standard such as Wi-Fi (registered trademark).

  In this example, the robot control device 50 is a device that controls the robot. The robot control device 50 generates a control signal based on an operation program input in advance. The robot control device 50 transmits the generated control signal to the robot 20 to cause the robot 20 to perform a predetermined operation. In this example, the predetermined work is a work in which the robot 20 operates the arm A to hold the object O placed on the upper surface of the work table TB and to place the grasped object O in a supply area (not shown). It is. The predetermined work may be another work instead of this.

  When the robot controller 50 causes the robot 20 to perform a predetermined operation, the robot control device 50 is the first position in the imaging range CA that includes the object O and can be imaged by the first imaging unit 11 and the second imaging unit 12. And the predetermined figure is projected by the projection part 5 in the 2nd position, and the 1st picked-up image in which the imaging range CA containing the predetermined figure projected on the 1st position was imaged by the 1st imaging part 11, and the said imaging range CA are A second captured image captured by the second image capturing unit 12, a third captured image captured by the first image capturing unit 11 including the predetermined image projected on the second position, and the captured image range CA. After associating the first feature point detected from the first captured image with the second feature point detected from the second captured image based on the fourth captured image captured by the second imaging unit 12 , First imaging unit 11 and second imaging Performing three-dimensional measurement based on the first image and the second captured image captured by 12. With this three-dimensional measurement, the robot control device 50 generates a three-dimensional point group of the object O. The robot control device 50 calculates the position and orientation of the object O based on the generated three-dimensional point group. Then, the robot control device 50 causes the robot 20 to perform a predetermined operation based on the calculated position and orientation of the object O. The process for the robot controller 50 to perform the three-dimensional measurement of the object O is the same as the process when the world coordinate system WC is replaced with the robot coordinate system RC in the description of the process of the flowchart shown in FIG. Since it is a process, explanation is omitted.

  Here, the robot control device 50 generates the aforementioned correspondence information in advance before causing the robot 20 to perform a predetermined operation. The process in which the robot control device 50 generates the correspondence information is the same as the process when the world coordinate system WC is replaced with the robot coordinate system RC in the description of the process in the flowchart of FIG.

  As described above, the robot control device 50 includes the three-dimensional measurement device 30 described in the embodiment, and performs predetermined work on the robot 20 based on the position and orientation of the object O calculated by the three-dimensional measurement device 30. Make it. Thereby, the robot control apparatus 50 can make the robot 20 perform a predetermined operation with high accuracy.

  In the robot system 2, the robot control device 50 and the three-dimensional measurement device 30 may be separate. In this case, the robot control device 50 acquires the position and orientation of the object O calculated by the three-dimensional measurement device 30 from the three-dimensional measurement device 30, and determines a predetermined value for the robot 20 based on the acquired position and posture of the object O. Let's do the work.

  As described above, the three-dimensional measurement apparatus 30 is an imaging range (an imaging range CA in this example) including an object (in this example, the object O), and a first imaging unit (in this example, the first imaging unit CA). Predetermined figures are projected by the projection unit (in this example, the projection unit 5) at the first position and the second position in the imaging range that can be imaged by the imaging unit 11) and the second imaging unit (in this example, the second imaging unit 12). A first captured image in which an imaging range including a predetermined figure projected and projected to the first position is captured by the first imaging unit; a second captured image in which the imaging range is captured by the second imaging unit; Based on a third captured image in which an imaging range including a predetermined graphic projected at two positions is captured by the first imaging unit and a fourth captured image in which the imaging range is captured by the second imaging unit. First feature detected from captured image After that associates the point, and a second feature point detected from the second image, performs three-dimensional measurement based on the image captured by the first imaging unit and the second imaging unit. Thereby, even if the target object is a glossy object or a translucent object, the three-dimensional measurement apparatus 30 can suppress an error due to the optical characteristics of the target object and accurately perform the three-dimensional measurement.

  The three-dimensional measuring device 30 also includes a first vector representing a difference between the position of the predetermined graphic in the first captured image and the position of the predetermined graphic in the third captured image, and the position of the predetermined graphic in the second captured image. The first feature point and the second feature point are associated with each other based on at least one of the position of the predetermined figure in the fourth captured image and the second vector representing the difference. Thereby, the three-dimensional measuring apparatus 30 can suppress the error due to the optical characteristic of the object based on at least one of the first vector and the second vector, and can accurately perform the three-dimensional measurement.

  Further, the three-dimensional measurement apparatus 30 includes a first vector and a change in distance from the position where the first imaging unit is provided and the position where the second imaging unit is provided to the first position and the second position. The first feature point and the second feature point are associated with each other based on correspondence information indicating correspondence between the change in value based on at least one of the second vector and the change in value. Thereby, the three-dimensional measuring apparatus 30 can suppress the error due to the optical characteristics of the object based on the correspondence information, and can accurately perform the three-dimensional measurement.

  In addition, the three-dimensional measuring device 30 generates a three-dimensional point group that represents the three-dimensional shape of the object based on the first feature point and the second feature point that are associated with each other. Thereby, the three-dimensional measuring apparatus 30 can accurately detect the position and orientation of the object based on the generated three-dimensional point group.

  In addition, the three-dimensional measurement device 30 calculates the position and orientation of the object based on the generated three-dimensional point group. Thereby, the three-dimensional measurement apparatus 30 can output the position and orientation of the target object calculated with high accuracy to another apparatus (in this example, the robot control apparatus 50).

  Further, the robot 20 performs a predetermined work (a predetermined work in this example) based on the result of the three-dimensional measurement performed by the three-dimensional measurement device 30. Thereby, the robot 20 can perform a predetermined operation with high accuracy.

  Further, the robot control device 50 causes the robot 20 to perform a predetermined operation based on the result of the 3D measurement performed by the 3D measurement device 30. As a result, the robot control device 50 can cause the robot to perform a predetermined operation with high accuracy.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and changes, substitutions, deletions, and the like are possible without departing from the gist of the present invention. May be.

  Further, a program for realizing the function of an arbitrary component in the above-described apparatuses (for example, the three-dimensional measuring apparatus 30 and the robot control apparatus 50) is recorded on a computer-readable recording medium, and the program is stored in the computer. The program may be read by the system and executed. Here, the “computer system” includes hardware such as an OS (Operating System) and peripheral devices. The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD (Compact Disk) -ROM, or a storage device such as a hard disk built in the computer system. . Furthermore, “computer-readable recording medium” means a volatile memory (RAM) inside a computer system that becomes a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line. In addition, those holding programs for a certain period of time are also included.

In addition, the above program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the above program may be for realizing a part of the functions described above. Further, the program may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

DESCRIPTION OF SYMBOLS 1 ... Three-dimensional measuring system, 2 ... Robot system, 10 ... Imaging part, 11 ... 1st imaging part, 12 ... 2nd imaging part, 20 ... Robot, 30 ... Three-dimensional measuring apparatus, 31 ... CPU, 32 ... Memory | storage part , 33 ... Input receiving unit, 34 ... Communication unit, 35 ... Display unit, 36 ... Control unit, 50 ... Robot control device, 361 ... Projection control unit, 363 ... Imaging control unit, 365 ... Image acquisition unit, 367 ... Feature inspection Output unit, 369... Corresponding processing unit, 375... Three-dimensional point group generation unit, 377.

Claims (8)

A predetermined figure is projected by the projection unit to the first position and the second position in the imaging range that includes the target object and can be imaged by the first imaging unit and the second imaging unit, and is projected to the first position. In addition, a first captured image in which the imaging range including the predetermined figure is captured by the first imaging unit, a second captured image in which the imaging range is captured by the second imaging unit, and the second position are projected. Based on a third captured image obtained by imaging the imaging range including the predetermined figure by the first imaging unit and a fourth captured image obtained by imaging the imaging range by the second imaging unit. After associating the first feature point detected from one captured image with the second feature point detected from the second captured image, the image captured by the first image capturing unit and the second image capturing unit Based on 3D measurement,
Three-dimensional measuring device. A first vector representing a difference between a position of the predetermined graphic in the first captured image and a position of the predetermined graphic in the third captured image; a position of the predetermined graphic in the second captured image; Associating the first feature point with the second feature point based on at least one of the position of the predetermined figure in the captured image and the second vector representing the difference;
The three-dimensional measuring apparatus according to claim 1. Changes in distance from the position where the first imaging unit is provided and the position where the second imaging unit is provided to the first position and the second position, the first vector, and the second Correspondence information indicating a correspondence relationship with a change in value based on at least one of the vectors and associating the first feature point with the second feature point based on correspondence information stored in advance.
The three-dimensional measuring apparatus according to claim 2. Generating a three-dimensional point group representing a three-dimensional shape of the object based on the first feature point and the second feature point associated with each other;
The three-dimensional measuring apparatus according to any one of claims 1 to 3. Calculating the position and orientation of the object based on the generated three-dimensional point cloud;
The three-dimensional measuring apparatus according to claim 4. A predetermined operation is performed based on the position and orientation of the object calculated by the three-dimensional measurement apparatus according to claim 5.
robot. Making the robot according to claim 6 perform the predetermined work;
Robot control device. The projection unit;
The first imaging unit;
The second imaging unit;
A robot according to claim 6;
A robot controller according to claim 7;
A robot system comprising:

Images