JP4649559B2 - 3D object recognition apparatus, 3D object recognition program, and computer-readable recording medium on which the same is recorded - Google Patents

Description

  The present invention relates to a three-dimensional object recognition apparatus for recognizing a three-dimensional object having a known shape from features such as an outline in a two-dimensional image photographed by a camera or the like, a three-dimensional object recognition program, and a computer-readable computer on which the same is recorded. The present invention relates to a recording medium.

  In order to enable accurate operation of parts and the like by a robot arm in a production line, a three-dimensional object recognition device has been developed in recent years that recognizes a pile of parts individually and recognizes the position and orientation of each part. . This three-dimensional object recognition apparatus first extracts features such as edges of a three-dimensional object, ie, contours, from an image obtained by photographing the three-dimensional object with a camera from a predetermined direction, and the distance to the nearest edge for each pixel constituting the photographed image. Respectively. Next, the three-dimensional object recognition device projects the three-dimensional object on the captured image in various positions and postures, and calculates the coordinates of each point constituting the edge. Then, the three-dimensional object recognition apparatus evaluates the position and orientation based on an error obtained by comparing the two, and recognizes the position and orientation with the highest evaluation as the position and orientation of the three-dimensional object.

  However, each time the position and orientation of the three-dimensional object are changed, calculating the distance to the nearest edge for each pixel constituting the captured image requires a high-performance processing device and increases the cost. Therefore, in order to solve this problem, it has been proposed to create a distance map in advance (see, for example, Patent Document 1). This distance map is obtained by assigning the distance to the nearest edge to each pixel constituting the captured image as a pixel value. By referring to this distance map, the distance to the nearest edge is calculated one by one. This saves you time and effort.

Japanese Patent Laid-Open No. 11-025291

  However, the conventional three-dimensional object recognition apparatus has a problem that the robustness is poor due to the influence of hiding the recognition object. That is, in a state in which parts and the like are stacked, when viewed from the camera direction, a part of the three-dimensional object that is a recognition target may be hidden by another object. And when the closest edge from a certain pixel is hidden, the distance to the closest edge is calculated incorrectly, so the position and orientation of the three-dimensional object cannot be correctly evaluated, resulting in erroneous recognition. appear.

  In addition, since the three-dimensional recognition system according to the present invention only selects an optimum one by evaluating a plurality of predetermined positions and postures, an error occurs compared to the original position and posture of the three-dimensional object. In some cases, the recognition accuracy is poor.

  The present invention has been made in view of such problems, and in a three-dimensional object recognition apparatus that recognizes a three-dimensional object from features such as contours in a two-dimensional image, improves the robustness by eliminating the influence of hiding. And a means for optimizing the position and orientation to increase the recognition accuracy.

  In order to achieve the above object, a three-dimensional object recognition apparatus according to claim 1 of the present invention includes a camera that captures an image by photographing a three-dimensional object that is a recognition target from a predetermined direction, a position of the three-dimensional object, and While changing the posture, among the sampling points constituting the edge of the three-dimensional object, the sampling points visible from the camera are respectively projected onto the camera image, and the coordinates of each projection point and the direction of the edge at each projection point are respectively projected. Projection point coordinate calculation means for calculating, a lookup table storage for storing a lookup table in which the position and orientation of the three-dimensional object, the coordinates of each projection point and the direction of the edge at each projection point are stored in association with each other Based on the original image acquired by the means and the camera, a plurality of pyramid images in which the resolution of the original image is reduced at different ratios are created A ramid image creating means; an edge extracting means for extracting the edge of the three-dimensional object for the pyramid image having the lowest resolution; and the nearest edge among the extracted edges to each pixel constituting the pyramid image having the lowest resolution A distance map creating means for creating a distance map with a direction having a distance to and a direction of the nearest edge as a pixel value, and stored in the lookup table on the distance map with the direction Further, the projection point mapping means for mapping each projection point, the direction of the edge at each projection point, and the direction of the nearest edge of the pixel corresponding to each projection point in the directional distance map are respectively compared. For a set of projection points that are approximately the same, Calculating the sum of squares of the distance to the nearest edge of the pixel group to be performed, and evaluating the position and orientation of the three-dimensional object based on the calculation result, and the three-dimensional by the position and orientation evaluation unit Position and orientation optimization means for optimizing a position and orientation evaluated to be close to the actual position and orientation of the object so that the sum of squares is minimized.

  Further, in the three-dimensional object recognition apparatus according to claim 2, the edge extraction unit extracts an edge of the three-dimensional object with sub-pixel accuracy, and the position / orientation evaluation unit determines the distance to the nearest edge as The length of the perpendicular dropped from the pixel group to the edge of the sub-pixel accuracy is used.

  The three-dimensional object recognition apparatus according to claim 3 is capable of visually recognizing a computer from a sampling point constituting the edge of the three-dimensional object while changing the position and orientation of the three-dimensional object as a recognition target. Projection points on the camera image, projection point coordinate calculation means for calculating the coordinates of each projection point and the direction of the edge at each projection point, the position and orientation of the three-dimensional object, and the projection point Lookup table storage means for storing a lookup table in which coordinates and edge directions at each projection point are stored in association with each other, and based on the original image acquired by the camera, the resolution of the original image is reduced at different ratios. A pyramid image creating means for creating a plurality of pyramid images, and the pyramid image having the lowest resolution; Edge extraction means for extracting the edge of the original object, and each pixel constituting the pyramid image with the lowest resolution, the distance to the nearest edge among the extracted edges and the direction of the nearest edge as pixel values A directional distance map creating means for creating a directional distance map, a projected point mapping means for mapping each of the projected points stored in the lookup table on the directional distance map; and The direction of the edge at each projection point is compared with the direction of the closest edge of the pixel corresponding to each projection point in the directional distance map. Calculate the sum of squares of the distance to the nearest edge Position and orientation evaluation means for evaluating the position and orientation of the three-dimensional object based on the position, and the position and orientation evaluated by the position and orientation evaluation means as being close to the actual position and orientation of the three-dimensional object, It is made to function as a position / orientation optimizing means for optimizing so as to be minimized.

  A computer-readable recording medium on which the three-dimensional object recognition program according to claim 4 is recorded is the one on which the three-dimensional object recognition program according to claim 3 is recorded.

  According to the three-dimensional object recognition apparatus of the first aspect of the present invention, the position / orientation evaluation means calculates the sum of squares only for the projected points whose edge directions substantially coincide with the orientation distance map. Therefore, the edge direction does not match for a pixel in which a part of a three-dimensional object is hidden by another object when viewed from the camera direction and the distance to the nearest edge is erroneously calculated. And excluded from the target of calculating the sum of squares. Thereby, the so-called hiding effect can be reduced and the robustness can be improved. Further, since the position and orientation stored in the lookup table are evaluated using the pyramid image having the lowest resolution, the processing speed can be increased. In addition, since the position and orientation optimization unit further optimizes the position and orientation evaluated to be close to the actual position and orientation, the position and orientation recognition accuracy can be improved.

  In addition, according to the three-dimensional object recognition apparatus according to the second aspect, the edge extraction of the pyramid image is performed with sub-pixel accuracy, so that the position and orientation recognition accuracy can be improved.

  Further, according to the three-dimensional object recognition program according to the third aspect, the same effect as the three-dimensional object recognition apparatus according to the first aspect can be obtained.

  According to the computer-readable recording medium on which the three-dimensional object recognition program according to the fourth aspect is recorded, the same effect as that of the three-dimensional object recognition apparatus according to the first aspect can be obtained.

The schematic diagram which shows the structure of the three-dimensional object recognition apparatus 1 which concerns on the Example of this invention. 7 is a flowchart showing a flow of processing by a three-dimensional object recognition program 8. Explanatory drawing for demonstrating the pyramid image 15. FIG. It is explanatory drawing for demonstrating edge extraction, Comprising: The figure which shows the state which expanded some original images 16 to the pixel level. Explanatory drawing for demonstrating the creation method of a lookup table.

  FIG. 1 is a schematic diagram illustrating a configuration of a three-dimensional object recognition apparatus 1 according to the present embodiment. The three-dimensional object recognition apparatus 1 includes a three-dimensional object 3 as a recognition target placed on a work table 2, two cameras 4 that photograph the three-dimensional object 3 from different directions, and a three-dimensional object 3. A robot arm 5 for gripping and a computer 6 for controlling the operation of the robot arm 5 based on a captured image input from each camera 4 are provided.

  As shown in FIG. 1, the computer 6 includes an image memory 7 for storing image data taken by the camera 4, a hard disk 9 for storing a three-dimensional object recognition program 8, and a three-dimensional read from the hard disk 9. A RAM (Random Access Memory) 10 that temporarily stores the object recognition program 8, a CPU (Central Processing Unit) 11 that calculates the position and orientation of the three-dimensional object 3 according to the three-dimensional object recognition program 8, and the image memory 7 A display unit 12 for displaying the image data and the calculation result by the CPU 11, an operation unit 13 composed of a mouse, a keyboard, and the like, and a system bus 14 for connecting these units to each other. In the present embodiment, the three-dimensional object recognition program 8 is stored in the hard disk 9, but instead, it may be stored in a computer-readable recording medium (not shown) and read out from this recording medium. Is possible.

  Hereinafter, a processing procedure by the three-dimensional object recognition program 8 will be described. FIG. 2 is a flowchart showing the flow of processing by the three-dimensional object recognition program 8. First, when an original image obtained by photographing the three-dimensional object 3 is input from the camera 4, the CPU 11 creates a plurality of pyramid images based on the original image (S1) and stores them in the image memory 7 shown in FIG. To do. FIG. 3 is an explanatory diagram for explaining the pyramid image 15. The pyramid image 15 is obtained by reducing the resolution of the original image 16 at a predetermined ratio. For example, when an original image 16 in which n pixels are arranged in both vertical and horizontal directions is input, the CPU 11 receives a first pyramid image 15A in which n / 2 pixels are arranged in both vertical and horizontal directions, and n in both vertical and horizontal directions. A second pyramid image 15B in which / 4 pixels are arranged and a third pyramid image 15C in which n / 8 pixels are arranged in both the vertical and horizontal directions are created. In this embodiment, the three-stage pyramid image 15 is created. However, the number of stages can be appropriately changed according to the size of the input image.

  Next, as shown in FIG. 2, the CPU 11 extracts the edge of the three-dimensional object 3 for the third pyramid image 15C having the lowest resolution (S2). Here, as the edge extraction, edge extraction with pixel accuracy is performed. FIG. 4 is an explanatory diagram for explaining edge extraction, and shows a state in which a part of the original image 16 is enlarged to the pixel level. According to edge extraction with pixel accuracy, an edge is extracted as a collection of edge-constituting pixels 17 filled in black in the figure (hereinafter, this edge is referred to as “pixel edge 18”). In this embodiment, edge extraction is performed with pixel accuracy so as to prioritize processing speed. However, when high recognition accuracy is required, edge extraction with sub-pixel accuracy may be performed. According to edge extraction with subpixel accuracy, as shown by a straight line in FIG. 4, edges are extracted with accuracy equal to or smaller than the adjacent pixel interval D (hereinafter, this edge is referred to as “subpixel edge 19”).

  Next, as shown in FIG. 2, the CPU 11 creates a distance map with direction (S3) and stores it in the RAM 8 shown in FIG. Although a detailed distance map is not shown in the figure, the distance from the pixel to the nearest pixel edge 18 and the nearest pixel edge 18 to each pixel constituting the third pyramid image 15C subjected to edge extraction are shown. Are given as pixel values.

  Next, the CPU 11 maps the projection points stored in the lookup table stored in advance on the directional distance map (S4). This look-up table is created in advance by the CPU 11 and stored in the RAM 10 or the like according to the shape of the three-dimensional object 3, the position of the camera 4, or the like. As shown in FIG. 5, the creation method sets sampling points 21 on each edge 20 of the three-dimensional object 3, and determines whether each sampling point 21 is visible from the camera 4. Then, only the sampling point 21 determined to be visible from the camera 4 is projected onto the camera image 22, and the coordinates of the projection point 23 and the direction of the edge 24 at the projection point 23 are calculated. This operation is repeated while changing the position (three degrees of freedom) and posture (three degrees of freedom) of the three-dimensional object 3 sufficiently finely over the entire range that can be considered from the position of the camera 4 and the like. . A lookup table is created by storing the coordinates of the projection point 23 and the direction of the edge 20 at the projection point 23 in association with the position and orientation of the three-dimensional object 3. The CPU 11 sequentially arranges the projection points 23 stored in the lookup table on the distance map with direction based on the coordinates. In the distance map with direction, since the distance to the nearest pixel edge 18 is stored only for each pixel, when mapping the projection point 23, the coordinate value of the projection point 23 has a decimal part. The arrangement position of the projection point 23 may be determined by using bilinear interpolation.

  Next, the CPU 11 compares the direction of the edge 24 at each mapped projection point 23 with the direction of the nearest pixel edge 18 that the pixel corresponding to the projection point 23 has as a pixel value on the orientation distance map. . And about the projection point group in which both correspond, CPU11 calculates the square sum of the distance to the nearest pixel edge 18 which the pixel group corresponding to the projection point group has, and based on the calculation result, three-dimensional object 3 is evaluated (S5). That is, the position and orientation determined according to the look-up table based on the magnitude of the error when comparing the edge 24 composed of the projection points 23 and the pixel edge 18 in the pyramid image 15 </ b> A is the actual position of the three-dimensional object 3. Evaluate how close it is from position and orientation.

  Here, when calculating the sum of squares, the distance La from the target pixel 25 to the pixel edge 18 shown in FIG. 4 is used as the distance to the nearest edge. The distance La to the pixel edge 18 means the shortest distance to the edge constituent pixel 17 painted black in the drawing. As described above, when edge extraction for the pyramid image 15 is performed with subpixel accuracy, the distance Lb from the target pixel 25 to the subpixel edge 19 shown in FIG. 4 is used as the distance to the nearest edge. May be. The distance Lb to the subpixel edge 19 means the length of the perpendicular line 26 dropped from the target pixel 25 to the subpixel edge 19. Depending on the balance between the required processing speed and recognition accuracy, the distance La and the distance Lb may be mixed and used as the distance to the nearest edge. For example, the distance Lb may be used when the distance Lb is less than the adjacent pixel distance D, and the distance La may be used when the distance Lb is equal to or greater than the adjacent pixel distance D.

  If the CPU 11 determines that the position and orientation determined according to the lookup table are close to the actual position and orientation of the three-dimensional object 3 as a result of the evaluation, the CPU 11 determines that the position and orientation so that the sum of squares is minimized. The posture is optimized (S6). Here, for this optimization, a conventionally known Levenberg-Markert method is used. In this way, the sum of squares is calculated only for the projection points 23 whose edge directions substantially coincide with the orientation distance map, so that a part of the three-dimensional object 3 is hidden by other objects when viewed from the direction of the camera 4. The pixels in which the distance to the nearest edge is erroneously calculated are excluded from the targets for calculating the sum of squares because the directions of the edges do not match. Thereby, the so-called hiding effect can be reduced and the robustness can be improved. Further, the position and orientation recognition accuracy can be improved by optimizing the position and orientation so that the sum of squares is minimized. Note that the position and orientation optimization method is not limited to the Levenberg-Markert method, and any other conventionally known nonlinear optimization method may be used.

  Thereafter, the CPU 11 determines whether or not the position and accuracy optimized in S6 satisfy the required accuracy (S7), and when determining that the required accuracy is satisfied (S7: Yes), the first The position and orientation obtained for the pyramid image 15A are output as final results (S8), and the process is terminated. On the other hand, as a result of the determination in S7, if it is determined that the required accuracy is not satisfied (S7: No), it is determined whether there is an unprocessed pyramid image 15 (S9), and an unprocessed pyramid image. If it is determined that there is no 15 (S9: No), the result of the first pyramid image 15A is output as the final result (S8), and the process is terminated. On the other hand, if it is determined that there is an unprocessed pyramid image 15 (S9: Yes), the process returns to S2 and the same processing is performed on the remaining pyramid image 15, for example, the second pyramid image 15B. This is repeated until there is no unprocessed pyramid image 15. In this way, by performing processing on the pyramid image 15 having a higher resolution until the required accuracy is reached, the position and orientation of the three-dimensional object 3 can be recognized with higher accuracy. Of course, when a high processing speed is required, it may be determined in advance that the processing is completed at the pyramid image 15 at a predetermined stage, or the processing is performed only for the pyramid image 15 at the predetermined stage. It may be determined.

  The three-dimensional object recognition apparatus according to the present invention can also be used for operation control of devices other than the robot arm.

DESCRIPTION OF SYMBOLS 1 3D object recognition apparatus 3 3D object 4 Camera 8 3D object recognition program 15 Pyramid image 15C 3rd pyramid image 16 Original image 18 Pixel edge 19 Subpixel edge 21 Sampling point 22 Camera image 23 Projection point

Claims (4)

A camera that captures an image by photographing a three-dimensional object to be recognized from a predetermined direction;
While changing the position and orientation of the three-dimensional object, the sampling points that are visible from the camera among the sampling points constituting the edge of the three-dimensional object are respectively projected onto the camera image, and the coordinates of each projection point and each projection Projection point coordinate calculating means for calculating the direction of the edge at each point;
Look-up table storage means for storing a look-up table in which the position and orientation of the three-dimensional object are associated with the coordinates of the projection points and the direction of the edge at each projection point;
Pyramid image creation means for creating a plurality of pyramid images in which the resolution of the original image is reduced at different ratios based on the original image acquired by the camera;
Edge extraction means for extracting an edge of the three-dimensional object for the pyramid image having the lowest resolution;
Direction to create a directional distance map in which each pixel constituting the pyramid image with the lowest resolution has the distance to the nearest edge among the extracted edges and the direction of the nearest edge as pixel values With distance map creation means,
Projection point mapping means for mapping each of the projection points stored in the look-up table on the directional distance map;
The direction of the edge at each projection point is compared with the direction of the nearest edge of the pixel corresponding to the projection point in the directional distance map, and the pixel group corresponding to the projection point group in which both are substantially the same Position and orientation evaluation means for calculating the sum of squares of the distance to the nearest edge of the image and evaluating the position and orientation of the three-dimensional object based on the calculation result;
Position and orientation optimization means for optimizing the position and orientation evaluated by the position and orientation evaluation means as being close to the actual position and orientation of the three-dimensional object so that the sum of squares is minimized;
A three-dimensional object recognition apparatus comprising: The edge extraction means extracts the edge of the three-dimensional object with sub-pixel accuracy;
2. The three-dimensional object recognition according to claim 1, wherein the position and orientation evaluation unit uses a length of a perpendicular line dropped from the pixel group to the sub-pixel precision edge as a distance to the nearest edge. apparatus. Computer
While changing the position and orientation of the three-dimensional object that is the recognition target, among the sampling points that constitute the edge of the three-dimensional object, the sampling points that are visible from the camera are respectively projected onto the camera image, and the coordinates of each projection point and Projection point coordinate calculating means for calculating the direction of the edge at each projection point;
Look-up table storage means for storing a look-up table in which the position and orientation of the three-dimensional object are associated with the coordinates of the projection points and the direction of the edge at each projection point;
Pyramid image creation means for creating a plurality of pyramid images in which the resolution of the original image is reduced at different ratios based on the original image acquired by the camera;
Edge extraction means for extracting an edge of the three-dimensional object for the pyramid image having the lowest resolution;
Direction to create a directional distance map in which each pixel constituting the pyramid image with the lowest resolution has the distance to the nearest edge among the extracted edges and the direction of the nearest edge as pixel values With distance map creation means,
Projection point mapping means for mapping each of the projection points stored in the look-up table on the directional distance map;
The direction of the edge at each projection point is compared with the direction of the nearest edge of the pixel corresponding to the projection point in the directional distance map, and the pixel group corresponding to the projection point group in which both are substantially the same Position and orientation evaluation means for calculating the sum of squares of the distance to the nearest edge of the image and evaluating the position and orientation of the three-dimensional object based on the calculation result;
3 for causing the position and orientation evaluated by the position and orientation evaluation means to be close to the actual position and orientation of the three-dimensional object to function as position and orientation optimization means for optimizing the sum of squares to be minimized. Dimensional object recognition program.   A computer-readable recording medium on which the three-dimensional object recognition program according to claim 3 is recorded.

Images