JP2009128192A - Object recognition device and robot device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize a three-dimensional position attitude of a target object by distance data obtained from a distance sensor. <P>SOLUTION: The object recognition device for recognizing the position and attitude of an object comprises a model input means (1); a scene measurement means (2); a means for creating a pair of corresponding points (3); a grouping means for creating a group in reference to all pairs of corresponding points by repeating processing for adding other pairs of corresponding points having geometrical consistency to a group Gi (4); an agreement verification means for determining the position attitude of an object (5); and a means for improving the accuracy for calculating a final position attitude by improving the accuracy by repeating processing for correcting the position attitude (6). <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an object for recognizing the position and orientation of an object by collating the three-dimensional surface shape data of the target object to be recognized with the three-dimensional surface shape data obtained by sampling the actual environment obtained by using a distance sensor or the like. The present invention relates to a recognition device and a robot device.

  Conventionally, the position and orientation of an object are estimated from the current appearance by sufficiently teaching the appearance of the object in advance using a monocular vision sensor (see, for example, Patent Document 1).

In addition, the two-dimensional features of the object are extracted from the first to n images, and the three-dimensional information of the two-dimensional features of the object is added by the correspondence, and the three-dimensional position and orientation of the object is estimated. (For example, refer to Patent Document 2).
Japanese Patent No. 3300682 Japanese Patent No. 3525896

  When recognizing the position and orientation of an object using a monocular vision sensor, (1) It is necessary to teach a large amount of image data in advance, (2) Corresponding to the case where a part of the object is hidden It is difficult to do this, and (3) the recognition result depends on the illumination conditions.

  When recognizing the position and orientation of an object using multiple vision sensors, (1) it is necessary to teach the two-dimensional features of the object and it is difficult to deal with multiple products. (2) the recognition result depends on the illumination conditions. (3) Since a shape feature that can be accurately captured by a vision sensor and can be distinguished from other parts is necessary, there are problems such as large restrictions on the shape of an object that can be detected.

  The present invention has been developed to solve the above-described problems. That is, (1) it is not necessary to teach a large amount of image data in advance, and (2) in an environment where distance data can be measured, it does not depend on the texture conditions or illumination conditions of the object, and (3) it can easily handle a variety of products. (4) An object recognition device and a robot device that are easy to recognize even when a part of an object is hidden are provided.

A first feature of the present invention is an object recognition apparatus that recognizes the position and orientation of an object,
(1) Read three-dimensional shape data consisting of triangular mesh information that represents the surface of an object that connects the three-dimensional positions of the vertices and vertices, and creates a feature value that represents the local surface shape of the object with respect to the vertex of the model Model input means;
(2) Scene measurement means for measuring the actual environment with a sensor to obtain distance data and creating surface information of the object;
(3) Create a feature value representing the local surface shape of the object with respect to the scene vertex, calculate the similarity between the feature value of the scene vertex and the feature value of the model vertex, and the similarity is higher than the threshold value If there is a pair, corresponding point pair creating means for setting the scene vertex and the model vertex as a corresponding point pair Ci (i is a natural number of initial value 1 and maximum value n);
(4) A group Gi including the corresponding point pair Ci (i is a natural number having an initial value of 1 and a maximum value n) is created, and geometric consistency is established in relation to all the corresponding point pairs included in the group Gi. A grouping unit that repeats the process of adding another corresponding point pair having a group Gi to the group Gi, and creates a group based on all the corresponding point pairs;
(5) Based on the corresponding point pair included in the group Gi, a coordinate conversion expression Ti that minimizes the distance between the corresponding points of the model and the scene is calculated, and the result of moving the model according to the coordinate conversion expression Ti is the group Gi Is the initial position and orientation of the object indicated by and finds the nearest scene vertex to the model vertex adjacent to the model vertex included in the group Gi, and the distance between the model vertex and the scene vertex is below the first threshold value If the number of corresponding point pairs in the group is greater than the second threshold after repeating the process of adding the corresponding point pair to the group Gi, the object is present at the position and orientation obtained from the group Gi. Determination and execution of this process for all groups Gi to determine the position and orientation of zero or more objects present in the measurement data;
(6) Using the position and orientation of the model detected by the matching degree verification means as initial values, the model vertex and the vertex having the smallest distance are associated with each other so that the evaluation value between the vertex pair is minimized. The object of the present invention is to provide high precision means for calculating the final position and orientation by increasing the precision by repeating the process of correcting the position and orientation of the model.
“Model” refers to three-dimensional shape data of a target object to be recognized. It consists of a point cloud with a three-dimensional position and a triangular mesh that connects the points and represents the surface of the object.
“Scene” refers to three-dimensional shape data obtained by sampling an actual environment. It is generated from the measurement data by the distance sensor. Like the model, it consists of a point cloud and a triangular mesh.

The second feature of the present invention is that the feature quantity is a spin image.
The “spin image” is a feature amount that can be generated at an arbitrary point on the surface of the three-dimensional shape data and represents a three-dimensional shape feature around the point.

  The third feature of the present invention is that the three-dimensional shape data is 3D-CAD data.

  The fourth feature of the present invention resides in that, in the triangular mesh representing the surface of the data, the high accuracy means does not subject the vertex forming the contour portion of the surface to the object of the minimization process.

  The fifth feature of the present invention is that not only the high-precision means finds a scene vertex at a minimum distance from each vertex of the model and makes a corresponding point pair, but also the model vertex at the minimum distance from each vertex of the scene. It is to find a corresponding point pair.

  According to a sixth feature of the present invention, the high-accuracy means compares the initial position and orientation of the model with the highly accurate position and orientation, calculates the movement distance of the center of gravity, and the movement distance of the center of gravity is the third distance. If it is within the threshold value, the position / posture is registered as one of the final position / postures. If the moving distance of the center of gravity is larger than the third threshold value, the position / posture is set as the final position / posture. There is no registration as one of the postures.

  According to a seventh aspect of the present invention, there is provided a robot apparatus comprising: an object recognition apparatus having the first characteristic; and a control unit that controls a work position of the robot based on position and orientation information output from the object recognition apparatus. There is to do.

  According to the object recognition apparatus of the present invention, the 3D shape data (model) of the target object and the distance data (scene) obtained from the distance sensor are collated by performing high-speed positioning based on a feature amount such as a spin image. Thus, the three-dimensional position and orientation of the target object can be quickly recognized.

  When a spin image is used as a feature value, the spin image is a rotation-invariant feature value that captures the local three-dimensional shape feature of the object. An object can be recognized even in a situation where objects are mixed.

  When 3D-CAD data is used as model data, (1) it is possible to reduce the trouble of teaching with the actual product, (2) 3D-CAD has been introduced in the design of industrial parts, and the object to be recognized is already 3D- Since CAD data often exists, the amount of work can be reduced by utilizing existing data.

  By using the Delaunay triangulation method, a triangular mesh can be created quickly.

  By using the fourth spread method, it is possible to quickly detect corresponding point pairs with high similarity.

  In the triangular mesh representing the surface of the data, the vertices forming the contour portion of the surface are not subjected to the minimization process, thereby reducing the miscorrespondence and improving the accuracy.

  Not only find the scene vertex at the minimum distance from each vertex of the model and make it the corresponding point pair, but also find the model vertex at the minimum distance from each vertex of the scene and make it the corresponding point pair, so that the corresponding point from both directions You will find a pair. As a result, the number of convergence calculations can be reduced and the final accuracy can be improved.

  The initial position and orientation of the model is compared with the highly accurate position and orientation to calculate the movement distance of the center of gravity. If the movement distance of the center of gravity is within the third threshold value, the position and orientation are finalized. If the movement distance of the center of gravity is larger than the third threshold value, the corresponding pair of unreliable points can be obtained by not registering the position and orientation as one of the final positions and orientations. Groups can be removed.

  By using the position and orientation information output from the object recognition apparatus as described above, the robot can be quickly operated along the actual position and orientation of the object.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In each figure, common portions are denoted by the same reference numerals, and redundant description is omitted.

FIG. 1 is an overall configuration diagram of an object recognition apparatus according to the present invention. As shown in FIG. 1, an object recognition apparatus 40 according to the present invention includes a 3D sensor 41 that measures a distance to an object (work), an object recognition processing unit 42 that executes object recognition processing, and data that represents the three-dimensional shape of the object. A model database unit 43 for storing (for example, CAD data) is provided. The position and orientation of the object recognized by the object recognition processing unit 42 are output to the robot control unit 45, a control signal is output from the robot control unit 45 to the robot 46, and the hand 47 is moved to hold the object. Can do.
In the present invention, the self-position means a measurement position, for example, a position and orientation of 6 degrees of freedom in the outside world of the 3D sensor 41.
Although not shown, in the present invention, an odometer, a camera, a GPS, and an attitude sensor other than the distance sensor (3D sensor) may be optionally used as necessary. Hereinafter, an example using a distance sensor will be described.

  FIG. 2 shows an example of the configuration of the object recognition processing unit. As shown in this figure, the object recognition processing unit 42 includes an internal storage device 34 and a central processing unit 35. The object recognition processing unit 42 receives data from the data input device 32 and the external storage device 33 and sends the data to the robot control unit 45.

The data input device 32 includes the above-described distance sensor, and inputs a coordinate value on a three-dimensional shape to the object recognition processing unit 42. In addition, for example, a goniometer, an odometer or the like may be used together to input the position and orientation of the distance sensor and the movement distance. The data input device 32 may also have normal input means such as a keyboard.
The external storage device 33 is a hard disk, a floppy (registered trademark) disk, a magnetic tape, a compact disk, or the like. When the size of the environment model is large and the external storage device 33 cannot hold the coordinate values on the three-dimensional shape, the voxel position, the representative point, and the entire error distribution inputted to the internal storage device 34, which will be described later, For storing a coordinate value, a voxel position, and a representative point and a part or all of an error distribution of an input three-dimensional shape with respect to a partial range or the entire range of a model, and for executing the method of the present invention Memorize the program. The external storage device 33 stores CAD model data described later.
The internal storage device 34 is, for example, a RAM, a ROM, or the like, and the input coordinate value on the three-dimensional shape, the voxel position, and the representative point and a part of its error distribution or the partial range or the entire range of the environmental model Store the whole and store the calculation information.
The central processing unit 35 (CPU) functions as a model input unit, a scene measurement unit, a corresponding point pair creation unit, a grouping unit, a coincidence verification unit, and a high accuracy unit, and processes arithmetic and input / output intensively. Then, the program is executed together with the internal storage device 34.
The robot control unit 45 controls the movement of the robot arm and hand based on the position and orientation of the object received from the object recognition processing unit 42.

  The above-described apparatus of the present invention may be a combination of the above-described distance sensor and a normal PC (computer), or may be an apparatus in which the entirety is integrated. Moreover, you may integrate in the apparatus which can be self-propelled.

[1] Spin Image First, the spin image will be described.

[1.1] Definition of Spin Image A spin image is a rotation-invariant feature quantity that can be generated at an arbitrary point on the surface of three-dimensional shape data and represents a three-dimensional shape feature around that point.

FIG. 3 shows an example of CAD data and a spin image. When generating a spin image of data composed of a triangular mesh, it is generated for each vertex of the triangular mesh.
The similarity between the spin image of the vertex A of the model and the spin image of the vertex B of the scene means that the surface shape of the periphery is similar. This indicates that the points A and B may correspond when the model and the scene are collated.
When creating a spin image of the vertex P of data composed of a triangular mesh, first consider a cylindrical coordinate system with the normal of the vertex P as the central axis. In the cylindrical coordinate system, the distance to the normal is α, and the change in the normal direction is β.

  FIG. 4 shows an image of the cylindrical coordinate system. A spin image is created by plotting the vertices around the vertex P on a two-dimensional plane (referred to as a spin map) of α and β. At that time, α and β are quantized with a certain resolution, and the number of vertices existing in the frame is converted into luminance.

[1.3] Calculation of similarity Since a spin image is represented as a rotation-invariant digital two-dimensional image, similarity can be evaluated by image comparison means such as normalized correlation.
However, when creating a spin image of a scene, the value of a pixel that originally has a value may become zero due to hiding of an object, or a value may be input to a pixel that originally has a value of zero due to a messy background. is there. Therefore, when two spin images are compared, it is limited to compare pixels whose values are not zero in common (hereinafter, referred to as “pixels with overlap”).
In order to calculate the similarity C (P, Q) between the spin images P and Q, first, a normalized correlation coefficient R (P, Q) is calculated.

Ｎ：スピンイメージＰ、Ｑで重なりがある画素の数
ｐｉ：スピンイメージＰのｉ番目の画素の値
ｑｉ：スピンイメージＱのｉ番目の画素の値

単純にこの正規化相関係数Ｒ（Ｐ，Ｑ）を類似度としてしまうと、重なりが少ないほど類似度が高くなってしまう。例えば、ＰとＱで重なりがある画素が１画素しかない場合、その画素の値が一致すれば、類似度は最高値となってしまう。そこで、重なりがある画素が多いほど類似度が高くなるよう、重み付けを行うことが妥当である。 Formula 1 shows a calculation formula for the normalized correlation coefficient R (P, Q). However, in this normalized correlation, only pixels with overlap are targeted.

N: Number of pixels overlapping in the spin images P and Qpi: i-th pixel value of the spin image P qi: i-th pixel value of the spin image Q

If the normalized correlation coefficient R (P, Q) is simply used as the similarity, the similarity increases as the overlap decreases. For example, if there is only one pixel that overlaps P and Q, the similarity will be the highest value if the values of the pixels match. Therefore, it is appropriate to perform weighting so that the degree of similarity increases as the number of overlapping pixels increases.

FIG. 5 is a processing flow of 3D object recognition according to the embodiment of the present invention.
(1) Model input: Step S10
Read in the model of the object to be recognized and make preparations such as creating a spin image of the model. This process does not need to be performed every measurement. For example, when the target object is determined, the model input is performed only once when the system is activated.

(2) Scene measurement: Step S20
The actual environment is measured by a sensor, distance data is acquired, triangular mesh creation processing and normal calculation processing are performed, and a scene is created.

(3) Corresponding point pair creation: Step S30
A plurality of corresponding point pairs, which are pairs of model vertices and scene vertices, are created based on the similarity between the model and the scene spin image.

(4) Grouping: Step S40
From a plurality of corresponding point pairs, corresponding point pairs that can be established simultaneously are collected into one group. Create all possible groups.

(5) Verification of coincidence: Step S50
When coordinate transformation is performed for each group, the degree of matching between the model and the surface of the scene is evaluated to verify the group.

(6) Higher accuracy: Step S60
Using the transformation formula obtained as a result of verification as the initial position and orientation of the model, the model vertex and the vertex of the scene are associated with the vertex with the smallest distance, and the position and orientation of the model so that the evaluation value between this vertex pair is minimized The accuracy is improved by repeating the process of correcting the above, and the final position and orientation are calculated.

  After performing a series of object recognition processing (online processing), the external device is notified of the position and orientation of the obtained object. For example, processing such as moving the robot arm to the gripping position of the target object is performed.

Hereinafter, details of each step of the processing flow of FIG. 5 will be described.
[Model input]
FIG. 6 shows a model input processing flow according to the embodiment of the present invention.

(1) Reading three-dimensional shape data: Step S11
Input the 3D shape data of the target object. For example, a PLY file that is one of three-dimensional shape data expression formats is read. The PLY file includes three-dimensional positions of vertices and triangular mesh information that connects the vertices and represents the surface of the object.
FIG. 7 shows an example of a PLY format file according to the embodiment of the present invention.
FIG. 8 shows an example in which the PLY file according to the embodiment of the present invention is displayed by CAD software.

(2) Normal vector calculation: Step S12
Since the normal information of each vertex is required to set the cylindrical coordinates when creating the spin image, it is created here. The vertex normal is obtained by converting the average of the normals of the triangular mesh including the vertex into a unit vector.
FIG. 9 shows a normal calculation procedure according to the embodiment of the present invention.

(3) Creation of spin image of model vertex: Step S13
By setting cylindrical coordinates for each vertex and plotting the surrounding vertices on a spin map, a spin image of each vertex is created. Details of the spin image generation procedure will be described later.
In addition, the number of pixels whose value is not zero is stored for each spin image. A value ½ of the median becomes a parameter λ used when calculating the similarity between the spin images.

  The normal vector, spin image, and parameter λ that have been calculated once can be recorded as a file and read as required. Therefore, when the model for which the spin image has been created once is read again, (2) normal vector creation and (3) spin image creation processing for the vertex of the model can be omitted.

[Scene measurement]
FIG. 10 shows a scene measurement processing flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Distance data acquisition: Step S21
Using a distance sensor, an actual environment where 0 to n target objects and other objects are present is measured, and three-dimensional point cloud data is obtained. It is desirable that the distance sensor to be used is one that can obtain distance data as closely as possible at equal intervals.

(2) Mesh generation: Step S22
A triangular mesh representing the surface of the object is generated from the measured point cloud. As a method of creating a triangular mesh, there are (a) a method of dividing a square lattice and connecting diagonal lines, and (b) a Delaunay triangulation method. (A) Square lattice division is a method in which a point group is projected onto a planar square lattice, representative points are determined for each lattice, and adjacent representative points are connected in a fixed manner to form a triangle. (B) Delaunay triangulation is a method of dividing the point group so that the minimum angle is maximized when the point group is triangulated. In the following, (b) Delaunay triangulation method is used to obtain a more precise triangular mesh.
The measurement data is projected onto a plane, the points on the plane are divided into Delaunay triangles, and the mesh generation is performed by removing triangles whose side length is larger than the threshold when the point group is returned to the three-dimensional space.

(3) Normal vector calculation: Step S23
Since the normal information of each vertex is necessary as the cylindrical coordinate setting condition when creating the spin image, it is created here.

[Corresponding point pair creation]
In the corresponding point pair creation process, a plurality of corresponding point pairs of a model and a scene are created.
FIG. 11 shows a corresponding point pair creation processing flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Select one scene vertex and create a spin image: Step S31
Select a vertex in the scene and create a spin image for that vertex. Random pickup is performed when selecting one vertex of the scene, but there are other methods such as selecting the vertex at the position where the minimum distance from the selected vertex is the maximum, and the accuracy of the normal is good There is a method of selecting vertices (the number of vertices in the periphery is large and the sides of the triangular mesh are concentrated).

(2) Calculation of similarity between scene vertex and model vertex: Step S32
A similarity between the created vertex of the scene and each vertex of the model and the spin image is calculated using Equation (1).
For example, when the number of model vertices is 1000, the similarity is calculated for each pair of one scene vertex and 1000 model vertices, and 1000 similarities are calculated.

(3) Add a protruding and similar pair to the corresponding point pair: Step S33
If there is a pair having a particularly high degree of similarity among the 1000 pairs in the above-described example, the corresponding point pair Ci (Si, Mi) is created with the selected scene vertex and model vertex. . Si represents a scene vertex, and Mi represents a model vertex.

By the processing in steps S31 to S33, a corresponding point pair having one corresponding vertex of the scene as a corresponding point is created. This process is repeated for a certain ratio of the number of scene vertices (set as parameter α). For example, when 20% is set as the parameter α, if the number of scene vertices is 1000, the processes in steps S31 to S33 are repeated 200 times.
As a result, a plurality of corresponding point pairs Ci (Si, Mi) that connect the scene vertex and the model vertex are generated.
FIG. 12 shows an image of a corresponding point pair of a model and a scene according to the embodiment of the present invention. This figure shows the result of measuring an environment in which animal toys are arranged as a scene, and detecting corresponding point pairs using a pig as a model.

[Group]
In the grouping process, a process of making corresponding point pairs that can be established simultaneously as one group is repeated to create a plurality of groups.
FIG. 13 shows a grouping process flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Create a group Gi including only the corresponding point pair Ci: Step S41
First, the corresponding point pair Ci (Si, Mi) that connects the scene vertex Si and the model vertex Mi is added to the group Gi. At this point, the group includes only one reference point pair as a reference. That is, Gi = {Ci}.

(2) Determine corresponding point pair Cj that is most easily added to Gi: Step S42
A new corresponding point pair Cj that can be established simultaneously with the corresponding point pair included in Gi is determined. Geometric consistency is used as a criterion for whether it can hold simultaneously. Explain the concept of geometric consistency. When the corresponding point pair C1 (S1, M1) and C2 (S2, M2) are simultaneously established, the distance from S1 to S2 is the same as the distance from M1 to M2, and the S1 normal, the S2 normal, and the vector S1 The angles of the three corners formed by S2 and the angles of the three corners formed by the M1 normal, the M2 normal, and the vector M1 → M2 should be equal. From the degree of coincidence, it can be determined whether or not the corresponding point pairs C1 and C2 have geometric consistency.
FIG. 14 shows a conceptual diagram of geometric consistency according to the embodiment of the present invention.
For example, when evaluating whether C9 can be added to G1 = {C1, C4, C8}, if geometric consistency is established in all of C1-C9, C4-C9, and C8-C9, C9 is set in G1. It can be said that it can be added.

(3) Add Cj to Gi: Step S43
If Gi can maintain geometric consistency even if Cj is added to Gi, Cj is added to Gi. If Gi cannot maintain geometric consistency when Cj is added to Gi, the addition process for Gi ends.

(4) Sort the groups in descending order of the corresponding pair of included points: Step S44
After creating a group based on all the corresponding point pairs, the groups are rearranged in the descending order of the corresponding point pairs in the group.
15A and 15B show examples of grouped corresponding point pairs according to the embodiment of the present invention.

[Verify match]
In the matching degree verification process, the degree of matching between the model and the surface of the scene is evaluated to verify whether each group is appropriate as an object position / posture candidate.
FIG. 16 shows a verification processing flow of the degree of coincidence according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Creation of KD tree of scene vertex: Step S51
In order to speed up the later corresponding point pair expansion processing, a KD tree of scene vertices is created here. If the number of input data points is N and the number of reference data points is M points, the neighborhood point search processing amount is O (MN) for all searches, whereas O (NlogM) is created by creating a KD tree. ).

(2) Initial position and orientation calculation of group Gi: Step S52
Based on the corresponding point pairs included in the group Gi, a coordinate conversion formula Ti that minimizes the distance between the corresponding points of the model and the scene is calculated. Ti is a 4 × 4 rigid body transformation formula including a 3 × 3 rotation matrix R and a 3 × 1 translation vector t. The result of moving the model according to this conversion formula is set as the initial position and orientation of the object indicated by the group Gi.

(3) Extension of corresponding point pair: Step S53
For the model vertices adjacent to the model vertices included in group Gi, find the nearest scene vertices by KD tree search, and consider the 6-dimensional distance between the model vertices and the scene vertices (taking into account the deviation in the normal direction) If the distance is less than or equal to the threshold value, the corresponding point pair is added to the group Gi. This process is repeated until no corresponding point pair can be added.
FIG. 17 shows an example of corresponding point pair expansion processing according to the embodiment of the present invention. FIG. 4A shows before expansion, and FIG. 4B shows after expansion.

(4) Register Gi to position and orientation candidate G ′: Step S54
If the initial position and orientation are correct and the surface of the model and the scene match, the number of corresponding point pairs in the group should increase significantly due to the corresponding point pair expansion process. If the number of corresponding point pairs in the group is greater than a threshold value (β% of the number of model vertices), it is determined that this group is likely to capture an object, and the position / posture candidate is left. Otherwise, exclude this group.

This parameter β should be set according to the operating conditions. In general operation, if the number of corresponding point pairs is 10% or more of the model vertices, the group may capture the target object. Is expensive.
If the number of corresponding point pairs of the group Gi is larger than the threshold value, Gi is added to the position / posture candidate G ′. At that time, the scene vertices included in Gi are made unusable for the subsequent corresponding point pair expansion processing. This prevents position and orientation candidates from appearing in an overlapping manner.

(5) Rearrange G ′ in descending order of the number of corresponding point pairs: Step S55
After the matching degree verification processing is performed for all the groups and the position / orientation candidate G ′ is created, G ′ is rearranged in the descending order of corresponding point pairs.

[High precision]
The position and orientation of the model is determined by a coordinate transformation formula calculated from the corresponding point pair group obtained as a result of the matching degree verification, and this is used as the initial position and orientation to calculate the final position and orientation with high accuracy.
FIG. 18 shows a flow of high accuracy processing. Hereinafter, each step will be described.

(1) Calculate initial position and orientation from group G′i: Step S61
A coordinate conversion formula T′i that minimizes the distance between the vertexes of the model and the scene is calculated from the pair of corresponding points in the group G′i. T′i is a 4 × 4 rigid body transformation formula including a 3 × 3 rotation matrix R and a 3 × 1 translation vector t. The position and orientation of the model obtained by this T′i is set as the initial position and orientation at the start of high accuracy.

(2) Higher accuracy of position and orientation: Step S62
Details of the high accuracy will be described later.

(3) Model center-of-gravity movement distance calculation: Step S63
The initial position and orientation of the model is compared with the position and orientation obtained as a result of higher accuracy, and the center of gravity movement distance is calculated. If the moving distance of the center of gravity is larger than the threshold value (parameter δ), it is determined that the initial position and orientation are not reliable and the reliability is not reliable, and this group is removed.

(4) Registration of final three-dimensional position and orientation of object: Step S64
If the movement distance of the center of gravity is within a threshold value, the result of this high accuracy is registered as one of the final object positions and orientations.

  All the position / orientation candidates are improved in accuracy to obtain the final position / orientation of the object. When the target object does not exist on the scene or there may be a plurality of target objects, one position or orientation may not be finally obtained, or two or more positions and orientation may be obtained.

[Definition of high precision]
FIG. 19 shows the basic concept of high accuracy. As shown in the figure, the basics of high accuracy are that the model is placed in the initial position and orientation ((a) in the figure), and corresponding vertices are determined between the model and the scene ((b) in the figure). A model coordinate transformation formula that minimizes the sum of the distances between vertices is calculated, coordinate transformation is added to the model (FIG. (C)), and then a new correspondence between the model and the scene is created (FIG. ( The process d)) is repeated until the end condition is satisfied.

[Main parameters]
The main parameters for high accuracy are described below.
(1) Threshold value of distance between corresponding points: When a corresponding point pair of a model and a scene is created, if vertices that are extremely large are connected as corresponding point pairs, a correct conversion formula cannot be obtained. Therefore, vertices that are more than the threshold value do not correspond to each other.

(2) Normal angle threshold between corresponding points: When creating a corresponding point pair of a model and a scene, if vertices whose normal directions are greatly different are connected as corresponding point pairs, for example, different faces of a rectangular parallelepiped , And the correct conversion formula cannot be obtained. Therefore, when the angle difference between the normals is greater than or equal to the threshold value, it is not handled.

(3) Parameter indicating end condition: As an end condition for high accuracy, there are a method of ending when the distance between corresponding points is not reduced, a method of ending when the distance between the corresponding points is not reduced, and the like. At that time, the distance threshold and the upper limit of the number of repetitions are parameters.

[Improved accuracy]
The following improvements were made to normal high accuracy.
(1) In the triangular mesh representing the surface of the data, the correspondence regarding the vertices forming the contour portion of the surface is not subject to the minimization process. This reduces false correspondence and improves accuracy.
FIG. 20 shows an image of processing that does not use the contour portion.

(2) In addition to finding a scene vertex at the minimum distance from each vertex of the model and making it a corresponding point pair, add processing to find a model vertex at the minimum distance from each vertex of the scene and making it a corresponding point pair, Find the corresponding point pair from the direction. This reduces the number of convergence operations and improves the final accuracy.
FIG. 21 shows an image of determining a bidirectional correspondence point pair.

(3) In the process of minimizing the distance between corresponding points, instead of minimizing the simple Euclidean distance between corresponding points, the inner product of the vector representing the corresponding direction and the normal vector of the corresponding vertex is the Euclidean distance. Minimize the value multiplied by. As a result, an effect of minimizing the distance between the points and the surface is obtained instead of minimizing the distance between the points, and the accuracy can be improved. FIG. 22 shows an image of distance minimization considering the inner product.

(4) After calculating the distance between all corresponding points, a corresponding point pair having an extremely large distance between corresponding points is not subjected to the minimization process. This reduces false correspondence and improves accuracy. Specifically, a process is performed in which corresponding point pairs whose distance between corresponding points exceeds x times the average value (parameter, for example, 10 times) are excluded from the target.

  As described above, after performing high-speed feature-based positioning using a spin image, precise positioning is performed with high accuracy, and it is obtained from the three-dimensional shape data (model) of the target object and the distance sensor. The three-dimensional position and orientation of the target object can be recognized by collating the distance data (scene).

  A spin image is a rotation-invariant feature that captures the local three-dimensional shape of an object. Therefore, an object in a situation where part of the object is partially hidden or in which multiple objects are mixed Applicable to recognition.

  However, although feature-based positioning is fast, positioning is performed in a state where there are relatively few locations corresponding to the model and the scene, so the accuracy may be rough. Therefore, high-precision and high-precision position and orientation detection can be performed by using the feature-based positioning result as an initial position and performing precise positioning with high accuracy.

  The merit of introducing 3D-CAD data as model data is as follows. (1) It is possible to reduce the time and effort of teaching with the actual product. (2) In the design of industrial parts, 3D-CAD has been introduced, and 3D-CAD data already exists for objects to be recognized. The amount of work can be reduced by using.

  As a rough performance of the above object recognition method, when the number of model vertices is about 1000 points and the number of scene vertices is about 5000 points, if the model appears to some extent (about 15% or more of the entire surface of the model), about 3 seconds (CPU Celeron (registered trademark) 2.8 GHz, memory 512 MB) was able to recognize the position and orientation of the object.

  Although the measurement of the three-dimensional shape has been described as an embodiment, the two-dimensional shape can be similarly measured by looking at the two-dimensional shape as a special case of the three-dimensional shape.

  In addition, this invention is not limited to embodiment mentioned above, Of course, it can change variously in the range which does not deviate from the summary of this invention.

1 is an overall configuration diagram of an object recognition device according to an embodiment of the present invention. It is a figure which shows an example of a structure of the object recognition process part which concerns on embodiment of this invention. It is a figure which shows the example of CAD data and a spin image which concern on embodiment of this invention. It is a figure which shows the image of the cylindrical coordinate system which concerns on embodiment of this invention. It is a figure which shows the processing flow of the three-dimensional object recognition which concerns on embodiment of this invention. It is a figure which shows the model input processing flow which concerns on embodiment of this invention. It is a figure which shows the example of the file of the PLY format which concerns on embodiment of this invention. It is a figure which shows the example which displayed the PLY file which concerns on embodiment of this invention with CAD software. It is a figure which shows the normal calculation procedure which concerns on embodiment of this invention. It is a figure which shows the scene measurement process flow which concerns on embodiment of this invention. It is a figure which shows the corresponding point pair creation process flow which concerns on embodiment of this invention. It is a figure which shows the image of the corresponding point pair of the model and scene which concerns on embodiment of this invention. It is a figure which shows the grouping process flow which concerns on embodiment of this invention. It is a conceptual diagram of the geometric consistency which concerns on embodiment of this invention. (A) And (B) is a figure which shows the example of the grouped corresponding point pair which concerns on embodiment of this invention. It is a figure which shows the verification processing flow of a matching degree which concerns on embodiment of this invention. It is a figure which shows the example of the corresponding point pair expansion process which concerns on embodiment of this invention, Comprising: (A) shows before expansion and (B) shows after expansion. It is a figure which shows the high precision processing flow which concerns on embodiment of this invention. It is a figure which shows the basic concept of high precision. It is a figure which shows the image of the "process which does not use a contour part" performed for the improvement in precision in embodiment of this invention. It is a figure which shows the image of "bidirectional corresponding point pair determination" performed for improvement in precision improvement in embodiment of this invention. It is a figure which shows the image of the "minimization of the distance which considered the inner product" performed for the improvement in precision in embodiment of this invention.

Explanation of symbols

32 Data input device 33 External storage device 34 Internal storage device 35 Central processing unit 40 Object recognition device 41 3D sensor 42 Object recognition processing unit 43 Model database unit 45 Robot control unit 46 Robot

Claims (7)

An object recognition device that recognizes the position and orientation of an object,
Model input means for reading the 3D shape data consisting of information representing the surface of the object connecting the 3D positions of the vertices and the vertices, and creating a feature quantity representing the local surface shape of the object with respect to the vertex of the model;
A scene measurement means for measuring the actual environment with a sensor to obtain distance data and creating surface information of the object;
Creates a feature value that represents the local surface shape of the object with respect to the scene vertex, calculates the similarity between the feature value of the scene vertex and the feature value of the model vertex, and there is a pair whose similarity is higher than the threshold value Then, a corresponding point pair creating unit having the scene vertex and the model vertex as a corresponding point pair Ci;
Processing for creating a group Gi including the corresponding point pair Ci and adding another corresponding point pair having geometric consistency in relation to all the corresponding point pairs included in the group Gi to the group Gi And grouping means for creating a group based on all corresponding point pairs,
Based on the corresponding point pair included in the group Gi, the coordinate conversion formula Ti that minimizes the distance between the corresponding points of the model and the scene is calculated, and the result of moving the model according to the coordinate conversion formula Ti is the group Gi. Find the nearest scene vertex for the model vertex adjacent to the model vertex included in the group Gi, and the distance between the model vertex and the scene vertex is below the first threshold If the number of corresponding point pairs in the group is greater than the second threshold after repeating the process of adding the corresponding point pair to the group Gi, the object is in the position and orientation obtained from the group Gi. Consistency verification means that determines that the object exists and executes this process for all groups Gi to determine the position and orientation of zero or more objects present in the measurement data ,
Using the position and orientation of the model detected by the matching level verification means as an initial value, the model vertex and the vertex of the scene are associated with each other with the smallest distance, and the model position is such that the evaluation value between the vertex pair is minimized. An object recognizing device comprising high accuracy means for calculating a final position and orientation by increasing accuracy by repeating processing for correcting the orientation.   The object recognition apparatus according to claim 1, wherein the feature amount is a spin image.   The object recognition apparatus according to claim 1, wherein the three-dimensional shape data is 3D-CAD data.   The object recognition apparatus according to any one of claims 1 to 3, wherein the high-accuracy means does not set a vertex that forms a contour portion of a surface of data as a target of the minimization process.   The high-accuracy means not only finds a scene vertex at the minimum distance from each vertex of the model and makes it a corresponding point pair, but also finds a model vertex at the minimum distance from each vertex of the scene and makes it a corresponding point pair The object recognition device according to any one of claims 1 to 4, wherein   When the high accuracy means compares the initial position and orientation of the model with the highly accurate position and orientation to calculate the movement distance of the center of gravity, and the movement distance of the center of gravity is within the third threshold value , Register the position and orientation as one of the final position and orientation, and if the movement distance of the center of gravity is greater than the third threshold, do not register the position and orientation as one of the final position and orientation The object recognition device according to any one of claims 1 to 5, wherein The object recognition device according to claim 1;
A robot apparatus comprising: control means for controlling a work position of the robot based on the position and orientation information output from the object recognition apparatus.

Images