JP5251080B2 - Object recognition method - Google Patents

Description

  The present invention relates to a method for recognizing the position and orientation of an object by collating the three-dimensional surface shape data of a target object to be recognized with the three-dimensional surface shape data obtained by sampling an actual environment obtained using a distance sensor or the like. About.

Conventionally, model data obtained by approximating a model outline, unevenness, pattern, etc. from an image by a straight line, an arc, or the like is used. These feature data have three-dimensional position data obtained by stereo measurement or the like, and obtain the position and orientation of the target object by collating with the measurement data at three or more points (for example, refer to Patent Document 1).
Japanese Patent No. 3763215

  In the conventional method, (1) it is necessary for humans to specify contours and holes to be used as features, (2) the position and orientation are determined with a small number of correspondences, and therefore the accuracy is low. (3) an object in a two-dimensional image However, it is difficult to stably extract straight lines and arcs representing the outline of the image, and recognition stability is low.

The first feature of the present invention is that the scene data measured by a sensor capable of measuring a distance and the model data representing the three-dimensional shape of the target object are collated with respect to the feature quantity representing the surface shape of each point. In a 3D object recognition method for detecting a 3D position and orientation,
After calculating the characteristic amount of each point of the previous SL model data, we analyze the similarity of the feature amount of each point includes a first steps of selecting the point having the feature amounts effective for matching,
The first step is analyzing the similarity of the feature quantity of each point, and clustering the feature quantity of each point based on the similarity;
When there is a cluster including a predetermined percentage or more of the total number of points of the model data, excluding the points belonging to the cluster from the target of matching with the scene data; and
To include, in the.
“Model data” is composed of a point cloud having a three-dimensional position and a triangular mesh that connects the points and represents the surface of the object.
“Scene data” refers to three-dimensional shape data obtained by sampling an actual environment. It is generated from the measurement data by the distance sensor. Like model data, it consists of a point cloud and a triangular mesh.

The initial value of “ predetermined%” is given by a parameter. If a cluster having a very high degree of similarity (normalized correlation coefficient is almost zero) includes 10% or more of the number of model data points, that cluster is excluded. Using this value as an initial value, the model data and the scene data are actually collated repeatedly, and an optimum value is obtained by learning.

The second feature of the present invention is that a point that is effective for matching is selected based on the operation condition by inputting the object recognition operation condition before calculating the feature amount of each point of the model data. comprises the steps further information the second step, to specify (1) and the decomposing the model data into a plurality of parts, at least one of the floor or viewpoint as operational conditions of (2) the object recognition And determining a part not to be used for collation by inputting.

The third feature of the present invention is that the feature value is a spin image.
The “spin image” is a feature amount that can be generated at an arbitrary point on the surface of the three-dimensional shape data and represents a three-dimensional shape feature around the point.

  According to the object recognition method of the present invention, three-dimensional shape data (model data) of a target object and distance data (scene data) obtained from a distance sensor by performing high-speed positioning based on a feature amount such as a spin image. And the three-dimensional position and orientation of the target object can be quickly recognized.

Further, by inputting the operation conditions for object recognition and selecting points that are effective for verification based on the operation conditions, the data to be verified is reduced, so that the verification process is speeded up.
The operating conditions are, for example, (1) based on the condition that an object does not turn over (can be treated as an error if it does), and the bottom vertex of the object is not used for verification, (2) is visible The part that must be present is determined in advance, and that part is extracted and verified. (3) When the number of objects in the scene data is determined, the numerical value (number) is used for verification. .

  Further, (1) analyzing the similarity of the feature quantity of each point, clustering the feature quantity of each point based on the similarity, and (2) a cluster including a predetermined percentage or more of the total score of the model data If it exists, the step of excluding the points belonging to the cluster from the object of collation with the scene data also includes the data to be collated, and the collation process is speeded up.

  When a spin image is used as a feature value, the spin image is a rotation-invariant feature value that captures the local three-dimensional shape feature of the object. An object can be recognized even in a situation where objects are mixed.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings. In each figure, common portions are denoted by the same reference numerals, and redundant description is omitted.

FIG. 1 is an overall configuration diagram of an object recognition apparatus to which an object recognition method of the present invention is applied. As shown in FIG. 1, the object recognition device 40 includes a 3D sensor 41 that measures a distance to an object (work), an object recognition processing unit 42 that executes object recognition processing, and data that represents the three-dimensional shape of the object (for example, A model database unit 43 for storing CAD data). The position and orientation of the object recognized by the object recognition processing unit 42 are output to the robot control unit 45, a control signal is output from the robot control unit 45 to the robot 46, and the hand 47 is moved to hold the object. Can do.
In the present invention, the self-position means a measurement position, for example, a position and orientation of 6 degrees of freedom in the outside world of the 3D sensor 41.
As the 3D sensor, it is preferable to selectively use an active sensor that employs a triangulation method / light cutting method or a passive sensor that employs a stereo method using a vision sensor.
Although not shown, in the present invention, an odometer, a camera, a GPS, and an attitude sensor other than the distance sensor (3D sensor) may be optionally used as necessary. Hereinafter, an example using a distance sensor will be described.

  FIG. 2 shows an example of the configuration of the object recognition processing unit. As shown in this figure, the object recognition processing unit 42 includes an internal storage device 34 and a central processing unit 35. The object recognition processing unit 42 receives data from the data input device 32 and the external storage device 33 and sends the data to the robot control unit 45.

The data input device 32 includes the above-described distance sensor, and inputs a coordinate value on a three-dimensional shape to the object recognition processing unit 42. In addition, for example, a goniometer, an odometer or the like may be used together to input the position and orientation of the distance sensor and the movement distance. The data input device 32 may also have normal input means such as a keyboard.
The external storage device 33 is a hard disk, a floppy (registered trademark) disk, a magnetic tape, a compact disk, or the like. When the size of the environment model is large and the external storage device 33 cannot hold the coordinate values on the three-dimensional shape, the voxel position, the representative point, and the entire error distribution inputted to the internal storage device 34, which will be described later, For storing a coordinate value, a voxel position, and a representative point and a part or all of an error distribution of an input three-dimensional shape with respect to a partial range or the entire range of a model, and for executing the method of the present invention Memorize the program. The external storage device 33 stores CAD model data described later.
The internal storage device 34 is, for example, a RAM, a ROM, or the like, and the input coordinate value on the three-dimensional shape, the voxel position, and the representative point and a part of its error distribution or the partial range or the entire range of the environmental model Store the whole and store the calculation information.
The central processing unit 35 (CPU) functions as a model input unit, a scene measurement unit, a corresponding point pair creation unit, a grouping unit, a coincidence verification unit, and a high accuracy unit, and processes arithmetic and input / output intensively. Then, the program is executed together with the internal storage device 34.
The robot control unit 45 controls the movement of the robot arm and hand based on the position and orientation of the object received from the object recognition processing unit 42.

  The above-described apparatus of the present invention may be a combination of the above-described distance sensor and a normal PC (computer), or may be an apparatus in which the entirety is integrated. Moreover, you may integrate in the apparatus which can be self-propelled.

[1] Spin Image First, the spin image will be described.

[1.1] Definition of Spin Image A spin image is a rotation-invariant feature quantity that can be generated at an arbitrary point on the surface of three-dimensional shape data and represents a three-dimensional shape feature around that point.

FIG. 3 shows an example of CAD data and a spin image. When generating a spin image of data composed of a triangular mesh, it is generated for each vertex of the triangular mesh.
The similarity between the spin image of the vertex A of the model and the spin image of the vertex B of the scene means that the surface shape of the periphery is similar. This indicates that the points A and B may correspond when the model and the scene are collated.
When creating a spin image of the vertex P of data composed of a triangular mesh, first consider a cylindrical coordinate system with the normal of the vertex P as the central axis. In the cylindrical coordinate system, the distance to the normal is α, and the change in the normal direction is β.

  FIG. 4 shows an image of the cylindrical coordinate system. A spin image is created by plotting the vertices around the vertex P on a two-dimensional plane (referred to as a spin map) of α and β. At that time, α and β are quantized with a certain resolution, and the number of vertices existing in the frame is converted into luminance.

[1.2] Calculation of similarity Since a spin image is expressed as a rotation-invariant digital two-dimensional image, similarity can be evaluated by image comparison means such as normalized correlation.
However, when creating a spin image of a scene, the value of a pixel that originally has a value may become zero due to hiding of an object, or a value may be input to a pixel that originally has a value of zero due to a messy background. is there. Therefore, when two spin images are compared, it is limited to compare pixels whose values are not zero in common (hereinafter, referred to as “pixels with overlap”).
In order to calculate the similarity C (P, Q) between the spin images P and Q, first, a normalized correlation coefficient R (P, Q) is calculated.

Ｎ：スピンイメージＰ、Ｑで重なりがある画素の数
ｐｉ：スピンイメージＰのｉ番目の画素の値
ｑｉ：スピンイメージＱのｉ番目の画素の値

単純にこの正規化相関係数Ｒ（Ｐ，Ｑ）を類似度としてしまうと、重なりが少ないほど類似度が高くなってしまう。例えば、ＰとＱで重なりがある画素が１画素しかない場合、その画素の値が一致すれば、類似度は最高値となってしまう。そこで、重なりがある画素が多いほど類似度が高くなるよう、重み付けを行うことが妥当である。 Formula 1 shows a calculation formula for the normalized correlation coefficient R (P, Q). However, in this normalized correlation, only pixels with overlap are targeted.

N: Number of pixels overlapping in the spin images P and Qpi: i-th pixel value of the spin image P qi: i-th pixel value of the spin image Q

If the normalized correlation coefficient R (P, Q) is simply used as the similarity, the similarity increases as the overlap decreases. For example, if there is only one pixel that overlaps P and Q, the similarity will be the highest value if the values of the pixels match. Therefore, it is appropriate to perform weighting so that the degree of similarity increases as the number of overlapping pixels increases.

FIG. 5 is a processing flow of 3D object recognition according to the embodiment of the present invention.
(1) Model input: Step S10
Read in the model of the object to be recognized and make preparations such as creating a spin image of the model. This process does not need to be performed every measurement. For example, when the target object is determined, the model input is performed only once when the system is activated.

(2) Scene measurement: Step S20
The actual environment is measured by a sensor, distance data is acquired, triangular mesh creation processing and normal calculation processing are performed, and a scene is created.

(3) Corresponding point pair creation: Step S30
A plurality of corresponding point pairs, which are pairs of model vertices and scene vertices, are created based on the similarity between the model and the scene spin image.

(4) Grouping: Step S40
From a plurality of corresponding point pairs, corresponding point pairs that can be established simultaneously are collected into one group. Create all possible groups.

(5) Verification of coincidence: Step S50
When coordinate transformation is performed for each group, the degree of matching between the model and the surface of the scene is evaluated to verify the group.

(6) Higher accuracy: Step S60
Using the conversion formula obtained as a result of the verification as the initial position and orientation of the model, the final position and orientation are calculated by high-precision processing.

  After performing a series of object recognition processing (online processing), the external device is notified of the position and orientation of the obtained object. For example, processing such as moving the robot arm to the gripping position of the target object is performed.

Hereinafter, details of each step of the processing flow of FIG. 5 will be described.
[Model input]
FIG. 6 shows a model input processing flow according to the embodiment of the present invention.

(1) Reading three-dimensional shape data: Step S11
Input the 3D shape data of the target object. For example, a PLY file that is one of three-dimensional shape data expression formats is read. The PLY file includes three-dimensional positions of vertices and triangular mesh information that connects the vertices and represents the surface of the object.
FIG. 7 shows an example of a PLY format file according to the embodiment of the present invention.
FIG. 8 shows an example in which the PLY file according to the embodiment of the present invention is displayed by CAD software.

(2) Parts decomposition of 3D shape data: Step S12
Part division is to divide a three-dimensional shape into its constituent parts (hereinafter referred to as “parts” as appropriate). It is a process of dividing one model into a plurality of components by cutting at places where the curvature changes greatly.

(3) Part validity / invalidity determination: Step S13
Validity / invalidity judgment of parts is made by selecting the measurable part and the impossible part by specifying the floor and viewpoint for the 3D shape model by the operator or external device, and collating the measurable part. This is a process of determining parts that are valid for and parts that cannot be measured as invalid parts for collation. Invalid parts will no longer be used for verification. In addition, it is preferable that the operator has a function that allows the operator to directly specify valid / invalid parts.
(1) Designation of floor surface Rotate the model on an application that can rotate, scale, and translate the model, register the floor surface in a posture that removes the model from under the floor.
(2) Specifying the viewpoint Rotate the model on an application that can rotate, scale, and translate the model, and register the viewpoint from the position of the sensor so that the model is viewed.
(3) Sorting based on the floor surface Using the model size as a reference value, parts that are relatively close to the floor surface and parts whose surface normal is facing the floor surface and the surface is almost parallel to the floor surface cannot usually be measured. Therefore, it should not be used for verification.
(4) Selection based on viewpoint When a model is measured from the viewpoint, parts that cannot be measured due to the shadow of the model are not used for matching.

(4) Feature value calculation of each point of the model: Step S14
The feature amount of each point of the model is calculated. The case where the feature amount is a spin image will be described below. Note that it is not necessary to calculate a feature amount for a part determined to be an invalid part.
(4-1) Normal Vector Calculation Since normal information of each vertex is necessary for setting cylindrical coordinates when creating a spin image, it is created here. The vertex normal is obtained by converting the average of the normals of the triangular mesh including the vertex into a unit vector.
FIG. 9 shows a normal calculation procedure according to the embodiment of the present invention. As shown in the figure, the normal of each triangle around the vertex whose normal is to be obtained is obtained. The normal of the triangle can be calculated from the outer product of the vectors of the two sides of the triangle. The average of normals of all surrounding triangles is converted into a unit vector.

(4-2) Creation of spin image of each vertex of model A spin image of each vertex is created by setting cylindrical coordinates for each vertex and plotting the surrounding vertices on a spin map. Details of the spin image generation procedure will be described later.

  The normal vector, spin image, and parameter λ that have been calculated once can be recorded as a file and read as required. Therefore, when the model once created with the spin image is read again, (2) normal vector creation and (3) spin image creation processing for all vertices of the model can be omitted.

(5) Clustering of feature values: Step S15
The feature amount clustering is a process of dividing the feature amount in the feature space into a subset. For example, by agglomerative hierarchical clustering, each terminal node represents each point (each feature amount), and a binary tree (dendrogram) in which the cluster formed by merging is represented by a non-terminal node is generated. Agglomerative Hierarchical Clustering (1) Creates an initial state with N clusters containing only one object when data consisting of N objects is given, and (2) Between objects x1 and x2 Calculate the distance D (C1, C2) between the clusters C1 and C2 from the distance D (x1, x2) (dissimilarity) of (3), and (3) sequentially merge the two clusters with the closest distance, (4) A hierarchical structure is obtained by repeating this merging until all objects are merged into one cluster.
When the feature amount is a spin image, the distance D between the spin image of the vertex p1 and the spin image of the vertex p2 of the model is the normalized correlation coefficient R (P, Obtained from Q).

(6) Cluster validity / invalidity determination: Step S16
Cluster validity / invalidity determination is processing for determining whether collation is valid or invalid for each cluster. The determination is made based on the knowledge base related to object recognition according to the feature quantity to be used.
As an example, in a process that calculates the feature value of a point from the scene, calculates the feature value and similarity of each point of the model, and finds a correspondence that is prominent and has a high degree of similarity, the originally prominent correspondence Matching with a feature quantity that cannot be obtained is a wasteful process. Therefore, if a cluster contains a predetermined percentage or more of the total number of points in the model, it is clear that even if features similar to that cluster are obtained from the scene, it cannot be used for matching. A cluster including many points is determined to be invalid based on a knowledge base that is not used for the above.
The initial value of “predetermined%” is given by a parameter. When a cluster having a very high degree of similarity (normalized correlation coefficient is almost zero) includes 10% or more of the number of model data points, the cluster is excluded. Using this value as an initial value, the model and the scene are actually verified, and an optimal value is obtained by learning.
Also, as an example, when the feature quantity is a spin image, it is difficult in principle to collate a simple plane, so that a cluster representing a plane is determined to be invalid.

[Scene measurement]
FIG. 10 shows a scene measurement processing flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Distance data acquisition: Step S21
Using a distance sensor, an actual environment where 0 to n target objects and other objects are present is measured, and three-dimensional point cloud data is obtained. It is desirable that the distance sensor to be used is one that can obtain distance data as closely as possible at equal intervals.

(2) Mesh generation: Step S22
A triangular mesh representing the surface of the object is generated from the measured point cloud. As a method of creating a triangular mesh, there are (a) a method of dividing a square lattice and connecting diagonal lines, and (b) a Delaunay triangulation method. (A) Square lattice division is a method in which a point group is projected onto a planar square lattice, representative points are determined for each lattice, and adjacent representative points are connected in a fixed manner to form a triangle. (B) Delaunay triangulation is a method of dividing the point group so that the minimum angle is maximized when the point group is triangulated. In the following, (b) Delaunay triangulation method is used to obtain a more precise triangular mesh.
The measurement data is projected onto a plane, the points on the plane are divided into Delaunay triangles, and the mesh generation is performed by removing triangles whose side length is larger than the threshold when the point group is returned to the three-dimensional space.

(3) Normal vector calculation: Step S23
Since the normal information of each vertex is necessary as the cylindrical coordinate setting condition when creating the spin image, it is created here.

[Corresponding point pair creation]
In the corresponding point pair creation process, a plurality of corresponding point pairs of a model and a scene are created.
FIG. 11 shows a corresponding point pair creation processing flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Select one scene vertex and create a spin image: Step S31
Select a vertex in the scene and create a spin image for that vertex. Random pickup is performed when selecting one vertex of the scene, but there are other methods such as selecting the vertex at the position where the minimum distance from the selected vertex is the maximum, and the accuracy of the normal is good There is a method of selecting vertices (the number of vertices in the periphery is large and the sides of the triangular mesh are concentrated).

(2) Compare the created spin image and the clustering result of the model: Step S32
In order to evaluate whether the created spin image is a feature quantity effective for matching, the created spin image is compared with the clustering result of the model.

  If the created spin image belongs to a cluster effective for collation, the process proceeds to step S34. If the created spin image does not belong to a cluster effective for matching, this scene vertex is not used for matching, and another scene vertex is used for matching.

(2) Calculation of similarity between scene vertex and model vertex: Step S33
A similarity between the created vertex of the scene and each vertex of the model and the spin image is calculated using Equation (1).
For example, when the number of model vertices is 1000, the similarity is calculated for each pair of one scene vertex and 1000 model vertices, and 1000 similarities are calculated.

(3) Add a protruding and similar pair to the corresponding point pair: Step S34
If there is a pair having a particularly high degree of similarity among the 1000 pairs in the above-described example, the corresponding point pair Ci (Si, Mi) is created with the selected scene vertex and model vertex. . Si represents a scene vertex, and Mi represents a model vertex.

By the processing in steps S31 to S34, a corresponding point pair having a corresponding one vertex of the scene is created. This process is repeated for a certain ratio of the number of scene vertices (set as parameter αs). For example, when 20% is set as the parameter αs and the number of scene vertices is 1000, the processes in steps S31 to S34 are repeated 200 times.
As a result, a plurality of corresponding point pairs Ci (Si, Mi) that connect the scene vertex and the model vertex are generated.
FIG. 12 shows an image of a corresponding point pair of a model and a scene according to the embodiment of the present invention. This figure shows the result of measuring an environment in which animal toys are arranged as a scene, and detecting corresponding point pairs using a pig as a model.

[Group]
In the grouping process, a process of making corresponding point pairs that can be established simultaneously as one group is repeated to create a plurality of groups.
FIG. 13 shows a grouping process flow according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Create a group Gi including only the corresponding point pair Ci: Step S41
First, the corresponding point pair Ci (Si, Mi) that connects the scene vertex Si and the model vertex Mi is added to the group Gi. At this point, the group includes only one reference point pair as a reference. That is, Gi = {Ci}.

(2) Determine corresponding point pair Cj that is most easily added to Gi: Step S42
A new corresponding point pair Cj that can be established simultaneously with the corresponding point pair included in Gi is determined. Geometric consistency is used as a criterion for whether it can hold simultaneously. Explain the concept of geometric consistency. When the corresponding point pair C1 (s1, m1) and the corresponding point pair C2 (s2, m2) are simultaneously established, the distance from s1 to s2 and the distance from m1 to m2 are the same, and the s1 normal and the s2 normal And the angles of the three corners formed by the vector s1 → s2 and the angles of the three corners formed by the m1 normal, the m2 normal, and the vector m1 → m2 should be equal. From the degree of coincidence, it can be determined whether or not the corresponding point pairs C1 and C2 have geometric consistency.
FIG. 14 shows a conceptual diagram of geometric consistency according to the embodiment of the present invention.
For example, when evaluating whether C9 can be added to G1 = {C1, C4, C8}, if geometric consistency is established in all of C1-C9, C4-C9, and C8-C9, C9 is set in G1. It can be said that it can be added.

(3) Add Cj to Gi: Step S43
If Gi can maintain geometric consistency even if Cj is added to Gi, Cj is added to Gi. If Gi cannot maintain geometric consistency when Cj is added to Gi, the addition process for Gi ends.

(4) Sort the groups in descending order of the corresponding pair of included points: Step S44
After creating a group based on all the corresponding point pairs, the groups are rearranged in the descending order of the corresponding point pairs in the group.
15A and 15B show examples of grouped corresponding point pairs according to the embodiment of the present invention.

[Verify match]
In the matching degree verification process, the degree of matching between the model and the surface of the scene is evaluated to verify whether each group is appropriate as an object position / posture candidate.
FIG. 16 shows a verification processing flow of the degree of coincidence according to the embodiment of the present invention. Hereinafter, each step will be described.

(1) Creation of KD tree of scene vertex: Step S51
In order to speed up the later corresponding point pair expansion processing, a KD tree of scene vertices is created here. If the number of input data points is N and the number of reference data points is M points, the neighborhood point search processing amount is O (MN) for all searches, whereas O (NlogM) is created by creating a KD tree. ).

(2) Initial position and orientation calculation of group Gi: Step S52
Based on the corresponding point pairs included in the group Gi, a coordinate conversion formula Ti that minimizes the distance between the corresponding points of the model and the scene is calculated. Ti is a 4 × 4 rigid body transformation formula including a 3 × 3 rotation matrix R and a 3 × 1 translation vector t. The result of moving the model according to this conversion formula is set as the initial position and orientation of the object indicated by the group Gi.

(3) Extension of corresponding point pair: Step S53
For the model vertices adjacent to the model vertices included in group Gi, find the nearest scene vertices by KD tree search, and consider the 6-dimensional distance between the model vertices and the scene vertices (taking into account the deviation in the normal direction) If the distance is less than or equal to the threshold value, the corresponding point pair is added to the group Gi. This process is repeated until no corresponding point pair can be added.
FIG. 17 shows an example of corresponding point pair expansion processing according to the embodiment of the present invention. FIG. 4A shows before expansion, and FIG. 4B shows after expansion.

(4) Register Gi to position and orientation candidate G ′: Step S54
If the initial position and orientation are correct and the surface of the model and the scene match, the number of corresponding point pairs in the group should increase significantly due to the corresponding point pair expansion process. If the number of corresponding point pairs in the group is greater than a threshold value (βm% of the number of model vertices), this group is likely to be capturing an object and is left as a position / posture candidate. Otherwise, exclude this group.

This parameter βm should be set according to the operating conditions. In general operation, if the number of corresponding point pairs is 10% or more of the model vertex, the group may catch the target object. Is expensive.
If the number of corresponding point pairs of the group Gi is larger than the threshold value, Gi is added to the position / posture candidate G ′. At that time, the scene vertices included in Gi are made unusable for the subsequent corresponding point pair expansion processing. This prevents position and orientation candidates from appearing in an overlapping manner.

(5) Rearrange G ′ in descending order of the number of corresponding point pairs: Step S55
After the matching degree verification processing is performed for all the groups and the position / orientation candidate G ′ is created, G ′ is rearranged in the descending order of corresponding point pairs.

[High precision]
The position and orientation of the model are determined by a coordinate transformation formula calculated from the corresponding point pair group obtained as a result of the matching degree verification, and the final position and orientation are calculated by high-precision processing using this as the initial position and orientation. .
FIG. 18 shows a flow of high accuracy processing. Hereinafter, each step will be described.

(1) Calculate initial position and orientation from group G′i: Step S61
A coordinate conversion formula T′i that minimizes the distance between the vertexes of the model and the scene is calculated from the pair of corresponding points in the group G′i. T′i is a 4 × 4 rigid body transformation formula including a 3 × 3 rotation matrix R and a 3 × 1 translation vector t. The position and orientation of the model obtained by this T′i is set as the initial position and orientation.

(2) Higher accuracy of position and orientation: Step S62
Improve model position and orientation. Details will be described later.

(3) Model center-of-gravity movement distance calculation: Step S63
The initial position and orientation of the model is compared with the position and orientation obtained as a result of the high accuracy processing, and the moving distance of the center of gravity is calculated. If the moving distance of the center of gravity is larger than the threshold value (parameter δ), it is determined that the initial position and orientation are not reliable and the reliability is not reliable as a result of the high accuracy processing, and this group is removed.

(4) Registration of final three-dimensional position and orientation of object: Step S64
If the movement distance of the center of gravity is within the threshold value, the result of this high accuracy processing is registered as one of the final object positions and orientations.

  All the position / orientation candidates are improved in accuracy to obtain the final position / orientation of the object. When the target object does not exist on the scene or there may be a plurality of target objects, one position or orientation may not be finally obtained, or two or more positions and orientation may be obtained.

[Definition of high-precision processing]
FIG. 19 shows a basic concept of high accuracy processing. As shown in the figure, the basics of high-precision processing are that the model is placed in the initial position and orientation (Fig. (A)), and the corresponding vertex is determined between the model and the scene (Fig. (B)). The coordinate transformation formula of the model that minimizes the sum of the distances between vertices to be calculated is added, coordinate transformation is added to the model (FIG. (C)), and then a new correspondence between the model and the scene is created (FIG. (D)) is repeatedly performed until the end condition is satisfied.

[Main parameters]
The main parameters of the high accuracy processing are described below.
(1) Threshold value of distance between corresponding points: When a corresponding point pair of a model and a scene is created, if vertices that are extremely large are connected as corresponding point pairs, a correct conversion formula cannot be obtained. Therefore, vertices that are more than the threshold value do not correspond to each other.

(2) Normal angle threshold between corresponding points: When creating a corresponding point pair of a model and a scene, if vertices whose normal directions are greatly different are connected as corresponding point pairs, for example, different faces of a rectangular parallelepiped , And the correct conversion formula cannot be obtained. Therefore, when the angle difference between the normals is greater than or equal to the threshold value, it is not handled.

(3) Parameter indicating end condition: As an end condition of the high-precision processing, there are a method of ending when the distance between corresponding points is not shortened, a method of ending when repeated a certain number of times or more. At that time, the distance threshold and the upper limit of the number of repetitions are parameters.

[Improved precision processing]
The following improvements were made to the high-precision processing.
(1) In the triangular mesh representing the surface of the data, the correspondence regarding the vertices forming the contour portion of the surface is not subject to the minimization process. This reduces false correspondence and improves accuracy.
FIG. 20 shows an image of processing that does not use the contour portion.

(2) In addition to finding a scene vertex at the minimum distance from each vertex of the model and making it a corresponding point pair, add processing to find a model vertex at the minimum distance from each vertex of the scene and making it a corresponding point pair, Find the corresponding point pair from the direction. This reduces the number of convergence operations and improves the final accuracy.
FIG. 21 shows an image of determining a bidirectional correspondence point pair.

(3) In the process of minimizing the distance between corresponding points, instead of minimizing the simple Euclidean distance between corresponding points, the inner product of the vector representing the corresponding direction and the normal vector of the corresponding vertex is the Euclidean distance. Minimize the value multiplied by. As a result, an effect of minimizing the distance between the points and the surface is obtained instead of minimizing the distance between the points, and the accuracy can be improved. FIG. 22 shows an image of distance minimization considering the inner product.

(4) After calculating the distance between all corresponding points, a corresponding point pair having an extremely large distance between corresponding points is not subjected to the minimization process. This reduces false correspondence and improves accuracy. Specifically, a process is performed in which corresponding point pairs whose distance between corresponding points exceeds x times the average value (parameter, for example, 10 times) are excluded from the target.

  As described above, high-speed positioning based on feature values using spin images is performed, and then accurate positioning is performed by high-precision processing, which can be obtained from the three-dimensional shape data (model) of the target object and the distance sensor. The obtained distance data (scene) can be collated to recognize the three-dimensional position and orientation of the target object.

  A spin image is a rotation-invariant feature that captures the local three-dimensional shape of an object. Therefore, an object in a situation where part of the object is partially hidden or in which multiple objects are mixed Applicable to recognition.

  However, although feature-based positioning is fast, positioning is performed in a state where there are relatively few locations corresponding to the model and the scene, so the accuracy may be rough. Therefore, high-speed and high-accuracy position / orientation detection can be performed by using the feature-value-based positioning result as an initial position and performing precise positioning by high-precision processing.

  The merit of introducing 3D-CAD data as model data is as follows. (1) It is possible to reduce the time and effort of teaching with the actual product. (2) In the design of industrial parts, 3D-CAD has been introduced, and 3D-CAD data already exists for objects to be recognized. The amount of work can be reduced by using.

  As a rough performance of the above object recognition method, when the number of model vertices is about 1000 points and the number of scene vertices is about 5000 points, if the model appears to some extent (about 15% or more of the entire surface of the model), about 3 seconds (CPU Celeron (registered trademark) 2.8 GHz, memory 512 MB) was able to recognize the position and orientation of the object.

  Although the measurement of the three-dimensional shape has been described as an embodiment, the two-dimensional shape can be similarly measured by looking at the two-dimensional shape as a special case of the three-dimensional shape.

  In addition, this invention is not limited to embodiment mentioned above, Of course, it can change variously in the range which does not deviate from the summary of this invention.

1 is an overall configuration diagram of an object recognition device according to an embodiment of the present invention. It is a figure which shows an example of a structure of the object recognition process part which concerns on embodiment of this invention. It is a figure which shows the example of CAD data and a spin image which concern on embodiment of this invention. It is a figure which shows the image of the cylindrical coordinate system which concerns on embodiment of this invention. It is a figure which shows the processing flow of the three-dimensional object recognition which concerns on embodiment of this invention. It is a figure which shows the model input processing flow which concerns on embodiment of this invention. It is a figure which shows the example of the file of the PLY format which concerns on embodiment of this invention. It is a figure which shows the example which displayed the PLY file which concerns on embodiment of this invention with CAD software. It is a figure which shows the normal calculation procedure which concerns on embodiment of this invention. It is a figure which shows the scene measurement process flow which concerns on embodiment of this invention. It is a figure which shows the corresponding point pair creation process flow which concerns on embodiment of this invention. It is a figure which shows the image of the corresponding point pair of the model and scene which concerns on embodiment of this invention. It is a figure which shows the grouping process flow which concerns on embodiment of this invention. It is a conceptual diagram of the geometric consistency which concerns on embodiment of this invention. (A) And (B) is a figure which shows the example of the grouped corresponding point pair which concerns on embodiment of this invention. It is a figure which shows the verification processing flow of a matching degree which concerns on embodiment of this invention. It is a figure which shows the example of the corresponding point pair expansion process which concerns on embodiment of this invention, Comprising: (A) shows before expansion and (B) shows after expansion. It is a figure which shows the high precision process flow by the high precision process which concerns on embodiment of this invention. It is a figure which shows the basic concept of a highly accurate process. It is a figure which shows the image of the "process which does not use an outline part" performed for improvement of the high precision process in embodiment of this invention. It is a figure which shows the image of "bidirectional corresponding point pair determination" performed for the improvement of the high precision process in embodiment of this invention. It is a figure which shows the image of the "minimization of the distance which considered the inner product" performed for improvement of the high precision process in embodiment of this invention.

Explanation of symbols

32 Data input device 33 External storage device 34 Internal storage device 35 Central processing unit 40 Object recognition device 41 3D sensor 42 Object recognition processing unit 43 Model database unit 45 Robot control unit 46 Robot

Claims (3)

3D for detecting the 3D position and orientation of an object by collating scene data measured by a sensor capable of measuring a distance and model data representing the 3D shape of the target object with respect to a feature amount representing the surface shape of each point Te object recognition method smell,
After calculating the characteristic amount of each point of the previous SL model data, we analyze the similarity of the feature amount of each point comprising a first step of selecting the point having the feature amounts effective for matching,
The first step is analyzing the similarity of the feature quantity of each point, and clustering the feature quantity of each point based on the similarity;
When there is a cluster including a predetermined percentage or more of the total number of points of the model data, excluding the points belonging to the cluster from the target of matching with the scene data; and
A three-dimensional object recognition method comprising: Before calculating the feature value of each point of the model data, the method further includes a second step of selecting an effective point for collation based on the operation condition by inputting the operation condition of the object recognition,
The second step comprises decomposing the model data into a plurality of parts;
Determining parts not to be used for matching by inputting information specifying at least one of a floor surface and a viewpoint as the operation conditions of the object recognition; and
The three-dimensional object recognition method according to claim 1, comprising: The three-dimensional object recognition method according to claim 1, wherein the feature amount is a spin image.

Images