JP2018189510A - Method and device for estimating position and posture of three-dimensional object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a target in a three-dimensional scene more quickly.SOLUTION: The data of three-dimensional model point group and scene point group is acquired (S100). A point pair feature quantity that defines the geometrical relation of a point pair selected from the model point group and the point pair are recorded in association in a first table (S110). A plurality of key points are selected from the model point group on the basis of the data of model point group and the first table (S120). A second table is constructed leaving only the data, out of the data recorded in the first table, that pertains to the plurality of key points (S130). The data of the scene point group that represents the surface shape of one or more objects included in a three-dimensional scene is acquired (S140). The position and posture of a target in the three-dimensional scene are estimated on the basis of the data of scene point group and the second table (S150).SELECTED DRAWING: Figure 5

Description

  The present application relates to a technique for recognizing an object from a three-dimensional scene and estimating its posture.

  Various techniques for recognizing an object from a three-dimensional scene and estimating its position and posture have been developed. For example, Patent Literature 1 and Non-Patent Literature 1 disclose a method for effectively recognizing a three-dimensional object using a feature amount called a point pair feature (PPF).

U.S. Pat. No. 8,830,229

Drost et al. "Model globally, match locally: Efficient and robust 3D object recognition", 2010 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 13 June 2010, pp 998-1005

  The present application provides a technology capable of recognizing an object in a three-dimensional scene at higher speed.

  A method according to an aspect of the present invention is a method for estimating the position and orientation of an object in a three-dimensional scene. The method includes obtaining data of a model point group representing a three-dimensional shape of the surface of the object, and a point pair feature (Point) that defines a geometric relationship between the point pairs selected from the model point group. (Pair Feature) is calculated for each point pair, the point pair feature value and the point pair are associated and recorded in a first table, and based on the model point cloud data and the first table Selecting a plurality of key points from the model point group; and constructing a second table in which only the data relating to the plurality of key points remains among the data recorded in the first table; Obtaining data of a scene point group representing the surface shape of one or more objects included in the three-dimensional scene And estimating the position and orientation of the object in the three-dimensional scene based on the data of the scene point group and the second table.

  The comprehensive or specific aspect described above can be realized by an apparatus, a system, a method, an integrated circuit, a computer program, a recording medium, or any combination thereof.

  According to the embodiment of the present invention, an object in a three-dimensional scene can be recognized at a higher speed.

1 is a diagram schematically illustrating a robot system in an exemplary embodiment. 2 is a block diagram simply showing an example of a configuration of a control device 100. FIG. It is a figure which shows an example of the robot 200 provided with the three-dimensional sensor 300 and the control apparatus 100. FIG. It is a figure which shows the outline | summary of the object recognition process in embodiment. 3 is a flowchart showing an overall flow of operations executed by a signal processing circuit 120. It is a figure for demonstrating a point pair feature-value (PPF). It is a figure which shows an example of a hash table. FIG. 7 is a diagram shown in FIG. 6 of Patent Document 1. It is a flowchart which shows the selection process of the key point in embodiment. It is a figure which shows the example of the model of a cube shape. It is a figure which shows that PPF calculated from the point m0 and the other point m1 in the area | region MK is collated with PPF in a hash table. It is a figure which shows the example of a response | compatibility with the attitude | position calculated about each of these point pairs, and the combination of the point pair with the same PPF detected about the point m0. It is a figure which shows the example at the time of carrying out the coordinate transformation of the whole model using the coordinate transformation parameter (local coordinate) matched with the point pair (mp, mq) in a model. It is a figure which shows the example at the time of carrying out the coordinate transformation of the whole model using the coordinate transformation parameter (local coordinate) matched with the point pair (mx, my) in a model. It is a figure which shows the example of the reference area in a corner or an edge. It is a flowchart which shows the process of step S150 in FIG. 5 in detail. It is a figure for demonstrating a two-dimensional integral image. It is a figure for demonstrating a three-dimensional integral image. It is a figure which shows the model used for the simulation experiment. It is a graph which shows the simulation result about a synthetic scene. It is a graph which shows the simulation result about a synthetic scene. It is an image which shows an example of the recognition result about an industrial component. It is a graph which shows the example of the change of the recognition rate when changing the ratio of occlusion about an actual scene.

(Knowledge that became the basis of the present invention)
Prior to describing the embodiments of the present invention, the knowledge underlying the present invention will be described.

  A technique for recognizing a specific object from a three-dimensional scene and estimating its position and orientation is important for realizing a robot that performs bin picking, for example. Bin picking refers to gripping a specific object from a plurality of objects stacked at irregular positions and postures and transporting the object to a designated place. In order to realize a robot that performs such work, it is required to accurately and quickly recognize the position and orientation of a specific object from among a plurality of objects stacked randomly.

  Conventionally, it has been relatively easy to recognize an object by detecting a simple geometric shape such as a circle or a rectangle from three-dimensional data. For example, simple shaped objects such as cylindrical bolts, ring shaped bearings, and rectangular parallelepiped boxes could be recognized relatively accurately. However, it has been difficult to correctly detect an object having a shape other than a simple geometric shape (for example, an industrial part) by the above method.

  Patent Document 1 and Non-Patent Document 1 (hereinafter referred to as “Drost”) disclose a method for estimating the posture of an object more effectively than before by using a feature value called Point Pair Feature (PPF). doing. The PPF is a four-dimensional quantity that describes a geometric relationship between a displacement vector between two points in a three-dimensional point group (3D point cloud) and a normal vector at the two points. The PPF is the magnitude of the displacement vector between two points, the angle formed by the normal vector of the displacement vector and the first point, the angle formed by the normal vector of the displacement vector and the second point, and the angle between these two points. It is described by the angle between normal vectors. In the Drost method, the PPF is calculated for every pair of all two points in a model point group representing a three-dimensional shape of the surface of the object. The PPF and point pair information are recorded in the hash table. By using the PPF recorded in the hash table, a portion corresponding to the model point group is detected from the scene point group (scene cloud).

  The method disclosed in Drost can be applied to an object of an arbitrary shape, and can recognize an object with a certain degree of accuracy. However, according to the study by the present inventors, it has been found that there is a problem that the recognition speed is lowered in an object having many similar point pairs such as industrial parts.

  Based on the above findings, the present inventors have developed a novel object recognition technology that can solve the above-mentioned problems.

In one aspect of the invention, a method includes
A method for estimating the position and orientation of an object in a three-dimensional scene,
Obtaining model point cloud data representing a three-dimensional shape of the surface of the object;
A point pair feature (PPF) that defines the geometric relationship of the point pair selected from the model point cloud is calculated for each point pair, and the point pair feature and the point pair are associated with each other. And recording in the first table;
Selecting a plurality of key points from the model point group based on the data of the model point group and the first table;
Constructing a second table in which only the data relating to the plurality of key points is left among the data recorded in the first table;
Obtaining scene point cloud data representing the surface shape of one or more objects included in the three-dimensional scene;
Estimating the position and orientation of the object in the three-dimensional scene based on the data of the scene point group and the second table;
including.

  According to the above aspect, the second table in which only the data relating to the plurality of key points is left among the data recorded in the first table is constructed. Then, the position and orientation of the object in the three-dimensional scene are estimated based on the data of the scene point group and the second table. Thereby, the step which estimates the position and attitude | position of a target object can be sped up.

  The first table corresponds to the hash table disclosed in Drost. Since this hash table includes a lot of data, there is a problem that the calculation amount increases. When the number of points in the model point group is Nm, the number of point pairs recorded in the hash table is Nm (Nm−1). On the other hand, since the second table includes only data relating to some points (key points) selected from the model point group, the second table has a smaller number of data. If the number of key points is Nk (<Nm), the number of point pairs recorded in the hash table can be reduced to Nk (Nm−1). The number Nk of key points can be set to, for example, 1 / 10,000 times or more and 1/2 times or less the number Nm of points included in the model point group. In an example, the number Nk of key points can be set to, for example, 1/1000 times or more and 1/100 times or less the number Nm of points included in the model point group. When a small number of key points are selected in this way, the calculation time required for posture estimation using a hash table can be greatly shortened.

A key point is a set of preferred points that are carefully selected from a model point group based on a predetermined criterion. The key point is selected, for example, from the points that are determined to have high attitude uniqueness. The process of selecting a plurality of key points can include, for example, the following steps.
(A) A point mi in the model point group (i is an integer of 1 to Nm and Nm is the number of points included in the model point group) and a distance from the point mi is included in a reference region shorter than a threshold value. Point pair feature quantities with points mj (j is an integer of 1 to M and M is the number of points included in the reference area) are calculated.
(B) Referring to the first table, search for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount.
(C) For each of the searched point pairs, a coordinate conversion parameter for matching with a pair of points mi and mj is determined.
(D) Among the points mi, when the coordinates of the model point group are converted using each of the determined coordinate conversion parameters, the converted model point group is placed in the vicinity of all the points included in the reference region. A point mi having the smallest number when any point is located is selected as a key point.

  According to the above processing, a point with a high attitude uniqueness is selected as a key point, so that the subsequent estimation processing can be speeded up.

The process of estimating the position and orientation of the object can include, for example, the following steps.
(E) The distance from each point si in the scene point group (i is an integer of 1 or more and Ns or less, Ns is the number of points included in the scene point group) is the most distant 2 included in the model point group A part of the point group Sr is selected from a set of points that are less than or equal to the distance between the points.
(F) A point pair feature quantity between the point si and each point Srj (j is an integer of 1 to Ms) in the point group Sr is calculated.
(G) Referring to the second table, search for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount.
(H) For each of the searched point pairs, a coordinate conversion parameter for matching with a pair of point si and point Srj is determined, and a count number is determined for each posture indicated by the coordinate conversion parameter.
(I) A plurality of candidate postures are selected in descending order of the count number from the postures for which the count number has been determined.
(J) From the plurality of candidate postures, a posture having the highest degree of matching between the model point group and the scene point group is determined as a final posture.

  According to the above processing, since all points in the scene point group are collated with the model point group, the position and orientation of the object can be estimated more accurately. Furthermore, since the second table with a small amount of data is referred to, the time required for the collation process can be shortened.

  The methods disclosed herein may be implemented by a computer, processor, or processing circuit that executes a computer program. Such a computer program is stored in a memory in the apparatus, for example. The present disclosure includes a processing device or a robot including a memory storing such a computer program and a processor or a processing circuit that executes the computer program.

  Hereinafter, more specific embodiments of the present invention will be described. However, more detailed explanation than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The inventor provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and is not intended to limit the subject matter described in the claims. Absent. In the following description, the same or similar components are denoted by the same reference numerals.

(Embodiment)
[1. Constitution]
FIG. 1 is a diagram schematically illustrating a robot system in an exemplary embodiment of the present invention. This system includes a robot 200, a three-dimensional sensor 300, and a control device 100. In the present embodiment, the signal processing circuit 120 included in the control device 100 executes a method according to an aspect of the present disclosure by executing a program stored in the memory 130.

  The robot 200 includes a plurality of arms and an end effector (hand), and can grip and move an object. The robot 200 receives a command from the control device 100 and performs an operation (bin picking) by grasping a specific object from a plurality of objects 800 stacked randomly in the box and carrying the specific object to a specified position. . The robot 200 includes a plurality of electric motors and at least one control circuit for controlling each motor. These control circuits appropriately control each motor in accordance with a command from the control device 100, and cause the robot 200 to execute a desired operation.

  The three-dimensional sensor 300 images a plurality of objects 800 in the box, generates three-dimensional point cloud data, and outputs it. This three-dimensional point cloud data is hereinafter referred to as “scene point cloud data”. The three-dimensional sensor 300 acquires three-dimensional data such as a distance distribution in the region to be imaged, and generates and outputs three-dimensional point cloud data based on the data. The three-dimensional sensor 300 can be realized by, for example, a laser range finder, a TOF (Time of Flight) camera, or a stereo camera.

  The “three-dimensional point group” means a set of a plurality of points distributed on the surface of one or more objects existing in the three-dimensional space. In the present embodiment, in addition to the scene point cloud data acquired by the three-dimensional sensor 300, model point cloud data indicating a three-dimensional point cloud of an object to be detected (hereinafter sometimes referred to as “target”). Are prepared in advance. Such model point cloud data is stored in a recording medium included in the control device 100.

  The control device 100 is connected to the robot 200 and the three-dimensional sensor 300 by wire or wirelessly and controls these operations. The control device 100 recognizes an object to be detected in the three-dimensional scene based on the scene point cloud data acquired by the three-dimensional sensor 300 and the model point cloud data prepared in advance, and estimates its posture.

  FIG. 2 is a block diagram schematically illustrating an example of the configuration of the control device 100. The control device 100 is a general-purpose or dedicated computer, for example, and includes a control circuit 110, a signal processing circuit 120, and a memory 130. The control device 100 may include other components such as a communication circuit, an input / output interface, a recording device such as a hard disk drive, and a power supply circuit, which are omitted in FIG. In FIG. 2, in addition to the robot 200 and the 3D sensor 300, a display 400 is shown. The display 400 is connected to the control device 100 and displays a processing result by the signal processing circuit 120.

  The control circuit 110 can be realized by a processor such as a CPU (Central Processing Unit). The control circuit 110 controls the robot 200, the 3D sensor 300, and the signal processing circuit 120 by executing a computer program stored in the memory 130 (for example, DRAM or SRAM).

  The signal processing circuit 120 can be realized by a processor such as a DSP (Digital Signal Processor). The signal processing circuit 120 executes the computer program stored in the memory 130 to perform the object recognition process in the present embodiment.

  The configuration of the system shown in FIGS. 1 and 2 is merely an example, and various modifications are possible. For example, one or both of the three-dimensional sensor 300 and the control device 100 may be incorporated in the robot 200.

  FIG. 3 is a diagram illustrating an example of the robot 200 including the three-dimensional sensor 300 and the control device 100. The robot 200 includes a three-dimensional sensor 300 in the vicinity of the end effector 220. The robot 200 also includes a control device 100 in the housing. In this example, the control device 100 in the robot 200 executes a recognition process described later, and controls the arm and the end effector 220 according to the result. The robot 200 may also include a light source used to acquire 3D data. Such light sources may include, for example, lasers or light emitting diodes that emit visible or near infrared light.

[2. Operation]
Next, an object recognition process executed by the signal processing circuit 120 in the control device 100 will be described.

  FIG. 4 is a diagram showing an outline of object recognition processing in the present embodiment. As illustrated, in this embodiment, scene point cloud data is acquired by three-dimensional measurement by the three-dimensional sensor 300. Further, model point cloud data indicating the distribution of the three-dimensional point cloud of the model is prepared in advance. The signal processing circuit 120 acquires these two types of point cloud data and collates them. As a result, the same object as the model is recognized from the three-dimensional scene, and the result is output. The detection result can be output to the display 400 as image data, for example.

  FIG. 5 is a flowchart showing the overall flow of the operation executed by the signal processing circuit 120. The processing in this embodiment includes an offline phase and an online phase. The offline phase is a preparatory phase that is performed before the three-dimensional measurement. The online phase is a phase that is performed after performing three-dimensional measurement.

  The offline phase includes steps S100, S110, S120, and S130. In step S <b> 100, the signal processing circuit 120 acquires model point cloud data from the memory 130. Next, in step S110, the PPF between all two points in the model point group is calculated and recorded in the hash table. This hash table corresponds to the first table described above. In step S120, the signal processing circuit 120 selects a plurality of key points from the model point group based on the hash table. The plurality of key points are special points in the model point group that are determined to be highly unique in posture. Detailed processing in step S120 will be described later. In step S130, the signal processing circuit 120 reconstructs a hash table excluding data related to points other than key points. This reconstructed hash table corresponds to the aforementioned second table. The reconstructed hash table has a smaller amount of data than the original hash table.

  The hash table used in the online phase is constructed by the processing in the offline phase. This hash table contains only data relating to a small number of carefully selected key points that are considered desirable for accurately recognizing objects. For this reason, the operation in the online phase can be performed more quickly and accurately.

  The offline phase includes steps S140 and S150. In step S140, the signal processing circuit 120 first acquires scene point cloud data. The scene point cloud data is acquired by the three-dimensional sensor 300 based on an instruction from the control circuit 110 when three-dimensional recognition is started, and stored in the memory 130. The signal processing circuit 120 acquires scene point cloud data from the memory 130 in accordance with an instruction from the control circuit 110. In step S150, the signal processing circuit 120 estimates the position and orientation of the object in the scene based on the scene point cloud data and the reconstructed hash table. In this embodiment, a method obtained by improving the voting scheme disclosed in Drost is used. Although details will be described later, in the present embodiment, many posture candidates are selected for each point of the scene point group, and the posture estimated to be the most correct is determined from the candidates. Unlike Drost's method, posture verification is performed for all points in the scene point group, so that the posture of the object can be estimated more accurately.

[2-1. PPF and hash table]
A point pair feature quantity (PPF) and a hash table used in this embodiment will be described.

ｄは、点ｍ１から点ｍ２へのベクトルを表す。∠（ａ，ｂ）は、ベクトルａおよびｂのなす角度（０以上π以下の実数）を表す。ｆ１、ｆ２、ｆ３、ｆ４の意味は、以下のとおりである。
ｆ１：点ｍ１およびｍ２の間の距離
ｆ２：ベクトルｄと法線ベクトルｎ１との角度
ｆ３：ベクトルｄと法線ベクトルｎ２との角度
ｆ４：法線ベクトルｎ１、ｎ２間の角度 FIG. 6 is a diagram for explaining the PPF. PPF represents the relative position and orientation of two points with normal vectors. The two points are m1 and m2, and the normal vectors of those points are n1 and n2, respectively. The PPF at the points m1 and m2 is expressed as F (m1, m2). F (m1, m2) is represented by the following formula (1).

d represents a vector from the point m1 to the point m2. ∠ (a, b) represents an angle (a real number between 0 and π) formed by the vectors a and b. The meanings of f1, f2, f3, and f4 are as follows.
f1: Distance between points m1 and m2 f2: Angle between vector d and normal vector n1 f3: Angle between vector d and normal vector n2 f4: Angle between normal vectors n1 and n2

  In this embodiment, in step S110, the PPF between all two points in the model point group is calculated and stored in the hash table.

  FIG. 7 is a diagram illustrating an example of a hash table. As shown in the figure, the hash table defines the relationship between the PPF and the corresponding point pair using the PPF as a key. The components f1, f2, f3, and f4 in the PPF are discretized, and a plurality of PPFs whose values are close to each other are processed as the same PPF.

[2-2. Voting scheme]
The selection of key points in this embodiment uses a part of the voting scheme disclosed in Drost. For this reason, before explaining the keypoint selection process in the present embodiment, the voting scheme disclosed in Drost will be explained.

Drost has proposed an effective voting scheme that reduces the voting space in two dimensions. In the Drost method, in the online phase, PPF, that is, F (sr, si) is calculated from the reference point sr and other points si in the scene point group, and this PPF is searched from the hash table. Thereby, a model point pair (mr, mi) having a similar PPF is searched. For reference, FIG. 6 of Patent Document 1 is cited as FIG. As shown in FIG. 8, the following two steps are performed to align the model point pairs with the scene point pairs.
(A) The reference points of the point pairs (sr, si) and (mr, mi) are translated from the origin, and their normals are aligned with the x-axis. Let these parallel progressions be Tm and Ts.
(B) The model point pair (mr, mi) is rotated around the x axis until it matches the scene point pair (sr, si).

Since the model point pair (mr, mi) and the scene point pair (sr, si) have similar PPF, by rotating around the x axis, the position and normal of the point mi are changed to the position of the point si. And can be approximately matched to the normal. The rotation angle is α, and the rotation matrix is Rx (α). A pair (mr, α) of mr and α is called a local coordinate. By the above operation, a two-dimensional accumulator table T (mr, α) is generated. This table records the local coordinate votes, that is, how many times local coordinates appear. Local coordinates represent candidates for the posture of the point Sr. Coordinate conversion using the rotation angle α is expressed by the following equation (2).

  In the Drost method, some reference points sr are selected from the scene point group, and local coordinates are voted while changing si for each reference point sr. A final posture is determined by performing processing such as clustering and averaging a plurality of postures corresponding to a plurality of local coordinates having a large number of votes.

[2-3. Select keypoint]
Details of the key point selection process (step S120) in the offline phase will be described. The selection of key points is extremely important in the posture estimation algorithm in this embodiment. In the present embodiment, several key points are selected from the model point group, and points and postures corresponding to them are searched from the scene point group. For this reason, it is important to select good key points. In this embodiment, a highly unique point is selected from the model point group. Thereby, the accuracy of object recognition can be improved.

  In the present embodiment, in the offline phase, key points that are considered to be highly unique are selected from the model point group using the concept of the voting scheme described above. In order to select a key point, the point mi in the model point group and the surrounding point group are regarded as the scene point group. Then, the posture of the scene cloud is estimated using the voting scheme described above. If only a small number of poses can be estimated, the scene cloud is sufficiently special, that is, the mi is suitable as a key point.

  FIG. 9 is a flowchart showing key point selection processing in the present embodiment. As illustrated, the key points are selected by, for example, the following process.

  1) For all the points mi (i is an integer of 1 to Nm, Nm is the number of points included in the model point group) in the model point group, an area where the distance from mi is shorter than a threshold (referred to as a reference area) ) Is set as MKi (step S121).

2) PPFs of mi and each other point mj in MKi (j is an integer of 1 to M and M is the number of points included in the reference area) are respectively calculated (step S122). Each calculated PPF is searched from the created hash table (step S123). In this case, one or more corresponding model point pairs are found. Let that number be Cj. For each of the Cj model point pairs, the local coordinates (mr, α) are calculated and voted on the accumulation table (step S124). This vote is repeated until completion for the last point m _{M in} MKi.

  3) For all local coordinates (posture candidates) for which the number of votes is not zero, all model points are moved (translated and rotated) into the scene space (step S125). At this time, Equation (2) is used. When the model point after movement is located near all the points in MKi (position where the distance is less than the threshold), 1 is added to the score of mi (steps S126 and S127). This score is determined for each mi.

4) A point (group) having the lowest score among m ₁ to m _Nm is selected as a key point (step S128).

  Table 1 shows an example of the algorithm shown in the flowchart of FIG. By using such an algorithm, the signal processing circuit 120 can select a good key point with a high attitude uniqueness.

  In the present embodiment, points on the edge are not selected as key points. The score is always low, but normals at points on the edge are considered unreliable. However, if there is no problem in performance, a point on the edge may be included as a key point.

  FIG. 10A to FIG. 10F are diagrams for explaining the concept of score calculation in the present embodiment. Here, for the sake of simplicity, consider the case where the surface shape of the model is a legislative body.

FIG. 10A is a diagram illustrating an example of a cubic model. In this example, except for points on the edge, the normal of each point is perpendicular to the surface. m ₀ is a score calculation target point. A circle in the figure represents a region MK in which the distance from the point m ₀ is equal to or less than a threshold value. In the present embodiment, assuming that this region MK exists in the scene, PPF calculation and hash table search are performed.

FIG. 10B is a diagram showing that the PPF calculated from the point m ₀ and the other point m ₁ in the region MK is collated with the PPF in the hash table. In this example, point pairs (mp, mq) and (mx, my) having a PPF (F0) of (m ₀ , m ₁ ) and a common PPF are detected. Such PPF matching is performed for the combination of the point m ₀ and all other points in the region MK.

FIG. 10C is a diagram illustrating an example of correspondence between combinations of point pairs having the same PPF detected for the point m ₀ and postures calculated for the respective point pairs. Postures P0 to Pn represent the postures indicated by the local coordinates. For each of the postures P0 to Pn, the coordinate conversion is performed on the coordinate system including the area MK in which the model point group is a virtual scene.

FIG. 10D shows an example in which the entire model is coordinate transformed using coordinate transformation parameters (local coordinates) associated with a point pair (mp, mq) in the model. In this example, any point in the model point group after conversion exists in the vicinity (position where the distance is less than the threshold) of all the points included in the region MK. In such a case, it adds 1 to the score of the m _0.

FIG. 10E shows an example in which the entire model is coordinate-transformed using coordinate transformation parameters (local coordinates) associated with a point pair (mx, my) in the model. In this example, for some points included in the region MK, none of the converted model points exists in the vicinity (position where the distance is less than the threshold). In such a case, the score of m ₀ does not increase.

Above such scoring is performed for all the postures P0~Pn shown in FIG. 10C, the final score of the m ₀ is determined. By executing the same process for all points in the model point group, the score of each point is determined. Finally, in step S128, the point or point group with the lowest score is selected as the key point. It should be noted that a predetermined number of points may be selected as key points in order from the lowest score, not limited to the minimum score.

FIG. 10F is a diagram illustrating an example of reference regions at corners or edges. When a point A that is a corner or an edge is selected as a key point, the hatched area in FIG. 10F becomes the reference area. For such a point A, the score tends to be much smaller than the point m ₀ described above.

[2-3. Rebuild hash table]
The signal processing circuit 120 reconstructs the hash table after selecting a plurality of key points. That is, data that is not related to a plurality of key points is deleted from the hash table. In this embodiment, unlike Drost's approach in which PPFs between all model point groups are stored in a hash table, only the PPFs between the key point (s) and the other model points remain in the hash table. Assuming that the number of points in the model point group is Nm and the number of key points is Nk, the number of model point pairs stored in the hash table after reconstruction is Nk (Nm−1). Compared with Nm (Nm-1) in the Drost approach, the data amount is Nk / Nm times. Nk / Nm can be suppressed to a low value of about 1/1000 to 1/100, for example. As an example, when Nm = 3000 and Nk = 10, the data amount of the hash table is reduced to 1/300. Thereby, the calculation time in an online phase can be shortened.

[2-4. Posture estimation processing]
Next, a specific example of the posture estimation process in step S150 of FIG. 5 will be described.

  FIG. 11 is a flowchart showing the process of step S150 in more detail. In this embodiment, a voting scheme more suitable for bin picking than the Drost voting scheme is used. All points are selected, not part of the scene point cloud, and the PPF is calculated between the surrounding point clouds.

For each scene point si, a point group whose distance from si is shorter than the model size is referred to as Sphere _i . The model size means the distance between the two most distant points in the model point group. The signal processing circuit 120 selects a part of the point group Sphere _i at random (step S151). Instead of selecting at random, some points may be selected according to a predetermined rule (for example, every tenth). The point group selected here is Sr. The number of points in the selected point group Sr is Nr times the number of points included in the point group Spherei. Nr may be a number sufficiently smaller than 1, for example (Nr << 1).

  The signal processing circuit 120 calculates the PPF between the point si and the jth point Srj included in Sr (j is an integer between 1 and Ms, and Ms is the number of points in Sr) (step S152). Thereafter, the voting scheme described above is executed. The signal processing circuit 120 searches the reconstructed hash table for a model point pair having a PPF close to the calculated PPF (step S153). Assume that Cj model point pairs are found. The signal processing circuit 120 determines local coordinates for matching with (si, Srj) for each of the Cj model point pairs, and determines the number of votes for each local coordinate (step S154). Steps S152 to S154 are repeated until j becomes 1 to Ms.

  The above processing is executed for all points included in the scene point group. By searching for local coordinates from all scene points, the possibility that a correct posture is excluded is reduced. This scheme is suitable for applications such as bin picking to detect multiple objects in a scene. Furthermore, since the hash table is much smaller than before, the calculation time required for the voting scheme does not increase even though the PPF is calculated for more scene point pairs.

  As the number of votes of local coordinates increases, the posture indicated by the local coordinates is more consistent with many point pairs, and thus is more likely to be the correct posture. Therefore, some local coordinates with a large number of votes are selected for verification. The signal processing circuit 120 selects Nv local coordinates in order from the largest number of votes for posture verification (step S155). These local coordinates represent candidates for the posture of the object. The signal processing circuit 120 verifies the posture indicated by each local coordinate, and calculates a score indicating accuracy for each posture (step S156). Then, the posture having the highest score is determined as the final posture (step S157).

  In the present embodiment, at the voting stage, some point groups Sr are randomly selected from the scene point group. This is because calculating the PPF between all two points in the scene point group increases the calculation time. In the present embodiment, local coordinate candidates are selected by ranking rather than the absolute number of votes, so it is not necessary to calculate PPF between all two points. If Nr is constant for each si, the ranking result does not change significantly. The error is corrected if Nv (number of postures to be verified) is large. Nr can be suppressed to a small number of about 3 to 10%, for example. Nv is a value of about 8000 to 20000, for example.

  Table 2 shows an example of the algorithm shown in the flowchart of FIG. By using such an algorithm, the signal processing circuit 120 can determine a highly reliable posture in a relatively short time.

[2-5. Posture verification]
Next, the posture verification process in step S156 will be described in more detail.

  In the embodiment, unlike the methods disclosed in Patent Document 1 and Non-Patent Document 1, clustering of candidate postures is not performed. Verify the accuracy of more candidate poses.

  In the conventional posture verification method, the model point group is converted into the scene point group, and the closest scene point is searched for all the model points. The distance between these two points and the angle between their normals are compared. If each of the distance and angle is smaller than the threshold value, the model point is considered to match the scene point. If the number of matching model points is large enough, the posture is considered correct. For example, Nguyen et al. ("Determination of 3d object pose in point cloud with cad model. 21st Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp. 16, 2015) A method of fitting with a point is disclosed.

  This method is intuitive but takes a lot of time. This is because it is necessary to search for a point closest to all model points for all the posture candidates.

  In the present embodiment, a method that does not require finding a point closest to the model point is used. Specifically, a three-dimensional integrated image (integral image) is used. Thereby, the efficiency of posture verification can be greatly improved.

ｉｉ（ｘ，ｙ）は、積分画像の画素値を、ｉ（ｘ，ｙ）は、元画像の画素値を表している。 FIG. 12A is a diagram for explaining a two-dimensional integral image. In the two-dimensional integrated image, the pixel value at the point (x, y) is the sum of all pixel values in the region located at the upper left from the position (x, y) in the original image. That is, the pixel value of the integral image is expressed by the following formula (3).

ii (x, y) represents the pixel value of the integral image, and i (x, y) represents the pixel value of the original image.

The pixel value of the integral image is calculated by the following equation (4).

By using the integral image, as shown in the following formula (5), the sum of pixel values of an arbitrary rectangle can be calculated by referring to four arrays.

  The basic concept of the three-dimensional integral image is the same as that of the two-dimensional image. The only difference is that the searched space changes from a rectangle to a rectangular parallelepiped.

FIG. 12B is a diagram for explaining a three-dimensional integral image. The three-dimensional integral image is used to calculate the sum of arbitrary rectangular parallelepiped values. Here, each side of the rectangular parallelepiped is parallel to the coordinate axis. Similar to the equation (4), the three-dimensional integral image can be calculated by the following equation (6).

After constructing the three-dimensional integral image, the sum of the values of any rectangular parallelepiped can be calculated by the following equation (7).

  By using the concept of a three-dimensional integral image, the algorithm can be speeded up. In the present embodiment, a three-dimensional space in which a scene point group exists is handled as a three-dimensional image. One pixel value is the number of scene points in the pixel. Thereby, a three-dimensional integrated image of the three-dimensional space is generated. The number of scene points included in an arbitrary rectangular parallelepiped can be calculated by Expression (7).

  For all model points mi converted to a coordinate system including a scene point group, each point mi is arranged at the center of a cube whose sides are parallel to the coordinate axis. It is determined whether there are scene points around the point mi. If the number of points in the cube is not zero, any point in the cube may correspond to mi.

  Three 3D integration maps are generated to calculate the angle between normals. The three-dimensional integration map individually stores x, y, z components of the normal of the scene point group. The sum Σnx, Σny, Σnz of the three three-dimensional integration maps in the same cube can be considered as the normal of the entire scene points in that cube. If the angle between the normal of mi and (Σnx, Σny, Σnz) is smaller than the threshold, this model point has a corresponding scene point, and this model point can be called a correct point. The number of accurate points (accurate score) is used as a score for each posture.

  In this method, the accurate score is an important parameter in evaluating recognition performance. Since the pose has already been given, all model points are converted to the scene space. If the pose is accurate, many model points match the scene points after conversion. That is, the accurate score increases. It is possible to easily determine which posture is the most accurate by comparing the accurate score for a plurality of candidate postures.

  An advantage of using a three-dimensional integral image is that calculation efficiency can be improved. According to the experiments by the present inventors, for example, the posture can be verified 10,000 times or more in 2 seconds.

  Since the posture verification method in the present embodiment is very fast, unlike the method disclosed in Drost, posture clustering is not necessary. Furthermore, verification accuracy is also improved. In many cases, it is possible to estimate the posture more accurately than in the past by adopting the posture having the highest accurate score, instead of clustering similar postures and taking the average value.

[2-6. Detection of multiple objects]
Another advantage of the algorithm in this embodiment is that it is easy to find multiple objects in the scene. In this embodiment, a number of postures are verified. These verification results can also be used when verifying the postures of a plurality of objects in a scene. For example, the following method can be used.
1) Rank all postures according to score.
2) Let P1 be the posture selected first. The point group belonging to P1 is deleted from the scene point group.
3) Refresh the integral image space.
4) In the new integrated image space, verify the posture group ranked in step 1).
5) Select a new posture P2 with the highest score. The point group belonging to P2 is deleted from the scene point group.
6) Return to 3).

[3. effect]
As described above, according to the present embodiment, a hash table using only carefully selected points (key points) with high reliability is created. Thereby, it is possible to speed up the posture estimation processing of the object. Further, the PPF is calculated for the point group around each point mi of the model point group, and the coordinate conversion parameters (local coordinates) are voted by the coordinate calculation. The best posture is determined from the posture with a large number of votes by the above-described verification processing. Thereby, recognition with high speed and high reliability is possible.

  The inventors conducted experiments on a large number of synthesized scenes and actual scenes, and verified the effects of this embodiment. For each input scene and model, the point cloud is downsampled and the model score is about 3000. The number of key points Nk was 8 to 20, and the posture of Nv = 8000 to 20000 was verified. Nr was 3% to 10%.

  Sixteen models including industrial and non-industrial parts were used. FIG. 13 shows these models. In FIG. 13, (a) to (f) are the results of Hinterstoisser et al. ("Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes", Asian conference on computer vision, pp. 548-562. , 2012). (G)-(j) are from Mian et al. ("Three-dimensional modelbased object recognition and segmentation in cluttered scenes", IEEE Transactions on Pattern Analysis & Machine Intelligence, Vol. 28 (10): pp. 1584-1601, 2006, and "On the repeatability and quality of keypoints for local feature-based 3d object retrieval from cluttered scenes.Int. Journal of Computer Vision, Vol. 89 (2): pp. 348-361, 2010) (K) to (p) are industrial parts.

  In each experiment, the result was determined to be correct when the error was less than the threshold. The threshold was 1/20 of the distance between the two most distant points in the model for displacement and 12 degrees for rotation.

  FIG. 14A and FIG. 14B are graphs showing simulation results for the composite scene. FIG. 14A shows the rate (recognition rate) correctly recognized for each model. FIG. 14B shows the calculation time for each model. For comparison, the same conditions were used when the method disclosed in Drost was used. The synthesized scene was generated so as to include a plurality of the same objects for each scene. The number of objects in each scene was 5 to 15, and 4 objects were detected.

  As can be seen from FIG. 14A and FIG. 14B, for non-industrial parts, a good recognition rate and recognition speed could be achieved by both the method of this embodiment and the Drost method. However, for industrial parts, both the recognition rate and the recognition speed were better in the method of this embodiment than in the Drost method.

  FIG. 15 is an image showing an example of a recognition result for an industrial part. (A) shows the result when using the method of Drost, and (b) shows the result when using the method of this embodiment. In the figure, the hatched location indicates the recognized location. It can be seen that the method of the present embodiment can more accurately recognize industrial parts.

  FIG. 16 is a graph showing the results of an experiment performed on an actual scene. FIG. 16 shows an example of the change in the recognition rate when the occlusion ratio is changed. The occlusion ratio is a value obtained by subtracting one from the value obtained by dividing the total surface area of the model in the scene by the total surface area of the model. As shown in FIG. 16A, according to the algorithm of the present embodiment, it is possible to suppress a decrease in the recognition rate particularly when the occlusion ratio is high, as compared with the Drost algorithm.

  The above-mentioned [2-3. Keypoint selection], [2-4. Posture estimation processing], [2-5. Posture verification], [2-6. The processing described in [Detection of Multiple Objects] is merely an example, and various modifications can be made for each processing. For example, the score calculation method in the key point selection process shown in FIG. 9 may be modified as appropriate. Further, instead of the object posture estimation method described with reference to FIGS. 11 to 12B, for example, a method disclosed in Drost may be applied. Even when a known method such as Drost is applied, if a hash table including only PPF information related to key points in the embodiment of the present invention is used, an effect of shortening the calculation time can be obtained.

  As described above, the present disclosure includes the methods, programs, and apparatuses described in the following items.

[Item 1]
A method for estimating the position and orientation of an object in a three-dimensional scene,
Obtaining model point cloud data representing a three-dimensional shape of the surface of the object;
Point pair features that define the geometric relationship of the point pairs selected from the model point group are calculated for each point pair, and the point pair features and the point pairs are associated with each other. Recording in one table;
Selecting a plurality of key points from the model point group based on the data of the model point group and the first table;
Constructing a second table in which only the data relating to the plurality of key points is left among the data recorded in the first table;
Obtaining scene point cloud data representing the surface shape of one or more objects included in the three-dimensional scene;
Estimating the position and orientation of the object in the three-dimensional scene based on the data of the scene point group and the second table;
Including methods.

[Item 2]
The step of selecting the plurality of key points includes:
A point mi in the model point group (i is an integer of 1 to Nm, Nm is the number of points included in the model point group) and a point mj (in the reference region whose distance from the point mi is shorter than a threshold value) j is an integer greater than or equal to 1 and less than or equal to M, and M is the number of points included in the reference area),
Searching for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount with reference to the first table;
Determining a coordinate transformation parameter for matching each of the searched point pairs to a pair of points mi and mj;
Among the points mi, when the coordinates of the model point group are converted using each of the determined coordinate conversion parameters, any one of the converted model point groups is located near all the points included in the reference area. Selecting a point mi that has the smallest number of points as a key point;
The method according to item 1, comprising:

[Item 3]
3. The method according to claim 1, wherein the number of the plurality of key points is 1 / 10,000 times or more and 1/2 times or less the number of points of the model point group.

[Item 4]
4. The method according to any one of items 1 to 3, wherein the number of the plurality of key points is not less than 1/1000 times and not more than 1/100 times the number of points of the model point group.

[Item 5]
Estimating the position and orientation of the object includes
The distance from each point si in the scene point group (i is an integer of 1 to Ns, Ns is the number of points included in the scene point group) between the two most distant points included in the model point group Selecting a point cloud Sr from a set of points that are less than or equal to the distance;
Calculating a point pair feature quantity between the point si and each point Srj (j is an integer of 1 to Ms) in the point group Sr;
Searching for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount with reference to the second table;
For each of the searched point pairs, determining a coordinate transformation parameter for matching to a pair of point si and point Srj, and determining a count for each posture indicated by the coordinate transformation parameter;
Selecting a plurality of candidate postures in descending order of the number of counts from the postures that have determined the count number;
Determining a posture having the highest degree of matching between the model point group and the scene point group as a final posture from the plurality of candidate postures;
The method according to any one of items 1 to 4, comprising:

[Item 6]
A computer program that causes a computer to execute the method according to items 1 to 5.

[Item 7]
A memory storing the computer program according to item 6, and
A processor for executing the computer program;
A processing apparatus comprising:

[Item 8]
The processing apparatus according to item 7,
An end effector;
A control circuit for controlling the end effector based on the position and orientation information of the object estimated by the processing device;
Robot equipped with.

[Item 9]
The processing apparatus according to item 7,
With robots,
A three-dimensional sensor for acquiring the scene point cloud data;
A control circuit for controlling the robot based on the position and orientation information of the object estimated by the processing device;
A robot system comprising:

  The 3D object posture estimation method according to the embodiment of the present disclosure can be used, for example, in the field of image recognition for a robot that performs bin picking.

DESCRIPTION OF SYMBOLS 100 Control apparatus 110 Control circuit 120 Image processing circuit 130 Memory 140 Input interface 150 Output interface 200 Robot 300 Three-dimensional sensor 400 Display 800 Bulk object

Claims (9)

A method for estimating the position and orientation of an object in a three-dimensional scene,
Obtaining model point cloud data representing a three-dimensional shape of the surface of the object;
Point pair features that define the geometric relationship of the point pairs selected from the model point group are calculated for each point pair, and the point pair features and the point pairs are associated with each other. Recording in one table;
Selecting a plurality of key points from the model point group based on the data of the model point group and the first table;
Constructing a second table in which only the data relating to the plurality of key points is left among the data recorded in the first table;
Obtaining scene point cloud data representing the surface shape of one or more objects included in the three-dimensional scene;
Estimating the position and orientation of the object in the three-dimensional scene based on the data of the scene point group and the second table;
Including methods. The step of selecting the plurality of key points includes:
A point mi in the model point group (i is an integer of 1 to Nm, Nm is the number of points included in the model point group) and a point mj (in the reference region whose distance from the point mi is shorter than a threshold value) j is an integer greater than or equal to 1 and less than or equal to M, and M is the number of points included in the reference area),
Searching for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount with reference to the first table;
Determining a coordinate transformation parameter for matching each of the searched point pairs to a pair of points mi and mj;
Among the points mi, when the coordinates of the model point group are converted using each of the determined coordinate conversion parameters, any one of the converted model point groups is located near all the points included in the reference area. Selecting a point mi that has the smallest number of points as a key point;
The method of claim 1 comprising:   3. The method according to claim 1, wherein the number of the plurality of key points is 1 / 10,000 times or more and 1/2 times or less the number of points of the model point group.   The method according to claim 1, wherein the number of the plurality of key points is 1/1000 times or more and 1/100 times or less the number of points of the model point group. Estimating the position and orientation of the object includes
The distance from each point si in the scene point group (i is an integer of 1 to Ns, Ns is the number of points included in the scene point group) between the two most distant points included in the model point group Selecting a point cloud Sr from a set of points that are less than or equal to the distance;
Calculating a point pair feature quantity between the point si and each point Srj (j is an integer of 1 to Ms) in the point group Sr;
Searching for at least one point pair having a point pair feature amount similar to the calculated point pair feature amount with reference to the second table;
For each of the searched point pairs, determining a coordinate transformation parameter for matching to a pair of point si and point Srj, and determining a count for each posture indicated by the coordinate transformation parameter;
Selecting a plurality of candidate postures in descending order of the number of counts from the postures that have determined the count number;
Determining a posture having the highest degree of matching between the model point group and the scene point group as a final posture from the plurality of candidate postures;
The method according to claim 1, comprising:   A computer program for causing a computer to execute the method according to claim 1. A memory storing the computer program according to claim 6;
A processor for executing the computer program;
A processing apparatus comprising: A processing apparatus according to claim 7;
An end effector;
A control circuit for controlling the end effector based on the position and orientation information of the object estimated by the processing device;
Robot equipped with. A processing apparatus according to claim 7;
With robots,
A three-dimensional sensor for acquiring the scene point cloud data;
A control circuit for controlling the robot based on the position and orientation information of the object estimated by the processing device;
A robot system comprising:

Images