JP6766229B2 - Position and posture measuring device and method - Google Patents

Description

The present invention relates to a technique for measuring the position and orientation of an object whose shape is known.

In recent years, with the development of computer vision, research on technologies for performing more complex tasks by robots has been actively conducted. Assembling industrial products is a typical example of a complicated task. In order for the robot to perform assembly work autonomously, it is necessary to grip the parts with an end effector such as a hand. In order to grip a part with a robot hand, first, the actual environment captured by the camera is photographed. Then, it is necessary to measure the position and orientation of the target object in the actual environment by fitting the model information of the target object to the captured image, formulate a movement plan based on the measurement result, and actually control the actuator. ..

In the fitting of model information in the real environment for gripping parts by such a robot, robustness is required for noise in the real environment and the complexity of the shape and texture of the parts, and various studies have been conducted so far. It was.

In the method described in Patent Document 1, as a method of robustly estimating the position and orientation of the camera, first, a plurality of initial positions and orientations are generated at predetermined sampling intervals within a range in which the position and orientation of the camera can be taken. Then, a method has been proposed in which the fitting is repeatedly calculated in each initial position and posture, and the fitting result with the best result is used as the final result of the fitting.

Japanese Unexamined Patent Publication No. 2012-26974

However, the method disclosed in Patent Document 1 is based on the premise that a reasonable range can be set as the range in which the position and orientation of the camera can be taken, and if this premise is not satisfied, the effect of robustness cannot be sufficiently obtained. There is a problem.

The present invention has been made in view of the above problems, and an object of the present invention is to perform robust position / posture measurement.

In order to achieve the above object, according to the present invention, the position / orientation measuring device includes an acquisition means for acquiring at least one approximate position / orientation of the target object from an image including the target object, and the acquired approximate position. determining means for determining the resolution of the position, based on the resolution of the coarse position and orientation of the determined and the obtained rough position and orientation, as an initial value for the derivation of the position and orientation of the target object, at least one position It includes a generation means for newly generating a posture candidate, and a derivation means for deriving the position / orientation of the target object based on the model information of the target object and the position / orientation candidate generated as the initial value.

According to the present invention, robust position / posture measurement can be performed.

It is a figure which showed the structure of the system in 1st to 7th Embodiment. It is a figure which shows the module structure of a program. It is a figure which shows the structure of 3D model information. It is a flowchart which shows the whole fitting processing procedure. It is a figure explaining the method of generating the position-posture candidate from the distribution of the approximate position-posture. It is a flowchart which shows the calculation procedure of a position | posture. It is a figure which shows the principle of the search of the corresponding point. It is a figure explaining the relationship between the projected image of an edge and the detected edge. It is a figure which shows the method of generating the initial position-posture by modifying the possible distribution of position-posture. It is a figure which shows the 3D model in a schematic position posture. It is a figure which shows the principle of finding the distribution of the edge detection point from the 3D model of the approximate position and orientation. It is a figure which shows the luminance image which contains noise in the background of the object to be measured. It is a figure which shows the principle of finding the distribution of edge detection points from a luminance image. It is a figure which shows the concept of the 6th Embodiment. It is a figure which shows the configuration example of the robot system in 8th Embodiment.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In addition, the embodiment described below shows an example when the present invention is concretely implemented, and is one of the specific examples of the configuration described in the claims.

[First Embodiment]
In the present embodiment, a brightness image and a distance image of the target object are simultaneously acquired by the imaging device and the projection device, the three-dimensional model information is aligned with the brightness image on a two-dimensional scale, and the distance image is three-dimensional. The method of applying the present invention when aligning with the scale of the above and evaluating at the same time with the two-dimensional and three-dimensional scales will be described. In addition, although the example of simultaneously aligning with a two-dimensional scale and a three-dimensional scale will be described here, the present invention can be applied in exactly the same manner even if it is only a two-dimensional scale or only a three-dimensional scale. is there.

In the setting of the initial position of the three-dimensional model information, the distribution (rough position / orientation) that the position / orientation can take is acquired by fully searching the luminance image taken by the imaging unit. Based on the acquired approximate position / posture distribution, a plurality of initial position / postures are generated, alignment is performed with each position / posture candidate, and the most matched position / posture is output as the final alignment result.

By doing so, the position and orientation that can be a local solution can be comprehensively verified, and the robustness of alignment can be improved. The details of the processing procedure will be described below. In all embodiments of the present invention, unless otherwise specified, alignment means aligning a two-dimensional or three-dimensional shape model of a measurement target object with the position, position and orientation of the measurement target object in an image. Indicates that (fitting is performed).

FIG. 1 is a diagram showing a system configuration of the position / posture measuring device according to the first embodiment.

The measurement target object 101 is a measurement target object as a target for measuring a position and a posture (positional posture). In the present embodiment, for the sake of simplicity, it is assumed that one measurement target object 101 is placed as shown in FIG. However, the position / orientation measurement process described below does not largely depend on the shape, number, and mounting form of the objects to be measured. For example, the present invention is also applicable to measure the position and orientation of a certain measurement target object in a state where a plurality of irregularly shaped measurement target objects are mixed and piled up.

The projection device 102 projects the pattern light onto the measurement target object 101. The details of the projection device 102 will be described later.

The image pickup device 103 is a device for capturing a still image or a moving image, and in the present embodiment, it is used to image the measurement target object 101 and the measurement target object 101 on which the pattern light is projected by the projection device 102. Then, the image pickup device 103 sends the captured image (captured image) to the position / orientation measuring device 104. The image pickup apparatus 103 will be described in detail later.

The position / orientation measuring device 104 is connected to the projection device 102 and the image pickup device 103, holds a program in which the specific processing of the present invention is described, and the built-in CPU executes the program. It is a device that controls the projection device 102 and the image pickup device 103 and calculates a desired position / orientation measurement result.

Next, an example of the functional configuration of the position / posture measuring device 104 will be described with reference to the block diagram of FIG.

The 3D model information 202 is the shape information of the object to be measured, which is the CAD model itself that can be handled by the 3D CAD software, or the 3D CAD model converted into a plurality of polygon elements used in the Computer Graphics field. In this embodiment, a three-dimensional geometric model that imitates the shape of the measurement target object 101 and is composed of polygon elements is used. The structure of the three-dimensional model information will be described with reference to FIG.

A three-dimensional geometric model composed of polygon elements is composed of components such as points, lines, and surfaces as shown in FIG. FIGS. 3A to 3C show the same three-dimensional geometric model.

In the model information of the three-dimensional geometric model composed of polygon elements, for each vertex of the three-dimensional geometric model illustrated in FIG. 3 (a), as shown in FIG. 3 (d), the index of the vertex and the said It manages the three-dimensional coordinate values of the vertices.

Further, in this model information, for each side of the three-dimensional geometric model illustrated in FIG. 3B, as shown in FIG. 3E, the index of the side and the index of the vertices at both ends of the side are used. Is in control.

Further, in this model information, as shown in FIG. 3 (f), the index of the polygon, the index of the polygon, and the index of the polygon are used for each surface (polygon) of the three-dimensional geometric model illustrated in FIG. 3 (c). It manages the normal vector of the polygon.

Such three-dimensional model information 202 is stored in an appropriate memory in the position / attitude measuring device 104 or an external memory accessible to the position / attitude measuring device 104.

The approximate position / posture acquisition unit 203 acquires the possible position / orientation (approximately position / posture) of the measurement target object 101. The specific method will be described later.

The position / posture candidate generation unit 204 generates a plurality of position / posture candidates based on the distribution of the approximate position / posture obtained by the position / posture distribution determination unit 203. The specific method will be described later.

The position / posture candidate selection unit 205 selects one or more position / posture candidates for which the position / posture should be calculated from the plurality of position / posture candidates. The calculation time is shortened by screening out multiple position / orientation candidates. The specific method will be described later.

The position / posture calculation unit 206 aligns the three-dimensional model with the brightness image and the distance image obtained by the image pickup device 103 with the position / posture indicated by the position / posture candidate selected by the position / posture selection unit 205, and finally. Calculate the position and posture.

The image pickup device 103 is a camera and photographs a work space in which the measurement target object 101 is arranged. In the description of the present invention, the terms "coordinate system" and "positional posture" mean the "coordinate system" of this camera and the "positional posture" in this camera coordinate system, respectively. On the other hand, the "model coordinate system" represents a coordinate system defined for each model used to represent the positions of points and surfaces of a three-dimensional model of a part. The internal parameters of the camera (focal length, principal point position, lens distortion parameters) are calibrated in advance by, for example, the methods described in the following documents.
Z. Zhang, “A flexible new technology for camera calibration,” IEEE Transitions on Pattern Analysis and Machine Intelligence, vol. 22, no. 11, pp. 1330-1334, 2000

The projection device 102 projects pattern light with a liquid crystal projector. The internal parameters of the projector (focal length, principal point position, lens distortion parameters) are also calibrated in advance using the same principle as camera calibration. Of course, the projection device 102 is not limited to the liquid crystal projector, and may be another type as long as it can project the pattern light. For example, it may be a projector using DMD (Digital Mirror Device) or LCOS.

The position / orientation measuring device 104 controls the projection device 102 and the imaging device 103 to acquire a luminance image and a distance image of the object to be measured. The distance image is generated by the principle of triangulation based on the shape of the pattern light reflected in the image. Although this method is generally called active stereo, the present invention is not limited to active stereo, and the present invention can be similarly implemented even if a distance image is generated by another method such as passive stereo. ..

FIG. 4 is a flowchart showing a procedure of fitting processing of the information processing apparatus composed of the above elements. The outline of the processing procedure will be described in this flowchart, and later, particularly important points in the practice of the present invention will be described in detail.

(Step S401)
In step S401, the imaging unit 103 photographs the measurement target object 101 in the work space and acquires the luminance image and the distance image of the measurement target object 101.

(Step S402)
In step S402, the luminance image is fully searched to determine the possible distribution (plural approximate position / orientation) of the position / orientation of the measurement target object 101. It is widely practiced to search the entire image and calculate the approximate position and orientation of the object 101 to be measured in the image. For example, in the following non-patent documents, an image of a three-dimensional model of a part is used by using pattern matching. A method of calculating the position and posture in the inside is disclosed.
Hiroto Yoshii, "Approximate position and orientation detection of loosely stacked parts using ensemble classification wood", Image Recognition and Understanding Symposium (MIRU2010), 2010

In such a method, pattern matching is performed a plurality of times in the image, the result obtained by one matching is recorded as one vote, and finally the vote result is aggregated and presented as the result of the total search. This one vote has 6 parameters of the position and orientation of the object to be measured as information (however, in order to improve the efficiency of calculation, some dimensions may be classified and represented by representative values. is there). In the present invention, a plurality of votes with the six parameters of the position and orientation are used as the possible position and orientation distribution of the target component.

FIG. 5A is an example of the distribution of the approximate position and posture obtained in this way. In the present embodiment, for the sake of simplicity, the position and orientation are represented by two parameters of x and y, but the same can be performed in the position and orientation of six parameters in the actual fitting.

The method of searching for the position and orientation of a specific component in the image in the present embodiment is not limited to the luminance image, and may be a method of searching for three-dimensional information such as a distance image.

Further, in the present embodiment, the case where the image is completely searched has been described, but in the practice of the present invention, it is not always necessary to completely search the obtained image. For example, if it is known that the parts are always in the same place in the workspace, there is no problem even if the part is partially searched only in that area.

(Step S403)
In step S403, a predetermined number of position / posture candidates are generated based on the distribution of the approximate position / posture obtained in step S402.

First, the mean vector μ and the covariance matrix Σ are calculated from the distribution of the approximate position and orientation, and the Gaussian distribution is applied. FIG. 5B is a contour line in which the probability density of the two-dimensional Gaussian distribution thus obtained is constant is superimposed on FIG. 5A.

Next, a predetermined number of position / orientation candidates are generated according to the obtained Gaussian distribution. The predetermined number may be calculated back from the permissible time required for the system and an appropriate number may be set. FIG. 5C is a position / posture candidate generated in this way.

In this example, the distribution of the approximate position and orientation obtained in step S402 (points plotted by black circles in FIG. 5C) and the newly generated position and orientation candidates (plots by black triangles in FIG. 5C). It is generated so that the total number of the points) is 20 and these 20 points are set as the initial position / posture (initial position / posture generation).

In addition, in this embodiment, it is considered that the position / posture listed as the approximate position / posture has a certain rationality, and it is adopted as a position / posture candidate. However, in the practice of the present invention, it is possible to adopt only the newly generated position / orientation without adopting the one listed as the approximate position / orientation.

In addition, although the Gaussian distribution was applied as the probability density model, another distribution (predetermined distribution function) may be used as long as the distribution of the obtained approximate position and orientation can be summarized, and it is simpler. A square distribution may be applied to.

(Step S404)
In step S404, one or more predetermined number of position / orientation candidates having a high score (representing the degree of matching between the three-dimensional model and the luminance image and the distance image) are selected from the plurality of position / orientation candidates obtained in step S403. (Initial position / posture selection). Specifically, the position / posture update process and steps S601 to S606 in the position / posture calculation described later are performed once, and the scores obtained as a result are compared and a predetermined number is extracted in order from the highest one.

The total time required for fitting is shortened by performing the position / orientation calculation process accompanied by repetition only for the position / orientation candidates selected in this way. Here, too, the predetermined number may be set by calculating back from the permissible time required for the system. Further, the score here may be any value as long as it represents the degree of matching between the three-dimensional model and the luminance image and the distance image.

For example, a method of obtaining an error vector (described later) from a three-dimensional model, a luminance image, and a distance image and using the reciprocal of the norm can be considered, but other indexes may be used. In the selection process described here, the position / posture update process and steps S601 to S606 are performed once (corresponding once) before the evaluation by the score, but the number of times does not necessarily have to be one. Any method can be used to confirm the probability of a high-speed and reasonable initial position and orientation.

For example, the score may be calculated without updating the position / posture even once, or the score may be calculated after updating the position / posture twice or more. As described above, innumerable methods can be considered for the specific procedure of the selection process, but it is not necessary to use any of them in the practice of the present invention.

(Step S405)
In step S405, the final position / posture is determined for each of one or more predetermined number of position / posture candidates selected in step S404. The details of the position / orientation calculation process here will be described with reference to FIG. FIG. 6 is a flowchart showing a detailed procedure of the position / posture calculation process regarding one initial position / posture shown in step S405.

In the present embodiment, the position and orientation are calculated by repeatedly correcting the position and orientation candidates (hereinafter, represented by s) of the object to be measured by the Gauss-Newton method by iterative calculation. Here, based on the edge detected on the image and the estimated position and orientation, the distance to the edge on the three-dimensional model projected on the image, and each point and position constituting the point cloud data of the distance image. Based on the orientation, the position orientation is optimized so that the total distance to the surface converted into the coordinate system of the 3D measuring device is minimized.

More specifically, the signed distance between a point and a straight line on a two-dimensional image, and the signed distance between a point and a plane in a three-dimensional space are slightly changed by the first-order Taylor expansion. Expressed as a linear function of. Then, by formulating and solving a linear simultaneous equation for a minute change in the position and orientation such that the signed distance becomes 0, the position and orientation are repeatedly corrected by obtaining the minute change in the position and orientation of the component.

(Step S601)
In step S601, initialization is performed. Here, the position / posture candidate selected in step S404 is set as the initial value of the position / posture described as the “current position / posture” thereafter.

(Step S602)
In step S602, the association is performed. First, based on the current position and orientation, the projection of each edge of the three-dimensional model onto the image and the coordinate conversion of each surface to the coordinate system of the three-dimensional measuring device are performed. After that, the edge and point cloud data are associated.

FIG. 7 is a diagram showing a method of associating edges.

FIG. 7A is a diagram in which the edges of the three-dimensional model are projected onto the luminance image. First, control points 702 are set on the projected edges 701 so as to be evenly spaced on the image, and a search line 703 is set in the normal direction of the projected edges 701 for each control point. A one-dimensional edge is detected within a predetermined range of the search line with the control point as the origin, and the point closest to the control point among the detected edges is held as the corresponding point 1005.

FIG. 7B is a graph in which the origin is the control point, the horizontal axis is the search line, and the vertical axis is the absolute value of the luminance gradient. The edge on the previous image is detected as the extreme value of the absolute value of the brightness gradient of the pixel value. Here, the point 704 in which the extreme value of the absolute value of the luminance gradient is larger than the predetermined threshold value 705 and is closest to the control point is set as the corresponding point.

Next, the point cloud data is associated. For each point in the point cloud data, the closest surface in the three-dimensional space is searched and associated.

(Step S603)
In step S603, the coefficient matrix and the error vector for solving the linear simultaneous equations are calculated. Here, each element of the coefficient matrix is a first-order partial differential coefficient with respect to a minute change in position and orientation. For the edge, the partial differential coefficient of the image coordinates is calculated, and for the point cloud, the partial differential coefficient of the three-dimensional coordinates is calculated. The error vector is the distance between the projected edge and the detected edge on the image for the edge, and the distance between the face and the point of the model in the three-dimensional space for the point cloud data.

FIG. 8 is a diagram illustrating the relationship between the projected image of the edge and the detected edge. In FIG. 8, the horizontal direction and the vertical direction of the image are u-axis and v-axis, respectively. The position of a certain control point (the point that divides each projected edge at equal intervals on the image) on the image (u ₀ , v ₀ ), and the inclination of the edge to which the control point belongs on the image are u Expressed as the inclination θ with respect to the axis. The slope θ is calculated as the slope of a straight line connecting the coordinates of both ends on the image by projecting the three-dimensional coordinates of both ends of the edge onto the image based on s. The normal vector of the edge on the image is (sinθ, −cosθ). Further, the image coordinates of the corresponding points of the control points are (u', v'). Here, the point (u, v) on the straight line passing through the point (u', v') and having a slope of θ is

Can be expressed as (θ is a constant). here,

And said.

The position of the control point on the image changes depending on the position and orientation of the object to be measured. Further, the degree of freedom of the position and orientation of the object to be measured is 6 degrees of freedom. That is, s is a six-dimensional vector, and is composed of three elements representing the position of the object to be measured and three elements representing the posture. The three elements representing postures are represented by, for example, Euler angles and three-dimensional vectors whose direction represents the axis of rotation and whose size represents the angle of rotation. The image coordinates (u, v) of the control points that change depending on the position and orientation can be approximated as in Equation 2 by the first-order Taylor expansion in the vicinity of (u ₀ , v ₀ ). However, Δs _i (i = 1, 2, ..., 6) represents a minute change in each component of s.

It can be assumed that the position of the control point obtained by the correct s on the image is on the straight line represented by Equation 1. By substituting u and v approximated by the number 2 into the number 1, the number 3 is obtained.

However,

(Constant).

The three-dimensional coordinates in the coordinate system of the three-dimensional measuring device can be converted into three-dimensional coordinates (x, y, z) in the model coordinate system according to the position and orientation s of the component to be measured. It is assumed that a certain point is converted into a point (x ₀ , y ₀ , z ₀ ) in the model coordinate system according to the approximate position and orientation. (X, y, z) changes depending on the position and orientation of the object to be measured, and can be approximated as in Equation 4 by the first-order Taylor expansion in the vicinity of (x ₀ , y ₀ , z ₀ ).

In step S602, the equation in the model coordinate system of the surface of the three-dimensional model associated with a certain point in the point cloud data is ax + by + cz = e (a ² + b ² + c ² = 1, a, b, c, e. (Constant). It is assumed that (x, y, z) transformed by the correct s satisfies the plane equation ax + by + cz = e (a ² + b ² + c ² = 1). Substituting equation 4 into the plane equation gives equation 5.

However,

(Constant).

Equation 3 holds for all the edges associated in step S602. The number 5 is for holds for all the point group data association is performed in S602, the linear simultaneous equation is satisfied about Delta] s _i such as the number 6.

Here, the number 6 is expressed as the number 7.

The partial differential coefficient for calculating the coefficient matrix J of the linear simultaneous equations of Equation 6 is calculated by, for example, the method disclosed in the following documents.
V. Lepetit and P.M. Fua,'Keypoint recognition randomized trees,' IEEE Transitions on Pattern Analysis and Machine Intelligence, vol. 28, no. 9, 2006.

(Step S604)
At step S604, the based on the number 7, it obtains the Δs using generalized inverse matrix of the matrix ^{J (J T · J) -1} · J T. However, since there are many outliers due to false positives in edge and point cloud data, the robust estimation method described below is used. In general, the error dr (eq) becomes large at the edge (point cloud data) which is an outlier. Therefore, the degree of contribution to the simultaneous equations of Equations 6 and 7 increases, and the accuracy of Δs obtained as a result decreases.

Therefore, a small weight is given to the data having a large error dr (eq), and a large weight is given to the data having a small error dr (eq). The weight is given by Tukey's function as shown in Equation 8, for example.

c ₁ and c ₂ are constants. The weighting function does not have to be Tukey's function, and any function such as Huber's function that gives a small weight to data with a large error and a large weight to data with a small error may be used.

The weights corresponding to the respective data (edge or point group data) and w _i. Here, the weight matrix W is defined as in Equation 9.

Weight matrix W is, all but the diagonal component is a square matrix of 0, the weights w _i enters the diagonal component. Using this weight matrix W, the equation 7 is transformed into the equation 10.

The correction value Δs is obtained by solving the equation 10 as in the equation 11.

(Step S605)
In step S605, the position / posture is updated by the correction value Δs of the position / posture calculated in step S604.

(Step S606)
In step S606, it is determined whether or not the position and posture have converged, and if it is determined that the position and posture have converged, the processing is terminated with the position and posture at that time as the fitting result. If it is determined that they have not converged, steps S602 to S605 are repeated until they converge. The convergence test is determined to have converged when it is determined that the correction amount in step S605 is equal to or less than a predetermined value and there is almost no change. The method of determining convergence is not limited to this method, and for example, the method may proceed to the next step assuming that the method has converged after being repeated a predetermined number of times.

(Step S406)
In step S406, the scores of one or more predetermined number of position / posture calculation results obtained in step S405 are compared, and one or more position / posture showing a high score is output as the final fitting result. The score here is also an index similar to that described in step S404.

As described above, according to the present embodiment, the distribution of the approximate position / orientation of the measurement target object is determined from the result of the full search of the measurement target object in the image, and a plurality of position / orientations that can be local solutions based on this distribution are determined. By comprehensively verifying the candidates, it is possible to perform robust position and orientation measurement even in a noisy environment or an object with a complicated shape.

The method for calculating the position and orientation of the object to be measured in the present invention is not limited to the Gauss-Newton method. For example, the calculation may be performed by the Levenberg-Marquardt method, which is more robust, or by the steepest descent method, which is a simpler method. Further, other nonlinear optimization calculation methods such as the conjugate gradient method and the ICCG method may be used.

In the present embodiment, the position / orientation measuring device 104 acquires the captured image directly from the imaging device 103, but the present invention is not limited to this, and for example, the captured image captured by the imaging device 103 is accumulated. It may be acquired from the existing memory. In this case, the image pickup device 103 does not necessarily have to be connected to the position / attitude measuring device 104.

In the present embodiment, the combination of the three-dimensional model information of the object to be measured and the image including the parts captured by the imaging device has been described as an example of the combination of the shape information to be fitted, but the combination of the shape information is not necessarily this combination. It does not have to be, and any information that can be aligned by taking correspondence between the shape information may be used. As an example of the shape information of the object, in addition to the three-dimensional model information and the image, the three-dimensional point cloud information obtained by measuring with a laser range sensor or the like can be considered.

[Second Embodiment]
In the first embodiment, a multidimensional Gaussian distribution is applied from the approximate position / orientation distribution of the object to be measured, and position / orientation candidates are generated according to the distribution. However, a method of generating position / orientation candidates without applying a probability density model such as a Gaussian distribution is also conceivable.

As an example of the method, in the second embodiment, a method of totaling the distribution of the approximate position / posture at a predetermined sampling interval and generating the position / posture candidate based on the totaling result will be described. The following is a process that replaces step S403 in the description of the first embodiment for determining the position-posture distribution described so far.

First, the range for totaling the position-posture distribution is determined. FIG. 9A is a diagram showing a state in which bins for sampling are divided by setting the range of the captured image as x1 to x2 and y1 to y2 when the position / orientation distribution shown in FIG. 5A is given. The width d of the bin is a constant set in advance, and for example, a value obtained by dividing the long side size of the object to be measured by a predetermined value can be considered. Here, the long side size is set by dividing a measurement target object having a length of 50 mm by a predetermined value of 10 and setting 5 mm as d. Alternatively, for example, instead of the long side size of the object to be measured, a value obtained by dividing the photographing range by a predetermined number may be set as d.

Next, the above bins are totaled within the set range. FIG. 9B is a diagram in which the number of positions and postures existing in each grid is counted as the number of votes and is shown in the center of the grid. From the grid counted in this way, the fourth bin from the left and the third bin from the top that obtained the maximum of 4 votes are selected.

Finally, position / orientation candidates are generated at predetermined intervals in the selected bin. An appropriate value may be set for the predetermined interval based on the number of position / posture candidates obtained by back calculation from the permissible time required for the system. Further, the intervals do not necessarily have to be equal, and the number of position / orientation candidates may be randomly generated inside the bin. FIG. 9C is a diagram showing a state when nine position / posture candidates are generated in this way.

For the position / orientation candidate generated in this way, the processes after step S404 are executed in the same manner as in the first embodiment.

As described above, according to the present embodiment, the first embodiment can be carried out more easily and at high speed by collecting and using the distribution of the approximate position and posture at predetermined sampling intervals.

[Third Embodiment]
In the first and second embodiments, a method of generating position / posture candidates based on the distribution of the approximate position / posture has been described, assuming that a plurality of approximate position / posture distributions that can be taken by the measurement target object can be obtained. However, the generation of position and orientation candidates is not limited to the method based on the distribution of approximate position and orientation. For example, when estimating the approximate position / orientation by template matching or the like, the resolution of the estimated position / orientation is determined according to the number of matching templates and the step width to be searched. In such a case, ambiguity occurs in the estimated position / orientation within the range of the resolution of the approximate position / orientation with respect to the actual position / orientation. In the third embodiment, in view of this point, a method of generating the position / posture candidate based on the resolution in the approximate position / posture estimation will be described. The following is a process that replaces step S403, which is a step of generating a position / posture candidate in the description of the first embodiment described so far.

First, the resolution of the approximate position and orientation is determined. Here, the resolution of the approximate position / posture represents the minimum step width for expressing the position and the posture. For example, when the position resolution is 1 [mm], the approximate position and orientation cannot be estimated with a value smaller than 1 [mm]. The resolution of the approximate position and orientation is determined by the sampling width and accuracy of the approximate position and orientation, the size of the object to be measured, and the like. For example, when a pattern matching-based method as described in step S402 of the first embodiment is used as a method for estimating the approximate position and posture, the posture resolution is the sampling of the posture registered as the pattern used for matching. Determined from the width. For example, when posture parameters are sampled in a geodetic dome shape in increments of 8 degrees and pattern matching is performed, the posture resolution is 8 degrees, which is the sampling width. The position resolution is obtained by calculating the range in which the position estimation can be mistaken based on the posture resolution. Specifically, the maximum deviation width d [mm] of the position based on the posture is obtained based on the resolution ω [rad] of the posture parameter and the long side size w [mm] of the object to be measured based on the following equation. , Set the calculated d as the position resolution.

In the present embodiment, the resolution of the approximate position posture is determined based on the posture sampling width and the size of the object to be measured by the method shown above. However, the method for determining the approximate position and posture is not limited to the method shown above. For example, the position resolution may be set based on the resolution of the image used for estimating the approximate position and orientation, or based on the amount of noise of the image sensor and the shape of the object to be measured by the method shown in the following non-patent documents. It may be set based on the ambiguity of the estimated position / orientation estimation.
W. Hoff and T. Vincent, "Analysis of head pose accuracy in augmented reality," IEEE Transitions on Visualization and Computer Graphics, vol. 6, no. 4, pp. 319-334, 2000.

Further, a person may set it in advance as a setting parameter. Any determination method may be used as long as the resolution as the minimum possible step width of the approximate position and orientation can be determined, and the determination method does not impair the essence of the present invention.

Next, a position / orientation candidate is generated based on the resolution of the approximate position / orientation. Specifically, position / posture candidates are generated by sampling the position / posture at predetermined intervals within the range of the resolution width of the approximate position / posture centered on the approximate position / posture. The sampling interval may be set to an appropriate value based on the number of position / orientation candidates obtained by back calculation from the permissible time required for the system. Further, the intervals do not necessarily have to be equal, and the number of position / orientation candidates may be randomly generated. As the approximate position / posture, the average value of the distribution of the approximate positions / postures obtained in step S402 of the first embodiment may be used. Further, when a single approximate position / posture is acquired instead of the distribution of the approximate position / posture, it may be used. The method of generating the position / posture candidate is not limited to the above method. For example, as in the first embodiment, the range that the position / posture can take is approximated to the Gaussian distribution from the resolution of the approximate position / posture, and the Gauss The position and orientation may be generated based on the distribution. In this case, the average vector μ of the multidimensional Gaussian distribution with respect to the 6 parameters of the position / orientation is obtained from the average of the distribution of the approximate position / orientation acquired in step S402 of the first embodiment. Then, by setting the variance-covariance matrix Σ of the Gaussian distribution from the resolution width of the approximate position / orientation, the multidimensional Gaussian distribution is calculated based on the resolution of the approximate position / orientation. Then, based on the obtained Gaussian distribution, a predetermined number of position / posture candidates are sampled in the same manner as in step S403 of the first embodiment to generate position / posture candidates. The position / orientation based on the Gaussian distribution may be generated by using a Gaussian distribution based on the resolution and a distribution that approximates the approximate position / orientation distribution as shown in the first embodiment. In this case, specifically, a distribution obtained by adding the Gaussian distribution based on the above-mentioned resolution to the Gaussian distribution obtained by the method of step S403 of the first embodiment is calculated. Then, based on the obtained distribution, the position / posture candidate may be generated by the same method as in step S403 of the first embodiment.

As described above, in the generation of the position / posture candidate, as long as the position / posture candidate can be generated based on the resolution of the approximate position / posture, there is no particular limitation on the generation method, and there is no particular problem regardless of which method is used.

For the position / orientation candidate generated in this way, the processes after step S404 are executed in the same manner as in the first embodiment.

As described above, according to the present embodiment, in a scene in which the approximate position / orientation and its resolution are known, a candidate position / orientation is generated based on an erroneous distribution of the position / orientation determined based on the resolution. Robust position and orientation measurement can be performed even when the approximate position and orientation are acquired with coarse resolution.

[Fourth Embodiment]
In the first embodiment, the approximate position / orientation distribution that the measurement target object can take is acquired by a full search. However, at the site where the robot actually grips and assembles the object to be measured, the parts are arranged in advance, and it may be considered that the position and orientation of the arranged parts are almost constant without a full search.

In the fourth embodiment, a method of determining the position / orientation distribution in which the model information can exist is described based on the edge distribution included in the three-dimensional model information of the target component in the approximate position / orientation. The following is a process that replaces step S402 in the description of the first embodiment for determining the position / posture candidate distribution described so far.

FIG. 10 is a diagram showing a measurement target object in a known approximate position and posture.

The three-dimensional model 1001 is a three-dimensional model arranged in a substantially positional position.

The center of gravity 1002 is the center of gravity of the three-dimensional model, and when pointing to the position of the three-dimensional model in the coordinate system of the system, it is indicated by the coordinates of this center of gravity.

The texture 1003 is a vertical stripe texture (texture information) of the object to be measured. When an object to be measured having such a texture 1003 is aligned using an edge, it mistakenly corresponds to an adjacent vertical stripe when searching for a corresponding point, and as a result, a direction perpendicular to the vertical stripe. There is a risk that the fitting will be misaligned.

In the present embodiment, it is considered that the possible position / orientation (rough position / orientation) of the object to be measured is distributed in the direction perpendicular to the vertical stripes as described above, and the initial value is comprehensively set in this direction. Set and compare the results of calculating the position and orientation of each. Then, the one that matches well is finally output as the correct position and posture.

The details will be described below.

FIG. 11A is a diagram showing a state in which an edge is searched from a control point on the three-dimensional model as a starting point. Control points 1101, 1102, 1103 are examples of control points. This is the same as the control point 702 described in the first embodiment. For each control point, a search line is taken in the direction perpendicular to the edge on which the control point is placed, and the edge distribution is searched. The edge search is performed by the same method as described in step S602 of the first embodiment.

The detection point 1104 is an edge detection point when searching from the control point 1101 as a starting point. The relative position between the detection point 1104 and the control point 1101 at this time is recorded as a position where the center of gravity 1002 is replaced with the starting point.

Next, the edge is searched starting from the control point 1102. In this case, nothing is detected because there is no edge on the search line as shown in the figure. In the same way, the edge is searched for all the control points, and the relative position between the detected point and the control point is recorded as the position replaced with the center of gravity as the starting point.

FIG. 11B illustrates the distribution of the edge detection points recorded in this way (the distribution of the positions of the edge detection points detected from each control point replaced with the center of gravity as the starting point). is there. The two points plotted by X in the figure indicate the two points detected from the control point 1101. The 1104 in FIG. 11 (a) and the 1104 in FIG. 11 (b) correspond to each other. The points plotted by the black triangles in the figure indicate the points detected from other control points. In FIG. 11B, the plot symbols are used properly for the sake of explanation, but in the subsequent processing, these are all treated as points showing the same distribution. The distribution of the edge detection points obtained in this way indicates how many internal points of the 3D model can be erroneously matched in which direction when searching for the corresponding points from the control points of the 3D model.

Therefore, there are many local solutions at the positions of this distribution, and the robustness of fitting can be improved by generating position / orientation candidates according to this distribution. Using this distribution as a position / orientation candidate in the first embodiment, the processes after step S403 are performed.

In the present embodiment, the edge of the texture has been described as an example of the edge that may be miscorresponded, but the effect of the present invention is not limited to the edge derived from the texture. There is no difference even if it is caused by features other than texture such as edges derived from the shape of the object to be measured.

A simple example of an edge that is easily mishandled due to its shape is a shape that has a repeating pattern such as the jagged teeth of a saw. In such a shape, according to the method of the present embodiment, it is considered that the possible position / postures are distributed in the direction in which the jagged teeth are lined up, and the position / posture candidates are comprehensively generated to be optimal. Position and posture can be determined.

In the present embodiment, it is considered that the dominant error factor of the fitting handled in the present invention is the erroneous correspondence of the edge, and the distribution of the position / orientation candidates is arranged in the two-dimensional plane of the captured image as the translational component of the position / orientation in the xy direction. Asked only about.

However, the determination of the distribution of the position / orientation candidates is not limited to two dimensions, and can be similarly performed in three dimensions. The same process as the edge search starting from the control point on the three-dimensional model described with reference to FIG. 11 may be performed in three dimensions. Specifically, it is conceivable to search for a surface in the normal direction of the surface on which the control points on the three-dimensional model ride and generate a distribution.

Further, it is possible to generate a distribution of position / orientation candidates by similarly searching for two-dimensional and three-dimensional rotations. Specifically, each control point on the 3D model is searched for an edge in the arc direction within a predetermined rotation amount range on a plurality of sampled rotation axes, and when a feature is detected, the rotation axis and the rotation amount at that time are voted. There is a way to do it. That is, the gist of the fourth embodiment of the present invention is to improve the robustness by obtaining position / posture candidates from the approximate position / posture and the information of the three-dimensional model and generating a plurality of position / posture candidates.

As described above, according to the present embodiment, in the scene where the approximate position / orientation is known, the distribution of the position / orientation candidates is determined from the known approximate position / orientation and the three-dimensional model information, and a local solution is obtained based on this distribution. By comprehensively verifying the obtained multiple position and orientation, it is possible to perform robust position and orientation measurement even for an object with a complicated shape.

[Fifth Embodiment]
In the fourth embodiment, the position / posture distribution that the measurement target object can take in the virtual environment is obtained from the approximate position / posture of the measurement target object and the three-dimensional model. Then, when generating a corresponding point between the three-dimensional model of the approximate position and orientation of the object to be measured and the captured image, the problem of erroneously corresponding to another edge inherent in the component has been addressed.

Another problem is that the edges distributed around the component in the approximate position and orientation of the image obtained by the imaging device are erroneously dealt with. In the fifth embodiment, in a situation where the approximate position / orientation of the arranged measurement target object is known, the position / orientation candidate is based on the origin of the three-dimensional model information in the approximate position / orientation of the feature included in the captured image. A method for dealing with the above problem will be described by making a determination based on the relative position distribution.

The following is a process that replaces step S402 in the description of the first embodiment for determining the position-posture distribution described so far.

FIG. 12 is a luminance image obtained by imaging. The frame 1201 is an outer frame of a luminance image obtained from the imaging unit. The measurement target object 1202 is a measurement target object reflected in the luminance image. The horizontal stripe 1203 is a horizontal stripe (noise) reflected in the luminance image. The noise referred to here indicates that it is not the measurement target object 1202 itself. That is, it may be another object existing in the background of the object to be measured, or it may be caused by the background itself.

In this embodiment, it is assumed that the noise is caused by another object. In such a case, when fitting the measurement target object 1202 using the edge, the background of the horizontal stripes caused by other objects above and below the measurement target object is erroneously dealt with when searching for the corresponding point. There is a risk that the fitting will be shifted in the vertical direction in the figure.

In the present embodiment, it is considered that the possible position / orientation of the parts is distributed in the vertical direction in the drawing as described above, and the position / orientation candidates are comprehensively set in this direction, and the position / orientation is calculated for each. The results are compared, and the one that matches well is finally output as the correct position and orientation.

The details will be described below. (Here, in order to make the effect of the method easy to understand, the background of horizontal stripes will be described as an example, but the present invention can be carried out by the same method for any texture.)
FIG. 13A is a diagram showing a method of searching for an edge starting from a control point on a three-dimensional model on a luminance image.

Coordinate axis 1301 indicates the coordinate system of the system. The projected image 1302 shows the projection of the 3D model edge onto the luminance image in the approximate position and orientation. The projection point 1303 is a point where the center of gravity of the three-dimensional model in the approximate position and orientation is projected onto the luminance image.

Control points 1304, 1305, and 1306 are examples of control points. These are the same as the control points 702 described in the first embodiment. For each of these control points, a search line is taken in the direction perpendicular to the edge on which the control point is placed, and the edge is searched. The edge search is performed by the same method as described in S602 of the first embodiment. The detection point 1307 is an edge detection point when searching from the control point 1301 as a starting point. The position relative to the control point 1301 at this time is recorded as a replaced position starting from the center of gravity 1303. Next, the edge is searched starting from the control point 1302. In this case, one detection point is recorded in the right direction. In the same way, the edge is searched for all the control points, and the relative position between the detected point and the control point is recorded as the position replaced with the center of gravity as the starting point.

FIG. 13B illustrates the distribution of the edge detection points recorded in this way (the distribution of the positions of the edge detection points detected from each control point replaced with the center of gravity as the starting point). is there. The three points plotted with X in the figure indicate the three points detected from the control point 1305. 1307 in FIG. 13 (a) and 1307 in FIG. 13 (b) correspond to each other. The points plotted by the black triangles in the figure indicate the points detected from other control points.

In FIG. 13B, the plot symbols are used properly for the sake of explanation, but in the subsequent processing, these are all treated as points showing the same distribution. The distribution of the edge detection points obtained in this way represents how many erroneous correspondence points can be detected in which direction when the correspondence points are searched from the control points of the three-dimensional model. Therefore, there are many local solutions at the positions of this distribution, and the robustness of the fitting can be improved by generating the initial position / orientation according to this distribution. This distribution is used as the position / orientation distribution in the first embodiment to perform the processing after S403.

In the present embodiment, the edge of the texture has been described as an example of an edge that may be miscorresponded, but the effect of the present invention is not limited to the edge derived from the texture, but is applied to the edge or the background derived from the shape of the three-dimensional model. There is no difference even if it is caused by features other than texture such as the derived edge.

In the present embodiment, as in the fourth embodiment, the position-posture distribution is obtained only for the translational component in the xy direction of the position-posture in the two-dimensional plane of the captured image. However, the determination of the position-posture distribution is not limited to two dimensions, and can be similarly performed in three dimensions.

The same process as the edge search starting from the control point on the three-dimensional model described with reference to FIG. 13 may be performed in three dimensions. Specifically, it is conceivable to search for corresponding points on the distance image at intervals of one point in the normal direction of the surface on which the control points are placed, starting from the control points on the three-dimensional model, and obtain a distribution of the corresponding points.

Further, it is possible to generate a position / orientation distribution by similarly searching for two-dimensional and three-dimensional rotations. Specifically, each control point on the 3D model is searched for an edge in the arc direction within a predetermined rotation amount range on a plurality of sampled rotation axes, and when a feature is detected, the rotation axis and rotation amount at that time are voted. There is a way to do it. In short, regardless of the dimension to be handled, instead of searching the entire captured image to obtain the position / orientation distribution, only the periphery of the 3D model arranged in the approximate position / orientation of the captured image should be searched. It is the gist of the fifth embodiment of the present invention to generate a plurality of initial position postures and improve the robustness.

As described above, according to the present embodiment, in the scene where the approximate position / orientation is known, the position / orientation distribution is determined by searching around the three-dimensional model in the approximate position / orientation in the image, and based on this distribution. By comprehensively verifying a plurality of positions and orientations that can be local solutions, it is possible to perform robust position and orientation measurement even in a noisy environment or an object with a complicated shape.

[Sixth Embodiment]
In the fourth to fifth embodiments, the position and orientation of the three-dimensional model projected on the image are obtained for the image as the shape information to which the position and orientation distribution is fitted. However, as supplemented in each embodiment, the present invention does not have a restriction that one of the shape information must be an image, and if there is a corresponding region between the two shape information, the shape of any form The present invention is applicable in fitting information.

For example, it is also possible to apply the present invention when fitting three-dimensional shape information to each other. As an example, here, an example of fitting two mesh models generated by photographing a certain target object from two viewpoints based on the shape of a region overlapping with each other will be described. That is, each mesh model overlaps in at least one area. It will be described that the present invention can be applied even when the three-dimensional shape information is fitted to each other in this way.

FIG. 14 illustrates an example of fitting the mesh model 1401 (first shape information) drawn by the dotted line to the mesh model 1402 (second shape information) drawn in practice in two dimensions. It is a figure to do. Reference numeral 1403 is a control point defined on the shape information 1401. At each control point, a candidate for a fixed section corresponding point is searched for in the normal direction of the surface on which the point rests. The corresponding point candidates found by searching are plotted in the three-dimensional space by plotting the vector from the control point to the corresponding point candidate centering on the center of gravity of the shape information 1401 as in FIG. 11 (b) or FIG. 13 (b). Generates a distribution of approximate position and orientation candidates. This distribution is used as the position / orientation distribution in the first embodiment to perform the processing after S403.

[7th Embodiment]
In the first to sixth embodiments, the position-posture distribution was generated by different methods. Actually, instead of implementing these independently, by using the distribution of the sum of some or all of them as the position-posture distribution and generating a plurality of initial position-postures in S403 (initial position-posture generation), Robust position and posture measurement can be realized.

[8th Embodiment]
A preferred application example of the information processing apparatus according to the present invention is to use it for measuring the position and orientation of a target component in a system in which an industrial robot arm performs operations such as gripping, moving, and releasing a component. Hereinafter, an application example of the information processing apparatus according to the present invention will be described with reference to FIG.

FIG. 15 is a schematic diagram showing an example of a schematic configuration of an information processing system including an information processing device according to a seventh embodiment of the present invention. 1501 is a PC, and 1502 is a camera and a projector. The PC 1501 and the camera and the projector 1502 are the same as the systems described in the first to sixth embodiments, store a program in which the processing procedure described above is described, and measure the position and orientation of the target component 1505. It is possible. The 1503 controls a robot arm with a robot controller. Reference numeral 1504 is a robot arm, which has a movable shaft including a rotation shaft and / or a translational movement shaft. The robot controller is connected to a PC, and commands to the robot arm convert what is received from the PC into specific control commands to move the robot arm.

In addition to the above, the PC in the present embodiment further includes the following processing procedure. First, the relative relationship between the system coordinate system described in the first embodiments 1 to 5, that is, the camera coordinate system and the robot coordinate system defined by the robot arm is maintained. The PC measures the position and orientation of the target component and converts the result from the coordinate system of the camera to the coordinate system of the robot. Next, the robot arm is moved to a position / orientation in which the part can be gripped via the robot controller based on the position / orientation of the converted part in the robot coordinate system.

As described above, according to the present embodiment, in the robot system in which the camera and the project are installed on the arm, the robot arm is processed based on the measurement result of the position and orientation of the component according to the present invention to grip the unknown component. Can be done.

[Other Embodiments]
The present invention is also realized by executing the following processing. That is, software (computer program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network or various storage media, and the computer (or CPU, MPU, etc.) of the system or device reads the program. It is a process to be executed.

<Effect of embodiment>
As described above, it is robust by acquiring the approximate position and orientation of the target object, generating new initial position and orientation candidates based on the acquired approximate position and orientation, and deriving the position and orientation based on them. It enables position and orientation measurement.

In the first embodiment, the distribution of the approximate position / orientation is determined from the result of a complete search of the measurement target object in the image, and a plurality of positions / orientations that can be local solutions are comprehensively verified based on this distribution. It has made it possible to perform robust position and orientation measurement even in a noisy environment or an object to be measured with a complicated shape.

In the second embodiment, a method of carrying out the first embodiment more simply and at high speed is shown by collecting and using the distribution of the approximate position and posture at predetermined sampling intervals.

In the third embodiment, in a scene in which the approximate position / orientation and its resolution are known, a candidate position / orientation is generated based on a distribution in which the position / orientation determined based on the resolution can be erroneous. In the fourth embodiment, which enables robust position / orientation measurement even when the position / orientation is acquired, the known approximate position / orientation and the three-dimensional model are used in the scene where the approximate position / orientation is known. By generating position / orientation candidates from information and comprehensively verifying multiple position / orientations that can be local solutions based on the generated position / orientation candidates, robust position / orientation measurement can be performed even for objects with complicated shapes. Made it possible to do.

In the fifth embodiment, in a scene where the approximate position / orientation is known, a position / orientation candidate is generated by searching around the three-dimensional model in the approximate position / orientation in the image, and the position / orientation candidate is generated based on the generated position / orientation candidate. By comprehensively verifying multiple position and orientation that can be local solutions, it is possible to perform robust position and orientation measurement even in a noisy environment or an object with a complicated shape.

In the sixth embodiment, an example is shown in which the present invention is applicable to the fitting of shape information of any form as long as there is a corresponding region between the two shape information. This makes it possible to fit data captured from a plurality of directions to an object whose shape data is unknown, such as a CAD model, and combine the data.

In the seventh embodiment, the union of the position / posture candidates generated by all or a part of the methods for generating the position / posture candidates shown in the first to fifth embodiments is a plurality of position / posture candidates. It is possible to perform more robust position and posture measurement by using it as.

In the eighth embodiment, in a robot system in which a camera and a project are installed on an arm, an unknown part can be grasped by processing the robot arm based on the measurement result of the position and orientation of the part according to the present invention.

<Definition>
In the description of the present invention, as the combination of the shape information to be fitted, the combination of the three-dimensional model information and the image including the measurement target object taken by the camera is used in the first to fifth embodiments, and in the sixth embodiment And the combination of three-dimensional mesh models was explained as an example. However, this combination does not have to be the case, and any information that can be aligned (fitted) by correspondingly among the shape information may be used. As an example of the shape information of the object, in addition to the three-dimensional model information and the image, the three-dimensional point cloud information obtained by measuring with a laser range sensor or the like can be considered.

The projector is not limited to the liquid crystal projector, and may be another type as long as it can project the pattern light. For example, it may be a projector using DMD (Digital Mirror Device) or LCOS.

In the position / orientation calculation, an example of simultaneously aligning on a two-dimensional and three-dimensional scale has been described, but the present invention is applied in exactly the same manner even if it is only a two-dimensional scale or only a three-dimensional scale. It is possible.

In the position / orientation calculation, the method of calculating the position / orientation of the component to be measured is not limited to the Gauss-Newton method. For example, the calculation may be performed by the Levenberg-Marquardt method, which is more robust, or by the steepest descent method, which is a simpler method. Further, other nonlinear optimization calculation methods such as the conjugate gradient method and the ICCG method may be used.

In determining the distribution of the approximate position / orientation, the image when searching for the position / orientation of a specific part in the image is not limited to the luminance image, but may be a method of searching for three-dimensional information such as a distance image. Good.

In the determination of the distribution of the approximate position and orientation, the case where the image is completely searched has been described, but in the practice of the present invention, it is not always necessary to completely search the obtained image. For example, if it is known that the parts are always in the same place in the work space, there is no problem even if the part is partially searched only in that area.

In the position-posture distribution determination described in the fourth embodiment of the present invention, the search range of the position-posture distribution is not limited to only the translational component in the xy direction in the two-dimensional plane. The determination of the position-posture distribution is not limited to two dimensions, and can be similarly performed in three dimensions. The same process as the edge search starting from the control point on the three-dimensional model described with reference to FIG. 11 may be performed in three dimensions.

Specifically, it is conceivable to search for a surface in the normal direction of the surface on which the control points on the three-dimensional model ride and generate a distribution. Further, it is possible to generate a position / orientation distribution by similarly searching for two-dimensional and three-dimensional rotations. Specifically, each control point on the 3D model is searched for an edge in the arc direction within a predetermined rotation amount range on a plurality of sampled rotation axes, and when a feature is detected, the rotation axis and the rotation amount at that time are voted. There is a way to do it.

In the approximate position-posture distribution determination described in the fifth embodiment of the present invention, the search range of the position-posture distribution is not limited to only the translational component in the xy direction in the two-dimensional plane. The determination of the position-posture distribution is not limited to two dimensions, and can be similarly performed in three dimensions. The same process as the edge search starting from the control point on the three-dimensional model described with reference to FIG. 13 may be performed in three dimensions. Specifically, it is conceivable to search for corresponding points on the distance image at intervals of one point in the normal direction of the surface on which the control points are placed, starting from the control points on the three-dimensional model, and obtain a distribution of the corresponding points. Further, it is possible to generate a position / orientation distribution by similarly searching for two-dimensional and three-dimensional rotations.

In the determination of the distribution of the position / orientation candidates described in the fourth embodiment and the fifth embodiment of the present invention, the feature of the shape information is not limited to the edge derived from the texture, but is based on the association of S602. Any feature can be used. Features other than texture, such as edges derived from the shape of the 3D model, may be used.

In the generation of position / posture candidates, we considered that the position / posture listed as the approximate position / posture distribution has a certain rationality, and adopted it as a position / posture candidate. However, in the practice of the present invention, it is possible to adopt only the newly generated position / orientation candidate without adopting the one listed as the approximate position / orientation distribution.

In the position and orientation candidate generation, the Gaussian distribution was applied as the probability density model, but in reality, any distribution may be used as long as the obtained distribution can be summarized, or a square distribution can be applied more easily. You may.

In the generation of the position / orientation candidates described in the second embodiment of the present invention, when the initial values are generated at predetermined intervals in the selected bins, the intervals do not necessarily have to be equalized, and only the number of position / orientation candidates It may be randomly generated inside the bin.

In the position / orientation candidate selection, the score may be any value indicating the degree of matching between the three-dimensional model and the luminance image and the distance image. For example, a method of obtaining an error vector from a three-dimensional model, a luminance image, and a distance image and using the reciprocal of the norm can be considered, but other indexes may be used.

In the position / posture candidate selection, the position / posture update processing steps S601 to S606 are performed once before the position / posture candidate is evaluated by the score, but the number of times does not necessarily have to be once, and the position / posture candidate is optimal. Any method can be used to quickly confirm the probability of convergence to the solution. For example, the score may be calculated without updating the position / posture even once, or the score may be calculated after updating the position / posture twice or more. As described above, innumerable methods can be considered for the specific procedure of the selection process, but it is not necessary to use any of them in the practice of the present invention.

Claims (10)

An acquisition means for acquiring at least one approximate position / orientation of the target object from an image including the target object, and
A determination means for determining the resolution of the acquired approximate position / orientation, and
Based on the resolution of the coarse position and orientation of the determined and the obtained rough position and orientation, as an initial value for the derivation of the position and orientation of the target object, and generating means for newly generating at least one pose candidate ,
A position / posture measuring device including a derivation means for deriving the position / posture of the target object based on the model information of the target object and the position / posture candidate generated as the initial value. The position / orientation measuring device according to claim 1, wherein the acquisition means acquires the approximate position / orientation of the target object in the image by performing pattern matching on the image. The position / orientation measuring device according to claim 2, wherein the acquisition means performs pattern matching by searching in the image using model information of the target object. The approximate position / orientation acquired by the acquisition means is generated based on a predetermined resolution, and the generation means is at least based on the acquired approximate position / orientation and the resolution at which the approximate position / orientation is generated. The position / posture measuring device according to any one of claims 1 to 3, wherein one position / posture candidate is newly generated. The generation means is characterized in that at least one position / posture candidate is newly generated by changing at least one of the positions or the postures within the range of the predetermined resolution with respect to the acquired approximate position / posture. The position / posture measuring device according to claim 4. The position / posture measuring device according to claim 4 or 5, wherein the resolution is a minimum step width for expressing the position and posture of the approximate position / posture. One of claims 1 to 6, wherein the generation means newly generates at least one position / orientation candidate based on the acquired approximate position / orientation and the shape information of the target object. The position / orientation measuring device described in 1. The position / posture measuring device according to any one of claims 1 to 7.
A gripping means for gripping the target object and
An information processing device including a control means for controlling the gripping means based on the position / posture of the target object measured by the position / posture measuring device. An acquisition step of acquiring at least one approximate position / orientation of the target object from an image including the target object, and
The determination process for determining the resolution of the acquired approximate position and orientation, and
Based on the resolution of the coarse position and orientation of the determined and the obtained rough position and orientation, as an initial value for the derivation of the position and orientation of the target object, a generation step of newly generating at least one pose candidate ,
A position / posture measurement method including a derivation step of deriving the position / posture of the target object based on the model information of the target object and the position / posture candidate generated as the initial value. A computer program for causing a computer to execute the position / posture measurement method according to claim 9.