JP2016148649A - Information processing apparatus, control method therefor, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing apparatus configured to associate a model feature of a shape model with a measurement data feature on a two-dimensional image, with higher accuracy.SOLUTION: An information processing apparatus includes: a model input section 110 for inputting a shape model representing a shape of an object to be measured; a predicted value calculation section 120 which calculates a predicted image deterioration value of a two-dimensional image obtained by imaging the object; a search section 140 for searching for a candidate, for association of a local line feature, of the shape model of the object input from the model input section 110, in the two-dimensional image input from the image input section 130; an evaluation value calculation section 150 which estimates the magnitude of blur and slur of the two-dimensional image obtained in the image input section 130; an accuracy calculation section 160 which calculates accuracy on association of edge features held in the search section 140; and a correlation section 170 which calculates position and attitude of the object, on the basis of the edge feature on the two-dimensional image associated with the local line features in the shape model of the object and the accuracy calculated in the accuracy calculation section 160.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

  Along with the development of robot technology in recent years, robots are instead performing complicated tasks that have been performed by humans, such as assembly of industrial products. Such a robot performs assembly by gripping a part with an end effector such as a hand. In order for the robot to grip the part, it is necessary to measure the relative position and orientation between the part to be gripped and the robot (hand). The measurement of the position and orientation of such objects is not only for grasping parts by the robot, but also for self-position estimation for the robot to move autonomously, and alignment of the real space (real object) and virtual object in augmented reality It is applied for various purposes.

  As a method for measuring the position and orientation of an object, there is a method using a model feature of a three-dimensional shape model and a measurement data feature in a two-dimensional image obtained from an imaging device such as a camera. In Non-Patent Document 1, the position and orientation of an object are measured by applying a projection image of a three-dimensional shape model of the object represented by a set of line segments to an edge that is a measurement data feature on the two-dimensional image. A method is disclosed. In this method, a line segment in a three-dimensional shape model is projected on a two-dimensional image based on a rough position and orientation given as known information, and control is arranged discretely on the projected line segment. Edges corresponding to each of the points are detected on the two-dimensional image. Based on the correspondence between the control point (model feature) and the edge (measurement data feature) obtained in this way, the sum of squares of the distance on the image between the projected image of the line segment to which the control point belongs and the corresponding edge. The final position and orientation can be obtained by correcting the approximate position and orientation so that is minimized.

T. Drummond and R. Cipolla, “Real-time visual tracking of complex structures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.7, pp.932-946, 2002. RY Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE Journal of Robotics and Automation, vol.RA-3, no.4, 1987. (Referenced in form) Baba, Asada, Amano, “Trial of Image Synthesis with Calibrated Computer Graphics: Generation and Verification of Arbitrarily Focused Images Based on Camera Calibration”, Transactions of Information Processing Society of Japan 39 (7), 2180-2188,1998 (see the embodiment) ) Y. Hel-Or and M. Werman, “Pose estimation by fusing noisy data of different dimensions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.17, no.2, pp.195-201, 1995. Referenced in)

  However, in the above-described method, when an erroneous correspondence occurs between a model feature (control point) and a measurement data feature (edge), the object position and orientation estimation process fails or the position and orientation estimation is performed. There is a problem that accuracy is lowered. The present invention has been made in view of the above problems, and an object thereof is to realize a more accurate association between a model feature of a shape model and a measurement data feature on a two-dimensional image.

In order to achieve the above object, an information processing apparatus according to an aspect of the present invention includes the following arrangement. That is,
Based on a shape model representing the shape of the measured object, predicting means for predicting image degradation of the measured object that occurs in an image obtained by imaging the measured object by the imaging means;
Search means for searching for a measurement data feature corresponding to a model feature of the shape model for a two-dimensional image obtained by imaging the object to be measured by the imaging means;
Evaluation means for evaluating image deterioration using the two-dimensional image for the measurement data feature searched by the search means;
A calculation unit that calculates the accuracy of correspondence between the model feature and the measurement data feature based on the image degradation predicted by the prediction unit and the image degradation evaluated by the evaluation unit;
Collation means for collating the shape model with the object to be measured in the two-dimensional image based on the accuracy of correspondence between the model feature and the measurement data feature.

  According to the present invention, it is possible to realize a more accurate association between the model feature of the shape model of the object to be measured and the measurement data feature on the two-dimensional image. Therefore, for example, the position and orientation of the measured object can be estimated with higher accuracy.

The figure which shows the structure of the information processing apparatus in 1st Embodiment. The figure which illustrates the three-dimensional shape model of the measurement target object in embodiment. The flowchart showing the process sequence which measures a position and orientation in 1st Embodiment. The figure explaining the edge detection from an image. The flowchart showing the procedure which calculates the image degradation predicted value in 1st Embodiment. The flowchart showing the procedure which calculates the image degradation evaluation value of a two-dimensional image in 1st Embodiment. 6 is a flowchart for explaining a processing procedure of association weight calculation in the first embodiment. The flowchart explaining the process sequence of a position and orientation calculation in 1st Embodiment. The figure explaining the relationship between the projection image of a line segment, and the detected edge. The figure which shows the function structural example of the information processing apparatus in 3rd Embodiment. The figure which shows the structural example of the robot system in 4th Embodiment.

  Hereinafter, some preferred embodiments of the present invention will be described with reference to the accompanying drawings.

<First Embodiment>
In the first embodiment, the local line feature and the measurement data feature are associated with each other by using the blur and the blur of the image obtained from the two-dimensional image. That is, for each local line feature, the blur and blur size obtained in the two-dimensional image are predicted, and compared with the blur and blur size actually measured from the two-dimensional image. Correlate with a high contribution. In the first embodiment, the magnitude of blur and blur is determined by simulation from the shape represented by the shape model imitating the object to be measured and the relative moving direction and speed between the imaging device and the object to be measured that are determined in advance. Predict the size of blur and blur by calculating.

  FIG. 1A is a block diagram illustrating a hardware configuration example of the information processing apparatus 1 according to the present embodiment. In FIG. 1A, the CPU 101 implements various controls in the information processing apparatus 1 by executing a program stored in the ROM 102 or the RAM 103. The ROM 102 stores various programs and various data including programs executed when the CPU 101 is activated. The RAM 103 functions as a main memory for the CPU 101. For example, an application program read from the external storage device 105 is expanded in the RAM 103 for execution by the CPU 101. The display unit 104 performs various displays under the control of the CPU 101. The external storage device 105 is composed of, for example, a hard disk or the like, and accumulates application programs, shape model data, and two-dimensional images to be processed. The interface 106 communicates with an external device. For example, the imaging apparatus 20 is connected to the interface 106 and receives a two-dimensional image from the imaging apparatus 20. The bus 107 enables communication between the above-described components.

  FIG. 1B is a block diagram illustrating a functional configuration example of the information processing apparatus 1 in the present embodiment. Each function shown in FIG. 1B is realized by the CPU 101 executing a program stored in the ROM 102 and / or a program expanded from the external storage device 105 to the RAM 103. Obviously, some or all of the functional units may be realized by dedicated hardware. 1B, the information processing apparatus 1 includes a model input unit 110, a predicted value calculation unit 120, an image input unit 130, a search unit 140, an evaluation value calculation unit 150, an accuracy calculation unit 160, and a collation unit 170. . Moreover, the structure shown in FIG. 1 is a structure used as the application example of the information processing apparatus of this invention. Hereinafter, each functional unit of the information processing apparatus 1 will be described.

  The model input unit 110 inputs a shape model (a three-dimensional shape model stored in the external storage device 105 in this embodiment) representing the shape of the object to be measured. The shape model in this embodiment is a local 3D line segment information (hereinafter referred to as local line feature information) on an object contour composed of a 3D position and a 3D line segment direction as shown in FIG. It shall be comprised by. The shape model is provided to the search unit 140 via the model input unit 110.

  The predicted value calculation unit 120 calculates an image degradation predicted value of a two-dimensional image obtained by photographing the measurement object. In the present embodiment, the size D and the image of the blur due to defocusing are obtained using the three-dimensional shape model of the object to be measured, the approximate position / posture value, and the internal parameters of the imaging device that captured the two-dimensional image to be processed. A blur size B caused by parallel movement on the plane is set as an image degradation prediction value. Details of the process of acquiring the blur size D, the blur size B, and the predicted image degradation value will be described later.

  The image input unit 130 acquires a two-dimensional image obtained by photographing the object to be measured. The acquired two-dimensional image may be a gray image or a color image. In the present embodiment, a two-dimensional image is acquired from an external storage device (not shown) that stores a two-dimensional image captured in advance, but a two-dimensional image obtained by using the imaging device 20 is input as an image. The unit 130 may acquire directly from the imaging device 20. In any case, the two-dimensional image is held in association with the internal parameters of the imaging device 20 that captured the image.

  The search unit 140 searches the candidate for correspondence of the local line feature of the shape model of the measured object input from the model input unit 110 in the two-dimensional image input from the image input unit 130. This search is performed using the approximate values of the positions and orientations of all the local line features that make up the shape model, and the internal parameters of the imaging device that captured the two-dimensional image. In this embodiment, an edge feature in a two-dimensional image is detected as a corresponding candidate (measurement data feature). The edge feature is a point that becomes an extreme value of the density gradient detected by the differential filter. FIG. 4 is a diagram for explaining edge detection in the present embodiment. A projected image on the two-dimensional image of each line segment constituting the shape model is calculated by using the approximate position and orientation of the object to be measured and the calibrated internal parameters of the imaging device 20 that captured the two-dimensional image.

  As shown in FIG. 4A, the projected image of the line segment is also a line segment on the two-dimensional image (projected line segment 401). Next, control points 402 are set on the projected line segments 401 so as to be equally spaced on the two-dimensional image, and 1 is set in the normal direction of the projected line segments 401 for each set control point 402. Dimensional edge detection is performed (FIG. 4A). That is, edge detection is performed along a search line 403 having a normal direction of the projected line segment 401 through the control point 402 set on the projected line segment 401. Since the edge 404 of the two-dimensional image is detected as an extreme value of the density gradient of the pixel value, as shown in FIG. 4B, when there are a plurality of edges in the vicinity, a plurality of edges for one control point are present. May be detected. In this embodiment, all detected edges are held as hypotheses.

  The evaluation value calculation unit 150 estimates the blur size and blur size of the two-dimensional image obtained by the image input unit 130. In this embodiment, by applying a function representing the luminance change of the edge when blurring / blurring is applied to the luminance change of the pixel along the direction orthogonal to the edge in the two-dimensional image, each edge feature is applied. An image degradation evaluation value is calculated. The process for detecting the edge in the two-dimensional image is the same as the edge detection by the search unit 140. Details of the processing will be described later.

  The accuracy calculation unit 160 calculates the accuracy for each correspondence of the edge features held by the search unit 140 as a hypothesis of association with the local line feature of the shape model of the measured object. The accuracy is calculated based on the predicted image deterioration value obtained by the predicted value calculation unit 120 and the image deterioration evaluation value obtained by the evaluation value calculation unit 150. The details of the processing will be described later. . The matching unit 170 determines the position and orientation of the measured object based on the edge feature on the two-dimensional image corresponding to each local line feature in the shape model of the measured object and the accuracy calculated by the accuracy calculating unit 160. calculate. Details of the position and orientation calculation processing by the matching unit 170 will be described later.

  FIG. 3 is a flowchart showing a processing procedure for associating edge features with local line features in the first embodiment and calculating the position and orientation of the measured object on the two-dimensional image. First, the model input unit 110 inputs a shape model of an object to be measured and imports it into the information processing apparatus 1 (step S301). Next, the predicted value calculation unit 120 calculates an image deterioration predicted value of the two-dimensional image (step S302). For the calculation of the predicted image degradation value, the approximate values of the positions and orientations of all the local line features constituting the shape model of the object to be measured, the relative movement direction / velocity between the imaging device and the object to be measured, and a two-dimensional image The internal parameters of the imaging device at the time of acquisition are used. In the present embodiment, the approximate position or orientation value where the object is placed is used as the approximate value of the position and orientation of the local line feature. Details of the processing for calculating the predicted image degradation value of the two-dimensional image will be described later with reference to FIG.

  Next, the image input unit 130 inputs a two-dimensional image and takes it into the information processing apparatus 1 (step S303). The search unit 140 searches for a corresponding candidate for the local line feature of the shape model from the two-dimensional image input by the image input unit 130 (step S304). The corresponding candidate search method is as described above with reference to FIG. 4, and the approximate values of the positions and orientations of all the local line features of the shape model input by the model input unit 110 and the imaging device at the time of capturing a two-dimensional image Based on the internal parameters.

  The evaluation value calculation unit 150 estimates the blur size and blur size of the two-dimensional image obtained by the image input unit 130, and calculates an image degradation evaluation value (step S305). More specifically, the evaluation value calculation unit 150 performs edge detection from the two-dimensional image obtained by the image input unit 130. Then, the evaluation value calculation unit 150 applies a function representing the luminance change of the edge when blurring / blurring occurs with respect to the luminance change of the pixel along the direction orthogonal to the detected edge feature. The blur / blur width is estimated to obtain an image degradation evaluation value. Details of this processing will be described later. Next, the accuracy calculation unit 160 uses the image degradation prediction value obtained by the prediction value calculation unit 120 and the image degradation evaluation value obtained by the evaluation value calculation unit 150, and the local line feature obtained by the search unit 140. And the accuracy of the correspondence between the edge features (step S306). Details of the accuracy calculation processing will be described later with reference to FIG. Then, the collation unit 170 calculates the position and orientation of the measurement target object by applying the edge feature having the maximum calculated accuracy to the local line feature of the shape model (step S307).

  FIG. 5 is a flowchart showing the process of calculating the predicted image degradation value in step S302. In the present embodiment, image degradation due to the spread of the image due to blurring and / or blurring of the two-dimensional image of the model feature is predicted. Hereinafter, the image degradation predicted value calculation processing by the predicted value calculation unit 120 will be described with reference to the flowchart of FIG. First, the predicted value calculation unit 120 was used for the approximate values of the positions and orientations of all the local line features of the shape model, the relative moving direction / velocity between the imaging device and the object to be measured, and the shooting of a two-dimensional image. Internal parameters are acquired (step S501). As the relative movement direction and speed between the imaging apparatus and the object to be measured in the present embodiment, for example, based on design data when the object to be measured is arranged on a device (for example, a belt conveyor) that translates in one axial direction. Then, the movement amount and movement direction of the translational movement are acquired.

Next, the predicted value calculation unit 120 selects a local line feature in order to calculate the image degradation predicted value σ ₀ from the shape model (step S502). Then, the predicted value calculation unit 120 calculates the blur size D predicted for one selected local line feature using the following equation [Equation 1] (step S503).

In the above formula [Equation 1], f is the focal length of the taking lens, L _o is the focus position of the virtual viewpoint, L _n is the distance from the virtual viewpoint to the model point, F is the aperture value of the lens, and Δd is the size of the pixel Represents The focal length f and aperture value F of the photographic lens are acquired by referring to the specifications of the imaging device to be used. Ln is calculated from the approximate position and orientation input in step S501. Internal parameters such as the focal length, principal point position, and lens distortion parameters of the camera lens used for shooting are calibrated in advance by referring to the specifications of the equipment used or by the method described in Non-Patent Document 2. It is acquired by keeping.

  Next, the predicted value calculation unit 120 calculates the size of blur predicted for the same local line feature that predicted the size of blur in step S503 (step S504). In the present embodiment, the amount of movement of each local line feature on the two-dimensional image during the exposure time is used as the amount of blur. Specifically, the movement amount on the image plane during the exposure time of the projected local line feature, which is detected from the image obtained by projecting the shape model onto the two-dimensional image, is used. In step S504, the Jacobian of the projected local line feature obtained from the model projection image is calculated, and based on the relative moving direction / velocity between the Jacobian of the projected local line feature and the imaging device and the measured object input in step S502. The amount of blurring of the projected local line feature is calculated.

The Jacobian of the projected local line feature is a value representing a rate at which the projected local line feature on the image changes when the parameter of the position and orientation 6 degrees of freedom changes minutely. Based on the approximate position and orientation s of the object to be measured, the blur amount of the edge feature corresponding to the projected local line feature is (du, dv), and the normal direction of the projected local line feature is (n _u , n _v ) (unit vector ), The signed distance err _2D between the correspondences can be calculated by the following equation [Equation 2]. Note that du = u′−u and dv = v′−v.

Here, the position and orientation s of the measured object are 6-dimensional vectors, and three elements (s ₁ , s ₂ , s ₃ ) representing the position of the target object and three elements (s ₄ , s) representing the attitude. ₅ and s ₆ ). The three elements representing the posture are represented by, for example, expression by Euler angles or a three-dimensional vector in which the direction represents the rotation axis passing through the origin and the norm represents the rotation angle. The Jacobian of the projected local line feature is calculated by the partial differentiation of the inter-corresponding distance err _2D with the parameters of the position and orientation s as shown in the following equation [Equation 3].

Ｂはスカラーであり、露光時間中に画像平面上で、投影局所線特徴の２次元位置が移動する量を表す。以上の処理を全ての局所線特徴に対して行い、全ての局所線特徴に対してブレ量を算出する。 As described above, the Jacobian of the projected local line feature selected in step S502 is calculated. Using the Jacobian of the projected local line feature, between the local line feature and the edge feature in the two-dimensional image generated when the target object moves at the relative position / posture speed V during the exposure time t _i of the two-dimensional image. Can be calculated by the following equation [Equation 4].

B is a scalar and represents the amount by which the two-dimensional position of the projected local line feature moves on the image plane during the exposure time. The above processing is performed for all local line features, and the blur amount is calculated for all local line features.

Next, in the predicted value calculation unit 120, the image degradation predicted value σ ₀ of the local line feature selected in step S502 is calculated from the blur size D calculated in step S503 and the blur size B calculated in step S504. Calculation is performed using the following equation [Equation 5] (step S505).

The processes in steps S502 to S505 are repeated until the calculation of the image degradation prediction value σ ₀ is completed for all local line features (step S506). In this way, when the calculation of the image degradation prediction value σ ₀ is completed for all local line features, this processing is terminated.

  Next, a process of evaluating the blur / blur size from the two-dimensional image in step S305 will be described. In the present embodiment, image degradation caused by image spread due to blur and / or blur in a two-dimensional image of measurement data features is evaluated. FIG. 6 is a flowchart illustrating the image degradation evaluation value calculation process executed by the evaluation value calculation unit 150 in step S305. The evaluation value calculation unit 150 selects one of the edge features that are corresponding candidates detected in step S304 (step S601), and calculates an image degradation evaluation value for the selected edge feature (step S602).

数６において、ｘ_０、ｙ_０はエッジのある着目画素位置、ｒは着目画素位置からの距離、θはエッジの法線方向、ｔは着目画素からの探索範囲、σはボケの大きさとブレの大きさを統合した値（画像劣化評価値）を示す。なお、ｔは任意の正の値である。 The calculation of the image degradation evaluation value by the evaluation value calculation unit 150 will be described. The evaluation value calculation unit 150 calculates the blur / blur size of the edge feature from the position of the edge feature selected in step S601 and the normal direction of the edge. As an edge model used in the present embodiment, fitting is performed using an error function erf represented by the following formula [Equation 6], and σ is obtained as the magnitude of blurring and blurring.

In Equation 6, x ₀ and y ₀ are the target pixel position with an edge, r is the distance from the target pixel position, θ is the normal direction of the edge, t is the search range from the target pixel, σ is the size and blur of the blur. A value (image degradation evaluation value) obtained by integrating the sizes of the images is shown. Note that t is an arbitrary positive value.

数７において、Ｉ（ｘ，ｙ）は座標（ｘ，ｙ）における撮影画像の輝度値を示す。 The image degradation evaluation value σ is estimated by minimizing the evaluation function E represented by the following formula [Equation 7] by the steepest descent method, the iterative calculation by the Levenberg-Marquardt method, or the like.

In Equation 7, I (x, y) indicates the luminance value of the captured image at the coordinates (x, y).

  For all the edge features detected in step S304, the processes in steps S601 and S602 are repeated until the above-described evaluation of the blur / blur size is completed (step S603). When the evaluation of the blur / blur size has been completed for all the edge features detected in step S304, the process ends.

  FIG. 7 is a flowchart showing the accuracy calculation process in step S306. In the present embodiment, the closer the image degradation prediction value that is the degree of image degradation calculated by the prediction value calculation unit 120 and the image degradation evaluation value that is the degree of image degradation evaluated by the evaluation value calculation unit 150, the higher the value. The accuracy is calculated as follows. First, the accuracy calculation unit 160 selects one local line feature in the shape model of the object to be measured (step S701). Next, the accuracy calculation unit 160 selects one edge feature (correspondence candidate) searched by the search unit 140 in step S304 as a candidate for association with the local line feature selected in step S701 (step S702). ) And the accuracy is calculated (step S703). Hereinafter, calculation of accuracy will be described.

The accuracy calculation unit 160 refers to the calculation result of the evaluation value calculation unit 150 and acquires the image degradation evaluation value σ corresponding to the edge feature selected in step S702. In addition, the accuracy calculation unit 160 acquires the image degradation predicted value σ ₀ of the local line feature selected in step S701. The accuracy calculation unit 160 calculates the accuracy T from the image degradation prediction value σ ₀ and the image degradation evaluation value σ according to the following equation [Equation 8].

  Steps S702 to S703 are repeated until the accuracy T has been calculated for all corresponding candidates (edge features) searched by the search unit 140 for the local line feature selected in step S701. When the accuracy T has been calculated for all the corresponding candidates (edge features), the process proceeds to step S705. The accuracy calculation unit 160 repeats the above processing (steps S701 to S704) until all local line features in the shape model of the measured object are finished. When all the local line features are finished, this processing is finished.

  FIG. 8 is a flowchart for explaining the processing procedure of position / orientation calculation by the collation unit 170 in step S307. The collation unit 170 uses the Gauss-Newton method, which is one of nonlinear optimization methods, so that the shape model is applied to the measurement data, and outlines the position and orientation of the measurement target object (hereinafter, represented by a 6-dimensional vector s). The value is repeatedly corrected by an iterative operation.

  First, the collation unit 170 performs an initialization process (step S801). In the present embodiment, the approximate position or orientation value where the object to be measured is placed is input to the matching unit 170 as an approximate value. This approximate value is the same as that used in step S302. Next, the matching unit 170 selects an edge feature to be matched with the local line feature of the model (step S802). For the selection, the accuracy T of correspondence between the edge feature and the local line feature calculated by the accuracy calculation unit 160 is used. For example, when a plurality of edge features are detected as a matching hypothesis for one local line feature, a corresponding candidate (edge feature) having the highest accuracy T is associated with the local line feature and the subsequent processing I do.

  Subsequently, the collation unit 170 calculates a coefficient matrix and an error vector for calculating the position and orientation (step S803). Here, each element of the coefficient matrix is a first-order partial differential coefficient with respect to each element of the position and orientation when the distance between the point and the line on the image is a function of the position and orientation. The error vector is a signed distance on the image of the projected line segment and the detected edge for the edge, and a signed distance in the three-dimensional space between the point and the surface of the model for the point cloud data.

Next, derivation of the coefficient matrix will be described. FIG. 9 is a diagram for explaining the relationship between a line segment 901 that is a projected image and a detected edge 902. In FIG. 9, the horizontal and vertical directions of the image are the u axis and the v axis, respectively. The position on the image of a certain control point 903 (a point obtained by dividing each projected line segment at equal intervals on the image) is (u ₀ , v ₀ ), and the image of the line segment 901 to which the control point 903 belongs is displayed. Is expressed as an inclination θ with respect to the u-axis. The inclination θ is calculated as the inclination of a straight line obtained by projecting the three-dimensional coordinates at both ends of the line segment onto the image based on the position and orientation s of the measured object and connecting the coordinates at both ends on the image. The normal vector on the image of the line segment 901 is (sin θ, −cos θ). Further, the coordinates on the two-dimensional image of the corresponding point 904 of the control point 903 are (u ′, v ′). Here, a point (u, v) on a straight line (edge 902 indicated by a broken line in FIG. 9) passing through the point (u ′, v ′) and having an inclination of θ is expressed by the following equation [Equation 9]. (Θ is a constant). Here, d = u′sin θ−v′cos θ (constant).

The position of the control point on the image changes depending on the position and orientation of the measurement target object. The degree of freedom of the position and orientation of the measurement target object is 6 degrees of freedom. That is, the position / orientation s is a six-dimensional vector, and includes three elements representing the position of the measurement target object and three elements representing the attitude. The three elements representing the posture are represented by, for example, expression by Euler angles or a three-dimensional vector in which the direction represents the rotation axis passing through the origin and the norm represents the rotation angle. The image coordinates (u, v) of the control points that change depending on the position and orientation can be approximated by the first-order Taylor expansion in the vicinity of (u ₀ , v ₀ ) as in the following equation [Equation 10]. In Equation 10, Δs _i (i = 1, 2,..., 6) represents a minute change of each component of s.

位置及び姿勢の概略値によって算出された各制御点の奥行き（視点から制御点までの距離＝ｚ）を、画像上の誤差に乗じることによって３次元空間中の誤差に変換する。 Assuming that there is not much difference between the approximate value of position and orientation and the actual position and orientation, it can be assumed that the position of the control point obtained by the correct s on the image is on the straight line represented by Equation [9]. By substituting u and v approximated by Equation [Equation 10] into Equation [Equation 9], Equation [Equation 11] is obtained.

The depth of each control point (distance from the viewpoint to the control point = z) calculated based on the approximate values of the position and orientation is converted to an error in the three-dimensional space by multiplying the error on the image.

The simultaneous equations to be solved are as shown in Equation [12].

線形連立方程式の係数行列Ｊを算出するための偏微分係数の算出は、例えば非特許文献２に開示されている方法によって行うことができる。 In Expression [12], z ₁ , z ₂ ... Represent the depth of each edge. Here, Expression [Equation 12] is expressed as Expression [Equation 13]. In [Equation 13], J is a coefficient matrix, E is an error vector, and Δs is a correction value.

Calculation of the partial differential coefficient for calculating the coefficient matrix J of the linear simultaneous equations can be performed by a method disclosed in Non-Patent Document 2, for example.

Next, the collation unit 170 uses the generalized inverse matrix (J ^T · J) ⁻¹ · J ^T of the matrix J based on the equation [Equation 13] to calculate the position and orientation correction value Δs based on the least-squares criterion. (Step S804). However, since there are many outliers due to erroneous detection or the like in edge and point cloud data, the following robust estimation method is used. Generally, in the case of edge and point cloud data that are outliers, the value of the error vector on the right side of Equation [12] becomes large. Therefore, a small weight is given to data with a large absolute value of error, and a large weight is given to data with a small error. The weight is given by, for example, a Tukey function as shown in Equation [14].

上記式［数１４］において、ｃ_１、ｃ_２は定数である。なお、重みを与える関数はＴｕｋｅｙの関数である必要はなく、例えばＨｕｂｅｒの関数など、誤差が大きいデータには小さな重みを与え、誤差が小さいデータには大きな重みを与える関数であればなんでもよい。

In the above formula [Formula 14], c ₁ and c ₂ are constants. Note that the function that gives the weight need not be a Tukey function, and any function that gives a small weight to data with a large error and gives a large weight to data with a small error, such as a Huber function, for example.

The weight corresponding to each measurement data (edge or point cloud data) is defined as w _i . Here, a weight matrix W is defined as shown in Equation [15].

式［数１６］を式［数１７］のように解くことにより補正値Δｓを求める。

The weight matrix W is a square matrix of all zeros except for the diagonal component, and the weight w _i calculated according to the equation [Equation 14] is entered in the diagonal component. Using this weight matrix W, Equation [Equation 13] is transformed into Equation [Equation 16].

The correction value Δs is obtained by solving Equation [16] as Equation [17].

Returning to FIG. 8, the collation unit 170 corrects the approximate position and orientation values by using the position and orientation correction values Δs calculated in step S804 (step S805).

  Subsequently, the collation unit 170 performs convergence determination. If the convergence has been completed, the collation unit 170 ends. If not, the process returns to step S802 (step S806). In the convergence determination, when the correction value Δs is almost 0, or when the sum of squares of the error vector hardly changes after the correction and after the correction (when the difference before and after the correction is equal to or less than a predetermined threshold value) judge. In the above description, an example in which the Gauss-Newton method is used as the nonlinear optimization method has been described. However, the nonlinear optimization method is not limited to this, and other nonlinear optimization methods such as a Newton-Raphson method, a Levenberg-Marquardt method, a steepest descent method, and a conjugate gradient method may be used.

  The shape information held as a model of the object to be measured in the present embodiment may be two-dimensional or three-dimensional geometric information representing the target shape, and the expression format is not particularly limited. For example, when representing two-dimensional geometric information, it may be represented by a simple set of two-dimensional points or a set of two-dimensional lines. When representing 3D geometric information, a polygonal shape represented by a set of simple 3D points, a set of 3D lines representing edges, or a set of 3D points It may be expressed by information.

  In addition, as the relative movement direction / velocity between the imaging device and the measured object input in step S501, information on the movement direction and the movement amount assuming that the measured object is translated in a certain direction is input. Although an example has been shown, the present embodiment is not limited to this. For example, the sensor described above may be attached to the object to be measured, and the relative position and orientation speed between the imaging device and the object to be measured may be calculated from the information. Further, in a scene where the imaging device moves, the movement of the imaging device may be acquired. For example, when an imaging device is attached to the robot, the robot motion information may be input as the relative movement direction and speed between the imaging device and the object to be measured. Alternatively, a physical sensor that measures a position and orientation with six degrees of freedom such as a magnetic sensor or an ultrasonic sensor may be attached to the imaging apparatus, and the information may be input. In the present embodiment, any method may be used as long as the relative position / orientation speed between the imaging apparatus and the object to be measured can be calculated, and there is no particular limitation on the selection of means and devices.

  Further, the amount estimated as the image degradation prediction value in the prediction value calculation unit 120 is not limited to the above-described information. For example, any value that can represent the amount by which the two-dimensional image deteriorates due to relative movement between the imaging device and the object to be measured may be used, and the calculation method and expression are not particularly limited. For example, the magnitude of the blurring of the edge feature of the two-dimensional image in which the object to be measured is photographed may be expressed as a movement amount in the three-dimensional space calculated by back projecting to the three-dimensional space. Further, a point spread function (PSF) created based on the blur size D and the blur size B may be used as the image degradation evaluation value. In the above embodiment, the predicted value is acquired in consideration of the effects of both blur and blur. However, the predicted value may be acquired in consideration of the effect of only one of them.

  Further, although the edge feature is used as the measurement data feature that is a candidate for correspondence in the search unit 140 in the present embodiment, the feature is not limited to this. For example, any feature may be used as long as it can identify a position on a two-dimensional image, such as a point feature. Further, instead of using only specific types of features, correspondence between a plurality of types of features (for example, points and edges) and model features may be detected.

  In addition, the method of estimating the blur / blur size from a single image in the evaluation value calculation unit 150 is not limited to the method described above. For example, assuming that the spread of blur and blur spreads in a Gaussian distribution, the brightness distribution at the edge where blur and blur occurs is blurred from the standard deviation obtained when fitting the Gaussian function using the least square method. -You may ask for the size of a blur. In the present embodiment, the image degradation evaluation value is calculated for each edge feature, but the present invention is not limited to this. For example, a two-dimensional image may be divided into rectangular partial areas, and an image deterioration evaluation value may be calculated for each partial area.

  Further, although the accuracy calculation unit 160 calculates the accuracy using the formula [Equation 8], the accuracy calculation method is not limited to this. The function for calculating the accuracy is such that the accuracy is small when the difference between the image degradation evaluation value and the image degradation prediction value is large, and the accuracy is large when the difference between the image degradation evaluation value and the image degradation prediction value is small. Any function can be used. For example, for example, a Gaussian function, a Tukey function, a Huber function, or the like can be used.

  Further, the collation unit 170 collates the pair of the local line feature and the edge feature having the highest accuracy, but is not limited thereto. It is only necessary to preferentially select the calculated pair with the highest accuracy. For example, even if the pair with the highest accuracy is selected, it should not be used for matching if the accuracy is below a certain threshold. Also good. Similarly, even when only one pair of local line feature and edge feature is detected, the pair may not be used for matching if the accuracy is equal to or less than a threshold value. Furthermore, although the collation part 170 acquired the position and attitude | position of the to-be-measured object in this embodiment, it is not restricted to this. For example, local line feature positions on a two-dimensional image are held in advance at a plurality of positions and orientations, and local line features and edge features are collated by pattern matching to identify approximate positions and orientations, or objects You may specify the kind of.

  In step S302, step S501, and step S801, approximate values of the position and orientation of the object are given as approximate values of the position and orientation of the model of the object to be measured. However, the approximate position and orientation input method is not limited to this. For example, the measurement values obtained by the previous (previous time) measurement may be used as the approximate position and orientation. In addition, the speed and angular velocity of the object are estimated using a time series filter based on the measurement of the past position and orientation, and the current position and orientation are predicted based on the estimated speed and acceleration of the past position and orientation. You may use as a position and attitude | position. In addition, when the position and orientation of an object can be measured using another sensor, the output value from the sensor may be used as the approximate value of the position and orientation. For example, a magnetic sensor that measures the position and orientation by detecting the magnetic field generated by the transmitter with a receiver mounted on the object, and the position and orientation by shooting a marker placed on the object with a camera fixed to the scene. One example is an optical sensor for measurement. In addition, any sensor may be used as long as it measures a position and orientation with six degrees of freedom. Further, when an approximate position or posture where an object is placed is known in advance, the value may be used as an approximate value.

  As described above, in the first embodiment, the accuracy of association is calculated from the blurring and blurring of the image obtained from the two-dimensional image obtained by photographing the object to be measured, and the model feature and the measurement data feature are collated. Explained how to do. This makes it possible to correctly collate model features with measurement data features.

<Modification 1>
In the first embodiment, the collation unit 170 selects and collates a pair of local line features and edge features with high accuracy. However, the present invention is not limited to this. It is sufficient that the calculated pair with high accuracy is preferentially used for the matching process. For example, the accuracy of the correspondence between each local line feature and the edge feature is determined for each edge feature when performing position and orientation estimation. It is also possible to set the weight as a weight and to perform collation based on this weighting. This weighting method will be described. First, the accuracy T corresponding to each local line feature and edge feature is given to the diagonal component of the weight matrix W calculated by the formula [Equation 15]. By solving the equation [Equation 17] using the weight matrix W, the position and orientation correction value Δs is obtained, and the approximate position and orientation values are corrected. By the above method, it is possible to preferentially collate the calculated pair with high accuracy.

  Although the accuracy T itself is given as a value given to the diagonal component of the weight matrix W, it is not limited to this. A pair with high accuracy gives a large weight, and a pair with low accuracy gives a small weight. Any ratio of all accuracy divided by the maximum value of accuracy is given as a weight. Also good.

  As described above, in the first modification, the accuracy of association is set as the weight of each data feature from the blurring and blurring of the image obtained from the two-dimensional image obtained by photographing the measured object, and the local line feature is set. And how to perform edge feature matching. Thereby, collation can be performed with high accuracy.

Second Embodiment
In the first embodiment, the size of blur and blur occurring on a two-dimensional image is predicted from the model, and the accuracy is calculated by comparing with the blur and blur size of the image obtained from the two-dimensional image. A method of using the for matching with local line features has been described. In the second embodiment, an image deterioration prediction value of each local line feature is calculated from a two-dimensional image in which the blur and blur magnitudes are calculated in advance, and compared with the measured blur and blur magnitudes of the measured image, The degree is calculated, and the features having the highest accuracy are associated with each other. Hereinafter, a second embodiment will be described. The configuration of the information processing device and the processing procedure for measuring the position and orientation in the second embodiment are the same as those in the first embodiment (FIGS. 1 and 3).

In the second embodiment, in step S302, the predicted value calculation unit 120 estimates the size of blur and the size of blur from a two-dimensional image obtained before (previous time). The method for estimating the blur / blur size from the two-dimensional image is the same as the method for estimating the blur / blur size of the edge feature portion by the evaluation value calculation unit 150 of the first embodiment. However, the blur / blur size is measured at the projection position of the model feature instead of the edge feature portion. For example, the predicted value calculation unit 120 projects a shape model (for example, a local line feature) on the previously obtained two-dimensional image using the approximate position and orientation of the measured object. Then, an image degradation evaluation value is calculated for the projection position by the method described in steps S601 and S602, and this is used as an image degradation prediction value. Therefore, the image degradation predicted value σ ₁ of the local line feature obtained by the predicted value calculation unit 120 is blurred or blurred at the coordinate position of the model feature (for example, the local line feature) when the shape model is projected onto the two-dimensional image. Is the size of

The accuracy calculation unit 160 is searched by the search unit 140 from the predicted image degradation value obtained by the prediction value calculation unit 120 in step S302 and the image degradation evaluation value obtained by the evaluation value calculation unit 150 in step S305. Accuracy is calculated for each edge feature. The edge feature searched by the search unit 140 is a candidate for correspondence with the local line feature of the model of the object to be measured, and is stored in the memory as a hypothesis (step S304). The accuracy T is expressed as follows using the image degradation evaluation value σ of the edge feature on the two-dimensional image held as a hypothesis corresponding to the local line feature in the model and the image degradation prediction value σ ₁ of the local line feature in the model. This is calculated according to the equation [Equation 19]. When a plurality of hypotheses of correspondence between the edge feature in the two-dimensional image and the local line feature in the model are detected, the following formula [Equation 19] is applied to all the edges listed as candidates. Using this, the accuracy T is calculated.

  As described above, in the second embodiment, the size of blur and blur is calculated using a two-dimensional image measured in advance, and the accuracy of correspondence between features having similar blur and blur sizes is increased. Corresponding is performed. As a result, it is possible to correctly perform matching between features.

  Note that the predicted value calculation unit 120 in the second embodiment estimates the size of blur and the size of blur from the two-dimensional image obtained at the previous time (previous time), but is not limited thereto. Instead of the object to be measured, a two-dimensional image obtained by photographing an object that can simulate the object (hereinafter, a simulated object) may be used as the “two-dimensional image obtained last time”. In addition, the size of blur and the size of blur detected on a two-dimensional image obtained by photographing an object to be measured or a simulated object at a plurality of positions and postures in advance may be tabulated, and this table may be referred to. Good. That is, the predicted value calculation unit 120 may obtain the blur / blur size corresponding to the approximate position and orientation of the measured object as the image degradation predicted value by referring to this table. In addition, instead of the actual two-dimensional image, a predicted image degradation value may be obtained using a CG image created by projecting a shape model two-dimensionally. In simulating the blurring / blurring phenomenon on such a CG image, a known method, for example, a method disclosed in Non-Patent Document 3, can be used.

  As described above, by using the method of the second embodiment, the accuracy of association is calculated from the size of blur or blur using a two-dimensional image measured in advance or a created CG image, thereby creating a model. Features and measurement data features can be correctly verified.

<Third Embodiment>
In the first and second embodiments, the blur and blurring of an image that would occur in a two-dimensional image are estimated as image degradation prediction values (σ ₀ , σ ₁ ), and edges obtained from the two-dimensional image It was used to select the correspondence between features and local line features of the model. In the third embodiment, a predicted image deterioration value is used for associating a measured three-dimensional measurement point (hereinafter referred to as a three-dimensional point) with a surface feature of a model. In the third embodiment, the size of the blur and blur on the two-dimensional image of the local surface feature and the blur and blur on the two-dimensional image when the measured three-dimensional position of the three-dimensional point is projected on the two-dimensional image. A high degree of accuracy is assigned to the correspondence of features that are close to the size of and the correspondence is selected.

  FIG. 10 is a diagram illustrating a functional configuration example of the information processing apparatus 3 according to the third embodiment. Note that the hardware configuration of the information processing apparatus 3 is the same as that shown in FIG. The information processing apparatus 3 includes a three-dimensional point input unit 210 and a position calculation unit 220 in addition to the configuration of the information processing apparatus 1. Moreover, the structure shown in FIG. 10 is a structure used as the example of application of the information processing apparatus of this invention. Each function shown in FIG. 10 is realized by the CPU 101 executing a program stored in the ROM 102 and / or a program expanded from the external storage device 105 to the RAM 103. Obviously, some or all of the functional units may be realized by dedicated hardware.

  The three-dimensional point input unit 210 acquires the three-dimensional coordinates of the point group on the surface of the measured object. In the third embodiment, the three-dimensional coordinates of a three-dimensional point group acquired in advance using a distance sensor are acquired from a storage device stored outside, but the present invention is not limited to this. For example, a three-dimensional point group acquired using a distance sensor (for example, a three-dimensional measurement device) may be input.

  The position calculation unit 220 calculates a two-dimensional image position when the three-dimensional coordinates of the point group input by the three-dimensional point input unit 210 are projected on the two-dimensional image. The two-dimensional image position of the three-dimensional coordinates is calculated from the internal parameters of the imaging device that captured the two-dimensional image, and the relative position and orientation of the imaging device and the distance sensor used to measure the three-dimensional point. Is done. Note that the distance sensor and the imaging device are fixed to each other, and the relative position and orientation of both are assumed to be unchanged, and calibration is performed in advance. For example, a calibration object having a known three-dimensional shape is observed from various directions, and the difference between the position and orientation of the calibration object based on the two-dimensional image and the position and orientation of the calibration object based on the distance image is also obtained. In addition, the relative position and orientation of the distance sensor and the imaging device are obtained.

Note that the model input by the model input unit 110 in the third embodiment is a local three-dimensional object on the surface of the object composed of a three-dimensional position and a three-dimensional normal direction as shown in FIG. It is assumed to be constituted by plane information (hereinafter referred to as local surface features). The predicted value calculation unit 120 calculates the degradation predicted value σ ₀ of the local surface feature by the same method as in the first embodiment.

  The search unit 140 detects correspondence between the two-dimensional image input by the image input unit 130, the three-dimensional point input by the three-dimensional point input unit 210, and the local surface feature of the shape model input by the model input unit 110. In the association between the local surface feature and the three-dimensional point, the vicinity of the local surface feature is based on the approximate values of the positions and orientations of all the local surface features constituting the model of the measured object input by the model input unit 110. Are detected. When there are a plurality of three-dimensional points in the vicinity, all the detected three-dimensional points are held as hypotheses.

The evaluation value calculation unit 150 calculates, as the image deterioration evaluation value σ, the magnitude of blur / blur at each position on the two-dimensional image of the three-dimensional point calculated by the position calculation unit 220. The method for calculating the image degradation evaluation value σ is the same as that in the first embodiment. The accuracy calculation unit 160 is a hypothesis that is associated with the local surface feature of the model from the image deterioration prediction value σ ₀ obtained by the prediction value calculation unit 120 and the image deterioration evaluation value σ obtained by the evaluation value calculation unit 150. Corresponding accuracy T is calculated for each three-dimensional point feature held as. The calculation method of the accuracy T is the same as in the first embodiment.

  Based on the accuracy T calculated by the accuracy calculation unit 160, the verification unit 170 in the third embodiment uses a three-dimensional point to be used for verification among the three-dimensional points corresponding to each local surface feature in the model of the measured object. Select a point. Here, a pair of a local surface feature and a three-dimensional point having the highest accuracy T is selected, but the present invention is not limited to this. For example, even if the pair with the highest accuracy T is selected, it may not be used for matching if the accuracy T is below a certain threshold. Similarly, even when only one local surface feature / three-dimensional point pair is detected, the pair may not be used for matching if the accuracy is equal to or less than a threshold value. Then, the matching unit 170 calculates the position and orientation of the measured object using the selected three-dimensional point. A method similar to that of the first embodiment can be used as the position / orientation calculation method by the collation unit 170.

  In addition, as a distance sensor used for the measurement of the three-dimensional point in this embodiment, an active type sensor that captures the reflected light of the laser beam irradiated on the object with a camera and measures the distance by triangulation is used. However, the distance sensor is not limited to this, and may be a time-of-flight method that uses the flight time of light. These active distance sensors are suitable when an object having a small surface texture is a target. Moreover, the passive type which calculates the depth of each pixel by the triangulation from the image which a stereo camera image | photographs may be used. The passive distance sensor is suitable when an object having a sufficient surface texture is targeted. In addition, anything that can measure a three-dimensional point does not impair the essence of the present invention.

  As described above, in the third embodiment, the size of the blur and blur on the two-dimensional image of the local surface feature and the three-dimensional position of the actually measured three-dimensional point are projected onto the two-dimensional image. In the above description, a method for giving a high degree of accuracy to the correspondence between the features that are close to the size of blurring and blurring on the upper side, and collating the model features with the three-dimensional measurement points has been described. Thereby, it becomes possible to correctly collate the model feature with the three-dimensional measurement point.

<Modification 2>
In the third embodiment, the collation unit 170 selects and collates a pair of local surface features and three-dimensional points with high accuracy. However, the present invention is not limited to this. You may make it set the precision of a response | compatibility of each local surface feature and a ternary point as a weight for every three-dimensional point when performing position and orientation estimation. For example, as in the first modification, the accuracy level may be set as a weight for each three-dimensional point, and matching may be performed based on this weighting. As for this weighting method, as in the first modification, the accuracy T corresponding to each local surface feature and the three-dimensional point is given to the diagonal component of the weight matrix W calculated by the formula [Equation 15]. By solving the equation [Equation 17] using the weight matrix W, the position and orientation correction value Δs is obtained, and the approximate position and orientation values are corrected. By the above method, it is possible to preferentially collate the calculated pair with high accuracy.

  Although the accuracy T itself is given as the value given to the diagonal component of the weight matrix W, it is not limited to this as in the first modification. Any value can be used as long as a pair with high accuracy gives a large weight and a pair with low accuracy gives a small weight.

  As described above, in the second modification, the accuracy of association is set as the weight of each data feature from the blurring and blurring of the image obtained from the two-dimensional image obtained by photographing the object to be measured. The method of performing the three-dimensional point matching has been described. Thereby, collation can be performed with high accuracy.

<Modification 3>
In the third embodiment, the blur magnitude calculation method performed in step S504 may be the following method. The amount of movement of each local surface feature in the three-dimensional space during the exposure time is used as the magnitude of blurring. Specifically, the Jacobian of each local surface feature is calculated based on the approximate position and orientation s of the shape model, and the relative moving direction of the Jacobian of each local surface feature and the imaging device input in step S502 and the measured object The amount of blur of each local surface feature is calculated from the speed.

The local surface feature Jacobian is a value representing the rate at which the local surface feature in the three-dimensional space changes when the parameter of the position and orientation 6 degrees of freedom changes minutely. Based on the approximate position and orientation s of the object to be measured, a blur amount of the three-dimensional point corresponding to the local surface characteristics (dx, dy, dz), the normal direction of the local surface characteristics _{(n x, n y, n} z ) (Unit vector), the signed distance err _3D between the correspondences can be calculated by the following equation [Equation 20].

Then, similarly to the Jacobian of the local line feature, by performing partial differentiation on the inter-corresponding distance err _3D with each parameter of position and orientation s, the Jacobian matrix of the local surface feature is calculated as in the following equation [Equation 21]. .

Ｂはスカラーであり、露光時間中に三次元空間中の局所面特徴の３次元位置が移動する量を表す。以上の処理を全ての局所面特徴に対して行い、全ての局所面特徴に対してブレ量を算出する。 As described above, the Jacobian of the projected local line feature selected in step S502 is calculated. Using Jacobian of the projection local line feature, the object during the exposure time t _i of the 2-dimensional image is generated by moving a relative position and orientation velocity V, the distance change B between the three-dimensional point local areal feature And can be calculated by the following equation [Equation 22].

B is a scalar and represents the amount by which the three-dimensional position of the local surface feature in the three-dimensional space moves during the exposure time. The above processing is performed for all local surface features, and the shake amount is calculated for all local surface features.

  As described above, in the third modification, the method for calculating the magnitude of the blur from the amount of movement of the local surface feature in the three-dimensional space has been described. By using this method, it is possible to accurately obtain the predicted value of the blur amount of the model feature.

<Modification 4>
The coefficient matrix and error vector calculation method for calculating the position and orientation performed in step S803 in the third embodiment may be the following method.

The three-dimensional coordinates of the point group represented by the camera coordinates are converted into three-dimensional coordinates (x, y, z) in the coordinate system of the measurement target object using the position and orientation s of the measurement target object. It is assumed that a certain point in the point cloud data is converted into measurement target object coordinates (x ₀ , y ₀ , z ₀ ) by the approximate position and orientation. (X, y, z) changes depending on the position and orientation of the object to be measured, and is approximated by the first-order Taylor expansion in the vicinity of (x ₀ , y ₀ , z ₀ ) as shown in Equation 23. it can.

The equation of the local surface feature associated with a point in the point cloud data in the object coordinate system to be measured is expressed as ax + by + cz = e (a ² + b ² + c ² = 1, a, b, c, e is a constant). Assume that (x, y, z) transformed by the correct s satisfies the plane equation ax + by + cz = e. By substituting equation [Equation 23] into the equation of the plane, equation [Equation 24] is obtained.

式［数２４］はsの各成分の微小変化Δsi（i=1,2,・・・,6）についての方程式であるため、式［数２５］のようなΔsiに関する線形連立方程式を立てることができる。

Since the equation [Equation 24] is an equation for a minute change Δsi (i = 1, 2,..., 6) of each component of s, a linear simultaneous equation regarding Δsi like the equation [Equation 25] is established. Can do.

  Here, Expression [Equation 25] is expressed as Expression [Equation 13]. Using this coefficient matrix J and error vector E, the position and orientation are calculated by the same method as in the first and second embodiments. By using the method of Modification 3 above, the position and orientation of the measurement object can be calculated more accurately.

<Fourth embodiment>
In the first and second embodiments, the model feature and the measurement data feature are collated only from the correspondence between the local line feature and the edge feature, and in the third embodiment, from the correspondence between the local surface feature and the three-dimensional point only. It is not limited. Matching may be performed using both the correspondence between the local line feature and the edge feature and the correspondence between the local surface feature and the three-dimensional point. According to the method of the present embodiment, by calculating the accuracy of correspondence from the blur and the size of the blur obtained from the two-dimensional image obtained by photographing the measured object, the measured edge feature and local line feature, and Both 3D points and local surface features can be correctly matched.

The configuration of the information processing apparatus in the fourth embodiment is the same as that in the first, second, and third embodiments. The matching unit 170 in the fourth embodiment is based on the accuracy of the correspondence between the edge feature and the local line feature and the accuracy of the correspondence between the three-dimensional point and the local surface feature calculated by the accuracy calculation unit 160.
-Edge features corresponding to each local line feature and local surface feature in the model of the object to be measured,
Select an edge feature and a 3D point to be used for matching from the 3D points. Subsequently, a coefficient matrix and an error vector for calculating the position and orientation are calculated based on the selected correspondence. In the present embodiment, in order to obtain the position and orientation by using the edge feature and the three-dimensional point together, the equation [Equation 12] and the equation [Equation 25] are combined and Δs _i like the equation [Equation 26] is obtained. Linear simultaneous equations can be established.

  Here, Expression [Equation 26] is expressed as Expression [Equation 13]. Using this coefficient matrix J and error vector E, the position and orientation are calculated by the same method as in the first, second, and third embodiments.

  Not limited to this, as described in the first and second modified examples, the height of accuracy may be set as a weight for each three-dimensional point, and collation may be performed based on this weighting. As long as the accuracy T itself is not given, any value may be used as long as a large weight is given to a pair with high accuracy and a small weight is given to a pair with low accuracy, as in the first and second modifications.

  By using the method of the fourth embodiment described above, it is possible to collate using both the correspondence between the local line feature and the edge feature and the correspondence between the local surface feature and the three-dimensional point. With the method of this embodiment, it is possible to more accurately collate model features with measurement data than to perform collation by each correspondence.

<Fifth Embodiment>
One suitable application example of the information processing apparatuses 1 and 3 will be described below. That is, there is an application example in which the position and orientation of an object to be measured is estimated based on a two-dimensional image obtained from an imaging apparatus, and the object is gripped by an industrial robot arm. Hereinafter, an application example of the information processing apparatuses 1 and 3 to the robot system will be described with reference to FIG. FIG. 11 is a diagram illustrating a configuration example of a robot system that holds the measured object 40 using the robot 60 based on the position and orientation estimation results by the information processing apparatuses 1 and 3.

  The robot 60 has a movable axis composed of, for example, rotation and / or translational movement axes, and each movable axis is driven and controlled by a robot controller 61. For example, the robot 60 moves the hand to a position instructed by the robot controller 61 and grips an object. Since the position of the measured object 40 on the workbench changes, it is necessary to estimate the current position and orientation of the measured object 40 and to perform grip control of the robot. The imaging device 20 is a camera that captures a normal two-dimensional image, and is installed at a position where an object to be measured 40 such as a hand of an industrial robot arm can be imaged. The information processing apparatuses 1 and 3 estimate the position and orientation of the measured object 40 based on the two-dimensional image obtained from the imaging apparatus 20. The position and orientation of the measured object 40 estimated by the information processing apparatuses 1 and 3 are input to the robot controller 61, and the robot controller 61 grips the measured object 40 based on the input position and orientation estimation result. To control the robot arm.

  As described above, according to the robot system of the fifth embodiment, the estimation result of the position and orientation of the measured object 40 can be obtained from the information processing apparatuses 1 and 3 that perform the position and orientation estimation more accurately. Therefore, the robot 60 can grip the object to be measured 40 more reliably.

<Effect>
As described above, according to the first embodiment, by calculating the accuracy of the association from the blur and the magnitude of the blur obtained from the two-dimensional image obtained by photographing the object to be measured, Measurement data features can be correctly verified. Further, according to the third embodiment, the measured three-dimensional point and the three-dimensional point are calculated by calculating the accuracy of association from the blur and the blur of the image obtained from the two-dimensional image obtained by photographing the measured object. It becomes possible to correctly collate with the surface features of the three-dimensional shape model. According to the fourth embodiment, the edge feature in the two-dimensional image is calculated by calculating the correspondence accuracy from the blur and the blur of the image obtained from the two-dimensional image obtained by photographing the measured object. It becomes possible to correctly collate the line feature of the three-dimensional shape model and the actually measured three-dimensional point and the surface feature of the three-dimensional shape model at the same time. Further, according to the second embodiment, by calculating the accuracy of association from the size of blur or blur acquired in advance using a two-dimensional image that has been a measurement target in the past or a created CG image, Therefore, it is possible to correctly match the model feature and the measurement data feature. Furthermore, according to the fifth embodiment, the position and orientation of the object to be measured are estimated, and the robot system can grip and move the object to be measured based on the estimation result.

<Definition>
Note that the shape information input as the shape model of the object to be measured includes a simple set of two-dimensional points, a set of two-dimensional lines, a simple set of three-dimensional points, and a set of three-dimensional lines representing ridge lines. It may be represented by shape information of a polygon composed of three three-dimensional points.

  Further, in the predicted value calculation unit, as the image degradation predicted value, the size of blur due to defocusing and the size of blur caused by parallel movement on the image plane are set as the image degradation predicted value. Absent. Any value that can represent the amount by which the two-dimensional image deteriorates due to the relative movement between the imaging device and the object to be measured may be used, and the calculation method and expression are not particularly limited. For example, the magnitude of the blurring of the edge feature of the two-dimensional image in which the object to be measured is photographed may be expressed as a movement amount in the three-dimensional space calculated by back projecting to the three-dimensional space. Also, a point spread function (PSF) created based on the size of blur or the size of blur may be used as the image degradation evaluation value. In addition to a predicted value that takes into account the effects of both blur and blur, a predicted value that takes into account the effect of only one of the two may be used.

  The two-dimensional image obtained by photographing the object to be measured input by the image input unit may be any method as long as a target image is obtained. For example, it may be a gray image or a color image. In the present invention, a two-dimensional image captured in advance is input, but the present invention is not limited to this. You may input the result image | photographed using the imaging device. The measurement data feature that is a candidate for correspondence detected by the search unit 140 may be an edge feature or any feature that can identify a position on a two-dimensional image, such as a point feature. Further, instead of using only specific types of features, correspondence between a plurality of types of features (for example, points and edges) and model features may be detected.

  The evaluation value calculation unit, for example, applies an edge feature by fitting a function representing the luminance change of the edge when blurring / blurring occurs with respect to the luminance change of the pixel along the direction orthogonal to the edge in the two-dimensional image. An image degradation evaluation value is calculated for each. Here, a Gaussian function or the like may be used as a function representing the luminance change in place of the error function as shown in Equation [6]. In the above-described embodiment, the deterioration evaluation value is calculated for each feature such as the corresponding point corresponding to the control point and the projection position of the three-dimensional point. However, the present invention is not limited to this. For example, a two-dimensional image may be divided into rectangular partial areas, and a deterioration evaluation value may be calculated for each partial area.

When the difference between the image degradation evaluation value (σ) and the image degradation predicted value (σ ₀ , σ ₁ ) is large, the accuracy is reduced, and the accuracy calculation unit calculates the accuracy between the image degradation evaluation value and the image degradation predicted value. If the difference is small, any method can be used as long as the accuracy is large. For example, the calculation may be performed using a Gaussian function, a Tukey function, a Huber function, or the like instead of the calculation method shown in Equation [8].

  The matching unit selects a model feature and image feature pair whose accuracy is equal to or higher than a certain threshold value and performs matching. Even if only one pair is detected at this time, it may not be used for collation as long as it is below the threshold. However, the present invention is not limited to this, and weighting may be performed so that a pair with high accuracy is preferentially used for the matching process. At this time, the value given by weighting may be any value as long as a pair with high accuracy gives a large weight and a pair with low accuracy gives a small weight. The collation unit may calculate the position and orientation of the object to be measured, hold the model feature positions in multiple positions and orientations in advance, and collate the model features with the measurement data features by pattern matching. The approximate position / orientation may be identified, or the type of object may be specified.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 1,3: Information processing apparatus, 110: Model input part, 120: Predicted value calculation part, 130: Image input part, 140: Search part, 150: Evaluation value calculation part, 160: Accuracy calculation part, 170: Collation part 210: three-dimensional point input unit, 220: position calculation unit

In order to achieve the above-described object, an image processing apparatus according to an aspect of the present invention includes a specifying unit that specifies a subject area from an image, and an output unit that outputs a trimmed image including the subject area using the image. a, output means, when a plurality of the subject region is specified by the specifying means, when a plurality of subjects area is than a predetermined condition is satisfied, outputs the trimming image which does not include any of the plurality of subject area and, if the plurality of objects area does not satisfy a predetermined condition, and outputs the trimmed image including a plurality of object area, the predetermined condition, at least, a plurality of object regions have a duplication of a predetermined ratio or more Including a plurality of subject areas, and a plurality of subject areas that are arranged with a distance longer than a predetermined value between the areas. .

In order to achieve the above object, an image processing method according to another aspect of the present invention includes a specifying step of specifying a subject region from an image, an output step of outputting a trimmed image including the subject region using the image, has, in the output step, when a plurality of the subject region is specified in a particular process, if a plurality of subjects area is than a predetermined condition is satisfied, the trimming image which does not include any of the plurality of subject area is output, as long as a plurality of object area does not satisfy a predetermined condition, the trimming image including a plurality of objects area is output, the predetermined condition is, at least, a plurality of object regions overlap over a predetermined ratio JP be disposed without having, and, a plurality of the subject region are arranged with a distance greater than a predetermined value between the regions, to include any one of To.

  Hereinafter, some preferred embodiments of the present invention will be described with reference to the accompanying drawings.

<First Embodiment>
In the first embodiment, the local line feature and the measurement data feature are associated with each other by using the blur and the blur of the image obtained from the two-dimensional image. That is, for each local line feature, the blur and blur size obtained in the two-dimensional image are predicted, and compared with the blur and blur size actually measured from the two-dimensional image. Correlate with a high contribution. In the first embodiment, the magnitude of blur and blur is determined by simulation from the shape represented by the shape model imitating the object to be measured and the relative moving direction and speed between the imaging device and the object to be measured that are determined in advance. Predict the size of blur and blur by calculating.

Claims (18)

Based on a shape model representing the shape of the measured object, predicting means for predicting image degradation of the measured object that occurs in an image obtained by imaging the measured object by the imaging means;
Search means for searching for a measurement data feature corresponding to a model feature of the shape model for a two-dimensional image obtained by imaging the object to be measured by the imaging means;
Evaluation means for evaluating image deterioration using the two-dimensional image for the measurement data feature searched by the search means;
A calculation unit that calculates the accuracy of correspondence between the model feature and the measurement data feature based on the image degradation predicted by the prediction unit and the image degradation evaluated by the evaluation unit;
An information processing apparatus comprising: collation means for collating the shape model with a measured object in the two-dimensional image based on the accuracy of correspondence between the model feature and the measurement data feature.   The information processing apparatus according to claim 1, wherein the image degradation is image degradation caused by image spreading due to blurring and / or blurring of a two-dimensional image.   The information processing apparatus according to claim 1, wherein the evaluation unit evaluates the image degradation by applying an error function to the measurement data feature of the two-dimensional image.   The prediction unit is configured to generate the image based on the shape model, a given position and orientation of the measured object, a change in relative position and orientation between the measured object and the imaging unit, and an internal parameter of the imaging unit. The information processing apparatus according to claim 1, wherein deterioration is predicted.   The prediction means is a coordinate position of a model feature of the pre-obtained image when the shape model is projected onto an image pre-obtained by photographing the object to be measured or an object simulating the object to be measured. 5. The information processing apparatus according to claim 1, wherein the image deterioration is acquired.   The information according to claim 1, wherein the predicting unit acquires image degradation based on a CG image created by projecting a shape model of the object to be measured two-dimensionally. Processing equipment.   2. The calculation unit according to claim 1, wherein the calculation unit calculates the accuracy so that the degree of image degradation predicted by the prediction unit and the degree of image degradation evaluated by the evaluation unit are closer to each other. The information processing apparatus according to any one of 6. The matching unit selects a correspondence between the model feature and the measurement data feature based on the accuracy;
The information processing apparatus according to claim 1, wherein: The information processing apparatus according to claim 8, wherein the collating unit preferentially selects a correspondence with a high accuracy calculated by the calculating unit.   The information processing apparatus according to claim 1, wherein the collating unit sets a weight related to the measurement data feature based on the accuracy. The information processing apparatus according to claim 1, wherein the collating unit is a unit that calculates a position and / or orientation of the measured object in a three-dimensional space. The imaging means;
The information processing apparatus according to claim 1, further comprising: an acquisition unit configured to acquire the two-dimensional image by imaging the object to be measured by the imaging unit. Based on a shape model representing the shape of the measured object, predicting means for predicting image degradation of the measured object that occurs in an image obtained by imaging the measured object by the imaging means;
Obtaining means for obtaining three-dimensional coordinates of a three-dimensional measurement point on the surface of the object to be measured;
An evaluation unit that evaluates image degradation at a position of a point group projected on the two-dimensional image based on the three-dimensional coordinates acquired by the acquisition unit in the two-dimensional image captured by the imaging unit;
Search means for searching for correspondence between the three-dimensional measurement points and model features of the shape model;
Calculation means for calculating the accuracy of correspondence between the model feature and the three-dimensional measurement point based on the image deterioration predicted by the prediction means and the image deterioration evaluated by the evaluation means;
Information comprising: a collating means for collating the shape model with a three-dimensional measurement point indicating the shape of the object to be measured based on the accuracy of correspondence between the model feature and the three-dimensional measurement point; Processing equipment. A measuring device that measures the three-dimensional coordinates of the three-dimensional measuring point;
The information processing apparatus according to claim 13, wherein the acquisition unit acquires dimensional coordinates from the measurement apparatus.   15. The apparatus according to claim 1, further comprising a control unit that controls a robot arm based on an estimation result of the position and orientation of the measured object estimated by the collating unit. Information processing device. Based on a shape model representing the shape of the measured object, a predicting step of predicting image degradation of the measured object that occurs in an image obtained by imaging the measured object by an imaging unit;
A search step for searching for a measurement data feature corresponding to a model feature of the shape model for a two-dimensional image obtained by imaging the object to be measured by the imaging means;
An evaluation step of evaluating image degradation using the two-dimensional image for the measurement data feature searched in the search step;
Based on the image degradation predicted in the prediction step and the image degradation evaluated in the evaluation step, a calculation step for calculating the accuracy of correspondence between the model feature and the measurement data feature;
Control of an information processing apparatus, comprising: a collation step of collating the shape model with an object to be measured in the two-dimensional image based on the accuracy of correspondence between the model feature and the measurement data feature Method. Based on a shape model representing the shape of the measured object, a predicting step of predicting image degradation of the measured object that occurs in an image obtained by imaging the measured object by an imaging unit;
An acquisition step of acquiring three-dimensional coordinates of a three-dimensional measurement point on the surface of the object to be measured;
An evaluation step of evaluating image degradation at the position of the point cloud projected on the two-dimensional image based on the three-dimensional coordinates acquired in the acquisition step using the two-dimensional image captured by the imaging means;
A search step for searching for correspondence between the three-dimensional measurement points and model features of the shape model;
A calculation step of calculating an accuracy of correspondence between the model feature and the three-dimensional measurement point based on the image deterioration predicted in the prediction step and the image deterioration evaluated in the evaluation step;
A collation step of collating the shape model with a three-dimensional measurement point indicating the shape of the object to be measured based on the accuracy of correspondence between the model feature and the three-dimensional measurement point. A method for controlling a processing apparatus.   The program for making a computer perform each process of the control method of the information processing apparatus of Claim 16 or 17.

Images