JP5698815B2 - Information processing apparatus, information processing apparatus control method, and program - Google Patents

Description

The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

  In recent years, with the development of robot technology, robots are beginning to perform complex tasks (for example, assembly of industrial products) that have been performed by humans instead. Such a robot performs assembly by gripping a part with an end effector such as a hand.

  In order for the robot to grip the part, it is necessary to measure (estimate) the relative position and orientation between the part to be gripped and the robot (hand). Such position and orientation measurements are not only used when the robot grips a part, but also for various purposes such as self-position estimation for the robot to move autonomously and alignment of the real space and virtual object in augmented reality. Used for

  In the measurement of position and orientation, a technique using a two-dimensional image captured by a camera or a distance image obtained from a distance sensor is known. For example, measurement by model fitting is known. In this measurement, a three-dimensional shape model of an object is applied to a feature or distance image detected from a two-dimensional image.

  In model fitting for a two-dimensional image, a position and orientation are measured by applying a projection image obtained by projecting a three-dimensional shape model onto the image to features detected from the two-dimensional image. In model fitting for distance images, each point in the distance image is converted into a 3D point group having 3D coordinates, and the position and orientation are measured by applying a 3D shape model to the 3D point group in 3D space. To do.

  As a method for measuring the position and orientation using a two-dimensional image, a method for measuring the position and orientation of a camera using an edge is known (Non-Patent Document 1). In this method, the three-dimensional shape of an object is represented by a set of line segments (wireframe model), and the projected position of the object is measured by applying a projected image of the three-dimensional line segment to the edge detected on the image. To do. Specifically, first, a three-dimensional line segment is projected onto an image based on approximate values of position and orientation, and edge detection is performed in the vicinity of the projected line segment. Next, the position and orientation of the target object are measured by nonlinear optimization so that the total sum of the distance between the projected image of the line segment based on the approximate values of the position and orientation and the corresponding edge on the image is minimized. To do.

  On the other hand, a method using an ICP (Iterative Closest Point) algorithm is known as a method for measuring position and orientation using a distance image (Non-patent Document 2). In this method, the position and orientation of an object are measured by fitting a three-dimensional shape model of the object to the three-dimensional point cloud data converted from the distance image. A search for the geometric feature of the 3D geometric model closest to each 3D point based on the approximate values of the location and orientation, and a position and orientation that minimizes the sum of the distance between the point and the geometric feature of the 3D geometric model Repeat the update.

  In the method of measuring the position and orientation of an object using a distance image, a high-load calculation process occurs when searching for a surface of a three-dimensional shape model corresponding to each point of point cloud data. In order to cope with this, Patent Document 1 discloses a technique that speeds up the correspondence search process when aligning a plurality of distance images. In this method, a minute surface (mesh) to which an index value is assigned is applied to the distance image, the index value is converted to a color that does not overlap, and each mesh color is drawn, and the mesh is drawn based on the shooting position and orientation of the distance image. Generate an index image. Then, a three-dimensional point group converted from another distance image is projected onto the index image based on the position and orientation at the time of shooting, and the corresponding color on the index image is obtained from the coordinate value of the projected three-dimensional point. To do. Thereafter, the index value corresponding to the acquired color is inversely converted to associate the mesh with the three-dimensional point converted from the distance image. As a result, the correspondence search is speeded up.

JP 2006-202152 A

T. Drummond and R. Cipolla, “Real-time visual tracking of complex structures,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.24, no.7, pp.932-946, 2002. P. J. Besl and N. D. McKay, “A method for registration of 3-D shapes,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol.14, no.2, pp.239-256, 1992.

  In the above-described alignment method using the index image (Patent Document 1), in order to align the distance images, all three-dimensional points extracted from the distance image are projected onto the two-dimensional image for association. Therefore, for example, when measuring the position and orientation of the target object from an image including an object other than the target object, all measurement points including measurement points other than the target object must be densely projected. There is a lot of waste.

  In the case of the technique disclosed in Patent Document 1, depending on the three-dimensional shape model, it is necessary to re-mesh the model so that the mesh does not collapse during projection. Therefore, it is difficult to directly use the CAD model of the target object as a three-dimensional shape model. Furthermore, dedicated hardware such as a GPU (Graphics Processing Unit) is required to draw a mesh on a two-dimensional plane at high speed.

  Furthermore, in the case of the method of Non-Patent Document 1, all the line segments that are geometric features of the three-dimensional shape model must be projected onto the image, including the line segments on the back surface (the part that is not measured) of the target object. . Therefore, even in the case of the technique of Non-Patent Document 1, processing is wasted.

  Here, the method using a two-dimensional image is suitable for an environment with many artificial objects such as a straight line, and the method using a distance image has, for example, a plurality of smooth surfaces. It can be said that it is suitable for an object.

  When using a two-dimensional image and using a distance image, the nature of the information to be measured is different. Therefore, the position and orientation measurement accuracy can be obtained by combining the model fitting to the two-dimensional image and the model fitting to the distance image. Improvement can be expected. In the model fitting to the two-dimensional image, as described above, a geometric feature of a three-dimensional shape model such as an edge is projected onto the two-dimensional image, and a process for searching for the corresponding geometric feature on the two-dimensional image is performed. In model fitting to a distance image, as described above, measurement points are projected instead of geometric features of a three-dimensional shape model. That is, the correspondence method in both is different.

  For this reason, it is not possible to perform association processing in the same framework, and when measuring the position and orientation of a target object using both a two-dimensional image and a distance image, association must be performed independently. It was.

  The present invention has been made in view of the above-described problems, so as to improve the efficiency of calculations performed when associating a three-dimensional shape model with measurement data, and to speed up the calculation of the position and orientation of the target object. Technology provided.

In order to solve the above problem, an information processing device according to one embodiment of the present invention provides:
An image acquisition means for acquiring two-dimensional image including the distance image and the target object indicates the distance to the Target object,
Obtaining means for obtaining a rough position and orientation of the target object;
On the distance image and the two- dimensional image corresponding to the geometric feature of the three- dimensional shape model of the target object on the basis of the approximate position and orientation of the target object and the photographing parameters at the time of photographing the distance image and the two- dimensional image. and associating means for associating explore the features of the above,
And deriving means for deriving the position and orientation of the target object based on the geometric features of the three-dimensional shape model associated with feature on the distance image and on the two-dimensional image by said correlating means,
It is characterized by comprising.

  According to the present invention, it is possible to improve the efficiency of calculations performed when a three-dimensional shape model is associated with measurement data. Thereby, the calculation speed of the position and orientation of the target object can be improved.

The figure which shows an example of a structure of the three-dimensional measuring apparatus concerning one embodiment of this invention. The figure which shows an example of the matching process by the correspondence search part 150 shown in FIG. The figure which shows an example of the matching process by the correspondence search part 150 shown in FIG. The flowchart which shows an example of operation | movement in the three-dimensional measuring apparatus 10 shown in FIG. FIG. 10 is a diagram illustrating an example of association processing according to the second embodiment.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
FIG. 1 shows an example of the configuration of a three-dimensional measuring apparatus according to an embodiment of the present invention. Note that the three-dimensional measuring apparatus 10 has a built-in computer. The computer includes main control means such as a CPU, and storage means such as ROM (Read Only Memory), RAM (Random Access Memory), and HDD (Hard Disk Drive). In addition, the computer may include other input / output means such as buttons, a display, or a touch panel, and communication means such as a network card. These components are connected by a bus or the like, and are controlled by the main control unit executing a program stored in the storage unit.

  The three-dimensional measuring apparatus 10 projects (arbitrary) points on the geometric features of the three-dimensional shape model onto a two-dimensional plane, and the two-dimensional plane (distance) in an area within a predetermined range from the projected point on the two-dimensional plane. Search for geometric features on the image. Thereby, the geometric feature of the three-dimensional shape model is associated with the geometric feature on the distance image, and the position and orientation of the target object are calculated based on the correspondence.

  Here, the three-dimensional measurement apparatus 10 includes an image acquisition unit 110, an approximate value acquisition unit 120, a model holding unit 130, a geometric feature selection unit 140, a correspondence search unit 150, and a position / orientation calculation unit 160. Configured.

  The image acquisition unit 110 acquires a distance image. In the present embodiment, the image acquisition unit 110 is realized by, for example, a distance sensor that captures a distance image. However, the image acquisition unit 110 is not limited to this, and simply acquires a distance image captured by an external distance sensor. Also good. The distance image is an image in which each pixel has depth information (distance information). The distance sensor may be an active type in which reflected light of laser light or slit light irradiated on the object is photographed with a camera and the distance is measured by triangulation. The distance sensor is not particularly limited as long as it can capture a distance image. For example, a Time-of-flight method that uses the time of flight of light may be employed, or a passive method that calculates the depth of each pixel by triangulation from an image captured by a stereo camera may be employed. In addition, it is assumed that the shooting parameters of the distance sensor (view angle, resolution, focal length, etc. at the time of shooting) are known. The shooting parameters are used in the correspondence search unit 150 described later.

  The approximate value acquisition unit 120 acquires approximate values of the position and orientation of the object with respect to the three-dimensional measurement apparatus 10. In the present embodiment, the position and orientation of the object with respect to the three-dimensional measurement apparatus 10 represent the position and orientation of the object with reference to the image acquisition unit 110, but are not necessarily based on the image acquisition unit 110. For example, if the relative position and orientation of the object with respect to the coordinate system of the image acquisition unit 110 are known and the position and orientation do not change, other parts in the three-dimensional measurement apparatus 10 may be used as a reference.

  In this embodiment, as the approximate value of the position and orientation of the object, the measurement value measured by the three-dimensional measurement apparatus 10 in the past (for example, immediately before) from the object is used. Is not necessarily such a value. For example, a time-series filtering process (for example, a linear filter or a Kalman filter) is performed on the measured values of the position and orientation of the object measured in the past, and the motion speed and angular velocity of the object are estimated. A value obtained by predicting the position and orientation of the object based on the estimation result may be used as the approximate value. The position and orientation of the object obtained from the sensor may be approximate values. Here, the sensor only needs to be able to measure the position and orientation of the object with six degrees of freedom, and the method (for example, magnetic type, optical type, ultrasonic type) is not particularly limited. If the approximate position or posture where the object is placed is known in advance, the value may be used as the approximate value. Furthermore, an approximate value of the position and orientation of the object may be measured by performing object recognition processing on a captured image or a distance image obtained by capturing a scene including the object.

  The model holding unit 130 holds three-dimensional geometric model data (hereinafter also referred to as a three-dimensional shape model) of a target object (sometimes simply referred to as an object). Since a three-dimensional shape model may be the same as that used in the past, detailed description is omitted here. For example, a set of points, information on a surface formed by connecting each point, and lines constituting the surface Defined by minute information.

  The geometric feature selection unit 140 selects a point on an arbitrary geometric feature from the three-dimensional shape model of the target object. In this embodiment, it is assumed that the three-dimensional shape model is composed of a plurality of NURBS (Non-Uniform Rational B-Spline) curved surfaces, and the geometric features of the model are uniformly sampled from each NURBS curved surface. And the normal. Of course, the present invention is not limited to this. For example, the geometric feature of the three-dimensional shape model may be a NURBS curved surface itself, or may be a mesh if the three-dimensional shape model is represented by a mesh.

  The geometric feature selection unit 140 performs the above selection based on information that prescribes geometric features that are easy to measure from a three-dimensional shape model and measurement data, and geometric features that are effective for calculation when calculating the position and orientation. May be. In other words, any geometric feature that can be used when the position and orientation calculation unit 160 described later calculates the position and orientation of the target object may be used. Since an arbitrary geometric feature is selected by the geometric feature selection unit 140, even if the position and orientation are measured for an image obtained by photographing an object other than the target object, the association is performed without increasing the calculation amount. Yes.

  The geometric feature selection unit 140 selects a point on the geometric feature of the three-dimensional shape model based on the imaging parameters of the distance sensor and the approximate values of the position and orientation of the object acquired by the approximate value acquisition unit 120. Thereby, only the geometric features that can be measured from the distance image are selected.

  Specifically, the three-dimensional shape model is drawn from all directions, and the geometric features of the three-dimensional shape model seen from each direction are registered in association with each direction. Accordingly, the geometric feature selection unit 140 selects a geometric feature registered in the direction closest to the line-of-sight vector calculated from the approximate values of the position and orientation of the object and the imaging parameters. Note that the inner product of the vector of each direction and the normal of the geometric feature may be compared, and only the geometric feature in which the direction vector and the normal of the geometric feature are opposed may be registered.

  Note that the selection of points on the geometric feature by the geometric feature selection unit 140 may be performed based on a user instruction. That is, the user may select manually while referring to the GUI on which the three-dimensional shape model of the target object is displayed. Further, the center of the surface of each geometric feature may be selected as a point on the geometric feature of the three-dimensional shape model. Other feature detectors as disclosed in “AE Johnson and M. Hebert,“ Efficient multiple model recognition in cluttered 3-d scenes, ”Proc. Computer Vision and Pattern Recognition, pp. 671-677, 1998.] May be selected as a point on the geometric feature of the three-dimensional shape model, and further on the distance image based on the approximate value of the position and orientation of the object acquired by the approximate value acquisition unit 120. In other words, a 3D shape model is projected on the 2D image based on the approximate values of the position and orientation of the object and the imaging parameters of the distance sensor. The points on the three-dimensional shape model sampled uniformly on the three-dimensional image are back-projected to the three-dimensional space, thereby calculating and selecting the points on the geometric feature that are uniform on the distance image. From the surface of the 3D shape model The method is not particularly limited as long as a point on the geometric feature of the model is selected.

  The correspondence search unit 150 associates the geometric feature of the three-dimensional shape model with the geometric feature on the distance image. Then, a set of the geometric feature of the associated three-dimensional shape model and the geometric feature on the distance image is output.

As shown in FIG. 2, the correspondence search unit 150 projects the point 240 selected by the geometric feature selection unit 140 onto a two-dimensional plane (projection plane) 220. This projection is performed based on the known shooting parameters (field angle, resolution, focal length, etc.) of the distance image and the approximate values of the position and orientation of the object acquired by the approximate value acquisition unit 120. “Expression 1” indicates that when the focal length is f, the three-dimensional coordinates (x, y, z) of the point selected by the geometric feature selection unit 140 are coordinates (u, v) is an equation for perspective projection conversion.
[Formula 1]
(U, v) ^T = (f · x / z, f · y / z) ^T

  Each pixel in the projection plane and the distance image uniquely corresponds to each other, and the distance to be associated from the vicinity 270 of the two-dimensional coordinates on the distance image corresponding to the point (projection point) projected on the projection plane. Geometric features on the image are searched. Since the point projection calculation can be performed by “Equation 1”, the correspondence search is also a two-dimensional search around the projection point. Therefore, since the calculation amount is small, the correspondence between the geometric features of the three-dimensional shape model and the geometric features on the distance image can be speeded up.

  As shown in FIG. 3, the search for the geometric feature on the distance image is performed within an area Ω330 formed by an elliptical area centered on the projection point. The search range may be a pixel at a projection point, or may be a rectangular shape having an arbitrary size. The size and shape of the search region Ω330 may be determined based on at least one of approximate values of the position and orientation of the object, shooting parameters of the image acquisition unit 110, and movement of the object or the image acquisition unit 110, for example. That is, the search region Ω330 may be a rectangular region or an elliptical region determined based on the line-of-sight direction of the image acquisition unit 110 and the normal direction of the geometric feature, or an elliptical region having a motion vector as a major axis. There may be.

In the present embodiment, the geometric feature on the distance image is the coordinates of a three-dimensional point obtained from depth information (distance information) included in the pixel corresponding to the projection point. As a method for searching for a geometric feature on the distance image, a measurement point in the search region Ω 330 having the shortest distance from the projection point of the measurement point 340 may be selected. "Formula 2" is an equation for obtaining the shortest distance of the projection point p = (u, v) coordinates g _i on the distance image of the measurement points measured in ^T.
[Formula 2]
argmin {|| g _i −p ||}, g _i ∈Ω

  It is not always necessary to employ this search method. For example, a feature amount is calculated by a feature detector for each pixel in the search region Ω 330 and is most similar to the feature amount of the geometric feature of the three-dimensional shape model. You may make it choose a thing. Further, for example, coordinates in a space calculated by statistical processing such as a median value or an average value of coordinates in a three-dimensional space of measurement points in the search region Ω330 may be selected. In addition, the geometric feature on the distance image is not limited to the three-dimensional point, and may be a curved surface or plane parameter applied to the measurement point in the search region Ω. Any geometric feature that can be used when calculating the position and orientation of an object may be used.

  Further, the normal of the geometric feature selected on the distance image is calculated, the calculated normal is compared with the normal of the geometric feature of the three-dimensional shape model, and the normal direction of each other exceeds a predetermined value. If they are different from each other, it may be possible to perform an erroneous correspondence exclusion process in which the geometric feature is not associated. The normal line of the geometric feature on the distance image is obtained by performing a principal component analysis on a three-dimensional point group near the geometric feature and using the third principal component as a normal line. When configured to perform the erroneous correspondence exclusion process, it is possible to reduce errors in which measurement data of different geometric features are associated with the geometric features of the three-dimensional shape model.

  The position / orientation calculation unit 160 calculates the position and orientation of the target object using a set of the geometric feature of the three-dimensional shape model and the geometric feature on the distance image associated by the correspondence search unit 150. In the calculation of the position and orientation of the target object in the present embodiment, the calculation process is repeated and the values obtained thereby are optimized. Thereby, the difference in the distance in the three-dimensional space of the set of the geometric feature of the associated three-dimensional shape model and the geometric feature on the distance image is minimized.

  In the present embodiment, the geometric feature of the three-dimensional shape model is a set of coordinates in the three-dimensional space and normal information, and the geometric feature on the distance image is the coordinates of the three-dimensional point. The geometric feature of the three-dimensional shape model is considered as a fine plane, and the distance between the plane on the three-dimensional shape model and the three-dimensional point calculated from the distance image is defined as a difference. The position and orientation of the target object are measured by minimizing this difference.

  Here, the difference depends on a geometric feature representation method. For example, if the geometric feature of the three-dimensional shape model is a point and the geometric feature on the distance image is a point, the distance between the points may be used. For example, if the geometric feature of the three-dimensional shape model is a surface and the geometric feature on the distance image is a surface, the distance between the surfaces may be used. For example, if the geometric feature of the three-dimensional shape model is a surface and the geometric feature on the distance image is a point, the distance between the surface and the point may be used. Furthermore, the difference may be a distance approximately calculated using an implicit polynomial or the like. In other words, the method is not particularly limited as long as the position and orientation of the target object are measured using an evaluation function based on a difference in distance between the geometric feature of the three-dimensional shape model and the geometric feature on the distance image. Absent.

  Next, an example of the operation in the three-dimensional measurement apparatus 10 shown in FIG. 1 will be described using the flowchart of FIG. Here, the flow of processing when measuring the position and orientation of an object will be described.

  The three-dimensional measuring apparatus 10 captures a distance image including depth information (distance information) from the distance sensor in the image acquisition unit 110. Thereby, a distance image is acquired (S110).

  When the acquisition of the distance image is completed, the three-dimensional measurement apparatus 10 acquires the approximate values of the position and orientation of the object with respect to the three-dimensional measurement apparatus 10 in the approximate value acquisition unit 120 (S120). As described above, the approximate values of the position and orientation of the object are the position and orientation measured in the past, the predicted value of the position and orientation taking into account the motion estimation results such as speed and angular velocity, the measured values by other sensors, 2 An object recognition result from a dimensional image or a distance image can be used. In this embodiment, approximate values of positions and orientations measured in the past are used as approximate values of the position and orientation of the object. Note that the processing order of S110 and S120 may be switched.

  Next, the three-dimensional measurement apparatus 10 starts the position / orientation measurement process of the target object using the distance image acquired in the process of S110, the approximate value acquired in the process of S120, and the like. In the position / orientation measurement processing according to the present embodiment, the Levenberg-Marquardt method is used. Specifically, the approximate values of the position and orientation of the target object are repeatedly corrected by iterative calculation. As a result, the difference in the distance in the three-dimensional space between the geometric feature of the three-dimensional shape model and the geometric feature on the distance image is minimized, and the value obtained thereby is used as the position and orientation of the target object. Of course, it is not limited to the Levenberg-Marquardt method, and an optimization method such as a Gauss-Newton method or a steepest descent method may be used. Also, other nonlinear optimization calculation methods such as a conjugate gradient method may be used.

  When the position / orientation measurement process starts, the three-dimensional measurement apparatus 10 first acquires the approximate value of the target object obtained in the process of S120 as an initial value in the position / orientation measurement process. Then, the geometric feature selection unit 140 selects a geometric feature of the three-dimensional shape model based on the approximate value (S130).

  When the selection is completed, the three-dimensional measurement apparatus 10 causes the correspondence search unit 150 to select 3 based on the approximate values of the position and orientation of the target object and the shooting parameters (field angle, resolution, focal length, etc.) of the distance sensor. The geometric features of the three-dimensional shape model are projected onto a distance image (two-dimensional plane). Then, the geometric features of the three-dimensional shape model are associated with the geometric features on the distance image, and a set of the geometric features of the associated three-dimensional shape model and the geometric features on the distance image is output (S140). In the position / orientation calculation unit 160, the three-dimensional measurement apparatus 10 updates the approximate values of the position and orientation of the target object using the nonlinear optimization method (S150).

  Finally, the three-dimensional measurement apparatus 10 performs convergence determination in the position / orientation calculation unit 160. If the calculated value has converged, the process ends (YES at S160), otherwise (NO at S160), the process returns to S130 again, and the above-described processes from S130 to S150 are repeated. In the convergence determination, for example, if the difference between the square sums of the error vectors before and after the position and orientation is updated is approximately “0”, it may be determined that the convergence has occurred.

  As described above, according to the first embodiment, when the geometric feature is associated with the three-dimensional shape model and the distance image, a point on the geometric feature of the three-dimensional shape model is selected, and the geometric feature on the selected geometric feature is selected. Project a point onto a distance image. Then, by searching for a geometric feature in a region within a predetermined range from the projection point on the distance image, the geometric feature of the three-dimensional shape model is associated with the geometric feature on the distance image. As a result, it is possible to efficiently perform the calculation performed when the three-dimensional shape model is associated with the measurement data.

(Embodiment 2)
Next, Embodiment 2 will be described. In the second embodiment, a case where a two-dimensional image in addition to a distance image is input as an input image will be described. FIG. 5 is a diagram showing an outline of processing for projecting the geometric feature of the three-dimensional shape model onto the two-dimensional plane and searching for the corresponding geometric feature in the two-dimensional image and the distance image.

  For both the distance image 440 and the two-dimensional image 450, the geometric features 410 and 420 of the three-dimensional shape model are projected onto the two-dimensional plane (projection plane) 430, and the geometric features on the corresponding image are projected to the two-dimensional plane. If the search is performed from above, the association can be performed.

  Therefore, it is possible to increase the efficiency of the calculation process for associating the three-dimensional shape model with the measurement data. In the present embodiment. The distance image 440 and the two-dimensional image 450 are taken from the same viewpoint. In this case, it is easy to compare the geometric features on the two-dimensional image and the geometric features on the distance image, and it is possible to determine the jump edge (isolated edge) in addition to reducing the false correspondence due to the influence of shadows. become.

  Here, the overall configuration of the three-dimensional measurement apparatus 10 according to the second embodiment is the same as that of the first embodiment, but the processing in each unit is slightly different. This point will be described. Here, the different processes will be described mainly.

  The image acquisition unit 110 acquires a two-dimensional image and a distance image. The distance image is the same as in the case of the first embodiment, and is captured by the distance sensor. The two-dimensional image is taken by a camera (imaging device) that takes a normal two-dimensional image. The captured two-dimensional image may be a grayscale image or a color image. For example, [RY Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE Calibrate in advance by the method disclosed in Journal of Robotics and Automation, vol.RA-3, no.4, 1987.].

  In the second embodiment, for example, the active distance sensor exemplified in the first embodiment is used. This is because an active distance sensor, a distance sensor using a stereo camera, and the like can simultaneously capture a normal two-dimensional image. Therefore, the two-dimensional image and the distance image are images taken from the same viewpoint. However, the two-dimensional image and the distance image do not necessarily have to be taken from the same viewpoint. If the imaging position and orientation of the imaging device in the two-dimensional image and the distance image are close and the geometric relationship between the two devices is known, either the two-dimensional image or the distance image is determined based on the geometric relationship between the two devices. An image obtained by performing projective transformation and capturing an image obtained from the same viewpoint may be used.

  The correspondence search unit 150 associates the geometric feature of the three-dimensional shape model selected by the geometric feature selection unit 140 with the geometric feature on the two-dimensional image and the distance image, and the geometric feature of the associated three-dimensional shape model. And a set of geometric features on the distance image and the two-dimensional image are output. As in the first embodiment, the correspondence search unit 150 performs the above-described correspondence by projecting the geometric feature of the three-dimensional shape model selected by the geometric feature selection unit 140 onto a two-dimensional plane (projection plane). The association between the three-dimensional shape model and the distance image is the same process as in the first embodiment. In associating the three-dimensional shape model with the two-dimensional image, an edge may be used as a geometric feature. Since the prior art may be used for the association by the edge, the detailed description is omitted here. It should be noted that feature detection as disclosed in [C. Harris and M. Stephens, “A combined corner and edge detector,” Proc. 4th Alvey Vision Conf., Pp.147-151, Manchester, UK, 1988.] It may be a feature point detected by a vessel. In addition, any geometric feature that can be used when the position and orientation calculation unit 160 measures the position and orientation of the target object may be used.

  Since the two-dimensional image and the distance image according to the present embodiment are images taken from the same viewpoint, each pixel in the projection plane, the two-dimensional image, and the distance image uniquely corresponds. Therefore, as in the first embodiment, the geometric feature on the image corresponding to the geometric feature of the three-dimensional shape model is two-dimensionally searched with the search region Ω within the predetermined range (region periphery) of the projection point. The search area Ω may be a rectangular or elliptical area as in the first embodiment. When searching for an edge on a two-dimensional image, a one-dimensional search may be performed on a line having an arbitrary length in the normal direction of the edge.

  Furthermore, the erroneous correspondence exclusion process described in the first embodiment may be performed on the two-dimensional image. The erroneous correspondence exclusion processing is performed based on, for example, comparison of feature amounts or edge direction. Also, for example, if a feature corresponding to an edge or feature amount detected on a two-dimensional image is not detected from a distance image, no association is performed assuming that the feature is an edge or feature amount caused by the influence of a shadow or the like. It is also possible to perform such a false correspondence exclusion process.

  Moreover, you may make it perform the process which determines whether the edge detected in the two-dimensional image is a jump edge using a distance image. Since the distance image and the two-dimensional image according to the present embodiment are taken from the same viewpoint, erroneous correspondence exclusion processing can be efficiently performed by using the two-dimensional image and the distance image together.

  The position / orientation calculation unit 160 calculates the position and orientation of the target object using a set of the geometric features of the three-dimensional shape model associated with the correspondence search unit 150 and the geometric features on the two-dimensional image and the distance image. To do. In the calculation of the position and orientation of the target object in the present embodiment, the calculation process is repeated to optimize the value obtained thereby. Accordingly, the difference in the distance between the geometric feature of the three-dimensional shape model and the geometric feature on the two-dimensional image in the three-dimensional space, and the distance between the geometric feature of the three-dimensional shape model and the geometric feature on the distance image in the three-dimensional space. Minimize the evaluation function based on both of the differences.

  In addition, since the depth information is not included in the two-dimensional image, when the geometric feature of the two-dimensional image is projected onto the three-dimensional space, for example, the projection is performed without the depth information. Of course, depth information may be assigned to each geometric feature of the two-dimensional image using a predetermined algorithm, a predetermined value, or the like.

  If the method is to measure the position and orientation of the target object based on the evaluation function based on the difference between the geometric feature of the three-dimensional shape model and the geometric feature on the two-dimensional image and the distance image, the method is It doesn't matter. Note that the flow of the position and orientation measurement processing according to the second embodiment is the same as that of the first embodiment, and thus the description thereof is omitted here.

  In the second embodiment, the geometric feature of the three-dimensional shape model to be associated may be a curved surface, a plane, or a curve. However, in order to project a complex shape such as a NURBS curved surface often used in a CAD model onto a two-dimensional plane, processing such as tessellation or cutting of the curved surface with contour lines is required, and dedicated hardware for geometric calculation such as GPU is required. If there is no wear, the time required for arithmetic processing will be great. Therefore, as in the first embodiment, the geometric features of the three-dimensional shape model are expressed as points or line segments, and the points or line segments are projected onto a two-dimensional plane. This projection speeds up the association without requiring dedicated hardware.

  As described above, according to the second embodiment, similarly to the first embodiment, it is possible to efficiently perform the calculation performed when the three-dimensional shape model is associated with the measurement data.

  The above is an example of a typical embodiment of the present invention, but the present invention is not limited to the embodiment described above and shown in the drawings, and can be appropriately modified and implemented within the scope not changing the gist thereof. .

It should be noted that the present invention can also take the form of, for example, a system, apparatus, method, program, or storage medium. Specifically, the present invention may be applied to a system composed of a plurality of devices, or may be applied to an apparatus composed of a single device.
(Other embodiments)
The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media. Then, the computer (or CPU, MPU, GPU, etc.) of the system or apparatus reads out and executes the program.

Claims (15)

An image acquisition means for acquiring two-dimensional image including the distance image and the target object indicates the distance to the Target object,
Obtaining means for obtaining a rough position and orientation of the target object;
On the distance image and the two- dimensional image corresponding to the geometric feature of the three- dimensional shape model of the target object on the basis of the approximate position and orientation of the target object and the photographing parameters at the time of photographing the distance image and the two- dimensional image. and associating means for associating explore the features of the above,
And deriving means for deriving the position and orientation of the target object based on the geometric features of the three-dimensional shape model associated with feature on the distance image and on the two-dimensional image by said correlating means,
An information processing apparatus comprising: Before SL on the basis of the imaging parameters and the approximate position and orientation of the target object, further comprising a projection means for projecting geometric features of the three-dimensional shape model of the object on the distance image and the upper two-dimensional image,
Said correlating means is an information processing apparatus according to claim 1, characterized in that associating searches the feature on said distance image and the two-dimensional image corresponding to the geometric features projected by the projection means . Selecting means for selecting a geometric feature of a three-dimensional shape model of the target object;
The information processing apparatus according to claim 2, wherein the projection unit projects the geometric feature selected by the selection unit onto the distance image and the two-dimensional image. 4. The association unit according to claim 2 or 3, wherein the association unit searches and associates the feature on the distance image and the two-dimensional image in an area within a predetermined range from the position of the projected geometric feature. Information processing device. The association means includes
The direction of the normal of the geometric features of the three-dimensional shape model is compared with the direction of the normal line of the feature on said distance image, if the normal direction of each other they are different beyond a predetermined value, the the information processing apparatus according to any one of claims 1 to 4 for the geometric features, characterized in that not performed corresponds with. The two- dimensional image is taken from the same viewpoint as the distance image,
The association means includes
Wherein detecting a two-dimensional image from said the two-dimensional image as a feature of the object edge, the edge features of the distance image corresponding to the edge of the detected If not, detected from the two-dimensional image an information processing apparatus according to any one of claims 1 to 5, characterized in that not performed correspondence between geometric features of the three-dimensional shape model. Geometric features of the three-dimensional shape model, the information processing apparatus according to any one of claims 1 to 6, characterized in that it is represented by a point or a line segment. Said deriving means, said geometric features of the three-dimensional shape model associated with associating means and the deviation between the feature on said distance image, the three-dimensional shape model that is associated by the associating means wherein the geometric features based on the deviation between the characteristic of the two-dimensional image processing apparatus according to any one of claims 1 to 7, characterized in that to derive the position and orientation of the target object. And geometric features of the the associated said 3D model, the distance and the deviation of the feature in the image, the correlating means the geometric characteristics of the 3D model that is associated with said on the two-dimensional image of to minimize the displacement of said front Symbol correlating means, the information processing apparatus according to claim 8, further comprising a control means for controlling repetition of processing by the deriving means. The information processing apparatus according to claim 8, wherein the shift is a difference in distance in a three-dimensional space. A method for controlling an information processing apparatus,
Image obtaining means, a step of acquiring a two-dimensional image including the distance image and the target object indicates the distance to the Target object,
An obtaining unit obtaining a rough position and orientation of the target object;
An associating means on the distance image corresponding to the geometric feature of the three- dimensional shape model of the target object based on the approximate position and orientation of the target object and the shooting parameters at the time of shooting each of the distance image and the two- dimensional image and a step of associating explore the feature on the two-dimensional image,
Derivation means, and deriving the location and orientation of the target object based and geometric features of the the associated said three-dimensional shape model to the feature on the distance image and on the two-dimensional image,
A method for controlling an information processing apparatus, comprising: Program for executing each means of the information processing equipment according to the computer in any one of claims 1 to 10. Image acquisition means for acquiring distance information indicating a distance to the target object and a two-dimensional image including the target object;
Obtaining means for obtaining a rough position and orientation of the target object;
By projecting the geometric feature of the three-dimensional shape model of the target object to the distance information and the two-dimensional image based on the approximate position and orientation of the target object, respectively on the distance information with respect to the projected geometric feature and the An associating means for searching and associating features on the two-dimensional image;
Derivation means for deriving the position and orientation of the target object based on the geometric features of the three-dimensional shape model associated by the association means and the features on the distance information and the two-dimensional image;
An information processing apparatus comprising: A method for controlling an information processing apparatus,
An obtaining unit obtaining distance information indicating a distance to the target object and a two-dimensional image including the target object;
An obtaining unit obtaining a rough position and orientation of the target object;
The associating means projects the geometric feature of the three-dimensional shape model of the target object on the distance information and the two-dimensional image based on the approximate position and orientation of the target object, respectively. Searching for and correlating features on the distance information and the two-dimensional image;
Deriving means for deriving the position and orientation of the target object based on the geometric features of the associated three-dimensional shape model and the features on the distance information and the two-dimensional image;
A method for controlling an information processing apparatus, comprising: A program for causing a computer to execute each unit of the information processing apparatus according to claim 13.