JP2008309595A - Object recognizing device and program used for it - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To perform pose calculation of a stereoscopic object surely in a short time based on the photographed images in an object recognizing device, where the pose of the stereoscopic object is computed by utilizing a plurality of fixed points of which the relative position relation with the stereoscopic object is determined and its relative position relation is known, without requiring a complicated constitution. <P>SOLUTION: This device includes: a photographing device 3 for photographing a stereoscopic object 2 where two or three fixed points are set; an inclination meter 4 for detecting the inclination of the photographing device 3 on the horizontal two axes respectively; a computing part for extracting each fixed point from the images photographed by the photographing device 3 to compute the pose of the stereoscopic object 2 using the position information of each fixed point on the images and each angle information on the two axes output from the inclination meter as parameters. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an object recognition device for specifying a pose from a three-dimensional object projected on a two-dimensional captured image.

  A method for identifying the pose of a 3D object projected from a two-dimensional captured image is used in a wide range of fields such as photogrammetry (photogrammetry), robotics, machine vision, and many researchers. Various methods have been proposed.

  As an example, a perspective n-point method is known in which the pose of an object is calculated from n known points (hereinafter referred to as fixed points) on the object. This perspective n-point solution method results in obtaining a rotation matrix and a movement vector indicating the correlation between the camera coordinate system and the object coordinate system, which is usually expressed by a fourth-order polynomial. An inverse problem is solved by an iterative optimization method based on the positions of fixed points or more on the image, and the pose of the object is calculated. The pose indicates the relative posture and / or position of the object with respect to the camera. If the three-dimensional coordinates of each fixed point in the spatial coordinate system with respect to the camera are obtained, the pose is calculated. Can do.

  However, since it takes a considerable amount of time to calculate from four fixed points, it becomes difficult to calculate the pose of an object in each frame of a moving image in real time, for example.

  Therefore, as shown in non-patent literature, a perspective three-point method is proposed in which only three fixed points are used. However, with this method, there are often cases where a plurality of solutions are calculated or cannot be solved. Conversely, a solution cannot be obtained unless it is under certain conditions.

  In both methods, the optimization method is used to solve the inverse problem, but depending on the initial value setting at that time, it may take a lot of time, or in some cases, an incorrect solution may be obtained or solved. There may be no.

Fischler, M.A., Bolles, R.C .: Randomsample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.Commun. ACM 24 (6) (1981) 381-395

  The present invention has been made to solve such a problem, and it is possible to calculate the pose of the three-dimensional object more reliably and in a short time from two to three fixed points on the three-dimensional object. This is the main intended issue.

  In order to solve this problem, the present invention provides the following means.

  That is, the object recognition device according to the present invention calculates a pose of a three-dimensional object using a plurality of fixed points whose relative positional relationship with a three-dimensional object is determined and whose relative positional relationship is known. ,

  An imaging device that captures the three-dimensional object in which two to three fixed points are set, an inclinometer that detects each tilt around two horizontal axes of the imaging device, and each of the images captured by the imaging device. A calculation unit that extracts fixed points and calculates the position of each fixed point on the image and the angle information about the two axes output from the inclinometer as parameters; It is characterized by that.

  According to such a configuration, since the angle information from the inclinometer is used, the reliability of the pose calculation result can be ensured even if there are two or three fixed points, and the fixed points can be set to a small number. It is possible to calculate the pose at. From this, for example, it becomes possible to easily calculate the pose of the object in each frame of the moving image in real time. This allows for various applications. For example, in a device that requires complicated operations, such as an analytical device, an information device, and a system device, when the user captures the device (that is, a three-dimensional object) with video, the device is recognized in real time, and the device operation location The operation method can be indicated by an arrow on the image. Furthermore, in terms of configuration, since it can be realized by software processing only by attaching an inclinometer to the imaging device, it does not promote significant complication from existing devices.

  When the calculation unit calculates the pose of the solid object using an iterative optimization method, the calculation unit uses three fixed points in order to be able to perform the calculation in a short time and more reliably. The position of each fixed point is set so that a straight line connecting two fixed points of the three fixed points is substantially vertical, and a straight line connecting the remaining one fixed point and one fixed point of the two fixed points is substantially horizontal. In addition, the calculation unit calculates a fixed point coordinate that should be originally calculated from the angle formed by the vertical axis and the image plane of the imaging device, using one angle information from the inclinometer instead of the angle, It is preferable that the fixed point coordinates calculated by using the assumed fixed point coordinates as an initial value are subjected to the iterative optimization.

  Because, by setting the three fixed points as described above, the angle formed by the vertical axis and the image plane of the imaging device approximates the value of one angle information output from the inclinometer. This is because the deviation of the initial value calculated by using the final solution can be prevented, and it is possible to reduce the time and the certainty of the iterative optimization calculation.

  According to the present invention configured as described above, it is possible to perform a quick and reliable pose calculation of a three-dimensional object based on a captured image without requiring a complicated configuration, and can be applied to various technologies. Can be greatly improved.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  An object recognition apparatus 1 according to the present embodiment recognizes a pose of a three-dimensional object 2 from an image of the three-dimensional object 2, and as shown in FIG. 3, an inclinometer 4 that detects each inclination of the imaging device 3 around two horizontal axes, and image data captured by the imaging device 3, and calculates the orientation of the three-dimensional object 2 from the image data And an information processing device 5. The object recognition device 1 can be applied to three modes in which the imaging device 3 is stationary and the three-dimensional object 2 is moving, the imaging device 3 is moving and the three-dimensional object 2 is stationary, and both are moving. However, in the following, the second example, that is, an example in which the object 2 is recognized when the imaging device 3 moves and the three-dimensional object 2 is stationary will be described.

  Here, the imaging device 3 is, for example, a video camera and captures a moving image of the three-dimensional object 2. As will be described later, the object recognition device 1 calculates the orientation of the three-dimensional object 2 from each piece of still image (frame) data constituting a moving image. A device that captures a still image, such as a machine, may be used, and in that case, imaging is performed in a state where both are stationary.

  The inclinometer 4 is called an inclinometer or a clinometer, and is attached to the imaging device 3 here to detect the inclination of the imaging device 3 around two orthogonal horizontal axes of the imaging device 3. Various types of inclinometers 4 having different measurement methods have been developed as this type of inclinometer 4, but in principle any type may be used. Of course, a sensor with good detection accuracy and a short measurement time is preferable. In this embodiment, a sensor with a precision of 0.001 ° to 0.1 ° is used.

  The information processing device 5 is, for example, a general-purpose computer including a CPU, an output device such as a memory and a display, an input device such as a keyboard, and the like according to an object recognition program stored in the memory. When the devices cooperate, the device functions as a calculation unit that calculates the posture of the three-dimensional object 2 from the image data.

  The outline of the function of this calculation unit will be described. The perspective n-point solution method is used to calculate the posture of the solid object 2 shown in the image. Here, the 3D set for the solid object 2 is calculated. Use a fixed point. The fixed point is one in which the relative positional relationship with the three-dimensional object 2 does not change and the relative positional relationship is known, that is, stored in the memory of the computer in advance. It is a characteristic point that can be recognized by (computer). For example, a characteristic shape portion (for example, a corner portion) of the three-dimensional object 2 may be set as a fixed point, or a display serving as a mark may be attached to the surface of the object 2 and set as a fixed point. However, it is necessary to be able to identify each fixed point (it is not enough to simply distinguish it from a fixed point, and if it is not unique, the information processing device 5 can identify which fixed point it is from the image data) Must not). In this embodiment, a recognition unit (not shown) that is a functional unit is provided in the information processing apparatus 5, and the recognition unit identifies each fixed point by performing image processing on image data.

  Next, a more specific posture calculation process by the calculation unit will be described in detail with an explanation of the principle.

<Premise>
As shown in FIG. 2, if the coordinate system of the object 2 is (M ₀ , u, v, w) and the coordinate system of the imaging device 3 is (O, i, j, k), the object coordinate system is used. Fixed point on object 2
Mi = (Ui, Vi, Wi) ^T is converted into coordinates (Xi, Yi, Zi) ^T in the coordinate system of the imaging apparatus 3 by the following conversion formula (Equation 1).

Here, R is a rotation matrix and T is a translation vector. The subscript i is for distinguishing three fixed points and takes three values of 0, 1, and 2.

First, from the rotation matrix R, the rotation matrix R is defined by the equation (Equation 2) as is well known.
α, β, and γ indicate rotation angles around the x, y, and z coordinate axes in the world coordinate system (object coordinate system), respectively. Rx (α), Ry (β), and Rz (γ) are the coordinate axes. Represents the transformation matrix when rotating around.

Specifically, it is as shown in the equation (Equation 3). Here, the x axis and the z axis are two horizontal axes orthogonal to each other, and y represents the vertical axis.

  By the way, since the inclinometer 4 outputs the angles α and γ, in order to calculate the rotation matrix R, only β needs to be calculated.

On the other hand, the translation vector T is expressed by the following equation (Equation 4).
Here, R ³ represents a three-dimensional space in the coordinate system of the imaging device 3.

Further, in the image G (shown in FIG. 4 (a) and the like), that is, in the two-dimensional coordinate system (C, i, j) of the video image, the coordinates at which the fixed point appears, that is, the position information m _i = (x _i , y _i ) can be calculated by the following equation (Equation 5) based on the pinhole image model.
However, f is a known value for the focal length of the imaging device 3.

<Attitude calculation using optimization method>
Based on the premise so far, the calculation unit calculates the posture of the three-dimensional object 2, but in order to do so, it is necessary to solve a non-linear quadratic or higher order function. In this embodiment, a solution is obtained by using an optimization algorithm such as Gauss-Newton method or Levenberg-Marquardt algorithm. Normally, these methods require four fixed points, but here, since the detection result of the inclinometer 4 is used, a solution can be obtained with three fixed points.

First, the following formula (formula 6) can be derived from the formula (formula 5).
Therefore, if Z ₀ and β are known, the posture of the three-dimensional object 2 can be calculated.

Here [psi _i, R ² → R, i.e. by defining a conversion function from 2D to 1D, [psi _i can be expressed by the following equation (7).

Therefore, Z ₀ and β for calculating the posture can be obtained by solving the following minimization problem of the cost function σ.

  In this embodiment, the Levenberg-Marquardt algorithm is used as a solution from the viewpoint of high robustness, but other methods may be used.

<Initial value setting>
In order to solve the minimization problem, it is necessary to give initial values of Z ₀ and β. However, any initial values may be used, and it is better to give initial values as close as possible to the final solution. The search calculation can be performed quickly and accurately.

  Specifically, the solution obtained from the previous frame (still image) in the moving image is given as an initial value. It is usually difficult to think that the posture of the three-dimensional object 2 changes greatly in a very short time between frames, because the solution in the previous frame often approximates the solution in the target frame.

  As described above, when there is a previous frame, the initial value can be easily set, but the problem of how to set the initial value in the first frame remains.

  In this regard, in the present embodiment, the initial value is set by the calculation unit using the following method.

  Here, since the solid object 2 is stationary, a straight line connecting two fixed points of the three fixed points is substantially vertical, and a straight line connecting the remaining one fixed point and one fixed point of the two fixed points is approximately. The position of each fixed point is set so as to be horizontal. In other words, as shown in FIG. 3, a straight line connecting three fixed points is configured to form a triangle, and three fixed points are set so that one of them is vertical and the other one is horizontal.

In such a configuration, Z ₀ can be expressed by the following formula (formula 9) from Thale's therem and the formula (formula 5).
Here, H is a point defined as (M ₀ M ₁ ) // (Hm ₀ ) and H∈ (OM ₁ ) as shown in FIG. As can be seen from this equation (Equation 9), Z ₀ can be calculated by calculating the translation vector T. To that end, そ の Hm ₀よ い can be calculated.

Therefore, ‖Hm ₀ ‖ can be expressed by the following equation (Equation 10).
Here, δ and s are determined depending on the value of α and the relative positions of y ₁ , y _c , and y ₀ , and Table 1 shows the relationship.

For example, if θ <0 (that is, α <0) and y ₁ ≦ y _c ≦ y ₀ (see FIG. 5), s = 1 and δ = | θ | −φ.

As can be understood intuitively from FIG. 4B, since the difference between α (which is obtained from the output from the inclinometer 4) and θ is considered to be very small, in this embodiment, Z ₀ is In order to calculate, θ is first assumed to be α, and calculation is started with Z ₀ obtained as a result as an initial value.

Next, for β, in order to obtain this value, it is necessary to calculate the rotation matrix R first. From the equations (Equation 1), (Equation 3), and (Equation 4), the following equation (Equation 11) can be derived. Note that the subscript i = 1 or 2 here.

Each coefficient is defined as follows.

As is apparent from this equation, there are two linear equations and two unknowns cos β and sin β. In this embodiment, β is obtained by using a fixed point M ₁ or M ₂ .

Next, more specific simulation results will be described.
In this embodiment, three fixed points are set so that each equilateral side forms a right-angled triangle of 10 cm, and a model equivalent to a case where the solid object 2 is stationary and the camera moves randomly, that is, a solid object on the image. A model in which 2 rotates randomly was set virtually. As the imaging device 3, a pinhole camera (focal length is 550 pixels and lens principal point coordinates are (305, 225)) is assumed. As an output signal from the inclinometer 4, a signal obtained by adding Gaussian noise having an average value of 0 and a standard deviation of 0.01 to an ideal signal was used. As imaging noise, Gaussian noise having an average value of 0 and a standard deviation of 0, 1, and 2 pixels was superimposed.

Detailed simulation conditions and the results are shown below.
Condition 1: The relative position of the three-dimensional object 2 with respect to the imaging device 3 was changed. Specifically, in the translation vector T = (Tx, Ty, Tz) ^T , Tx = 9 cm and Ty = −1 cm were fixed, and Tz / 10 was changed from 15 to 50 one by one.
Condition 2: The relative position and orientation of the three-dimensional object 2 with respect to the imaging device 3 were changed. Specifically, in translation vector T = (Tx, Ty, Tz) ^T , Tz = 30cm is fixed, Tx / 10 is changed from 2 to 20, one by one, and at each change, translation vector T and unit Ty was determined such that the angle formed with the vector i changed from 0 to 2π by π / 4.

  An example of the simulation result corresponding to condition 1 is shown in FIG. 6, and an example of the simulation result corresponding to condition 2 is shown in FIG. Here, the error from the ideal value of the posture of the solid object 2 (mean error (degree) after 500 trials) and the error from the ideal value of the position of the solid object 2 are the calculation results of the calculation unit. (The average error at the time of 500 trials, mean translation error) is plotted on the vertical axis.

  From now on, even in a noisy environment, if the imaging device 3 is close to the three-dimensional object 2, the error of the posture angle (β) actually stays below 4 °, and other angles (α, γ) accordingly. ) Can also be estimated to be small. The position error is also very small, and it is considered that there is no practical problem.

Next, a comparison result in a simulation when the pose is calculated by another conventional method and when the object recognition apparatus 1 according to this embodiment is used is shown. As another method, a method called POSIT that does not use the output from the inclinometer 4 and a method based on the old Levenberg-Marquardt algorithm (shown as Opt. Algo.) Were selected. The results are shown in Table 2. Each simulation result is an average of 1000 times, and Gaussian noise having an average value of 0 and a standard deviation of 1 pixel is superimposed as imaging noise.

In the case where there are two or three fixed points, no solution can be obtained by the conventional method. On the other hand, according to the object recognition apparatus 1, although not shown in Table 2, a probability of 60% at two fixed points, A solution of 93% probability with 3 fixed points and a probability of 99% or more with 4 fixed points or more. In Table 2, r _e is the mean attitude angle error, t _e represents the average position error. Further, the number of optimization iterations necessary for the calculation (the number of times until the solution is stabilized within a certain range) is compared with Opt. Algo. For example, 60 times or less is required, but Opt.Algo requires 120 times or more.

  Therefore, according to the object recognition apparatus 1 or the object recognition program, since the angle information from the inclinometer 4 is used, the reliability of the pose calculation result can be ensured even if there are two or three fixed points. In addition, since a fixed number of points can be set, the calculation time can be shortened.

  In particular, in the embodiment, the straight line connecting the two fixed points among the three fixed points is set to be substantially vertical, and an appropriate initial value is set as one angle information from the inclinometer 4 using the setting. Since the calculation is performed using, the effect becomes even more remarkable.

  In addition, this invention is not limited to the said embodiment or Example. For example, in the above-described embodiment, three fixed points are used. However, if the accuracy is allowed to be slightly reduced, two fixed points may be used. If accuracy is improved at the expense of calculation time, four or more fixed points may be used. May be used. When four or more fixed points are used, as described in the embodiment, the accuracy can be improved and the calculation time can be shortened as compared with the conventional one not using the inclinometer 4.

  If only three fixed points are simply set, depending on the imaging direction, one of the three fixed points may be hidden and calculation may not be possible. Therefore, a large number of fixed points (that is, four or more) are set so that there are three or more fixed points in the image no matter which direction is taken. It is more preferable that the information processing apparatus 5 selects three of the three to calculate the posture of the three-dimensional object 2. Of course, a recognition unit for identifying the fixed point to which the selected fixed point corresponds and the difference between the fixed points is necessary.

  Further, the object recognition apparatus 1 or the object recognition program of the present invention is incorporated, and in a device requiring complicated operations such as an analysis device, an information device, a system device, etc., the user can video the device (that is, the three-dimensional object 2) When an image is picked up, a real-time moving image manual system can be constructed in which the device is recognized in real time, and the operation location and operation method of the device are indicated by an arrow on the image.

  In addition, the present invention can be variously modified without departing from the spirit of the present invention.

1 is an overall schematic diagram of an object recognition apparatus according to an embodiment of the present invention. Coordinate system explanatory drawing for demonstrating the coordinate system in the embodiment. Fixed point explanatory drawing which showed the setting method of the fixed point in the same embodiment. The auxiliary figure for helping the understanding regarding the initial value setting in the same embodiment. The auxiliary figure for helping the understanding regarding the initial value setting in the same embodiment. FIG. 9 is a graph showing an example of a simulation result, in which a posture (angle) error and a position error are taken on the vertical axis. FIG. 9 is a graph showing an example of a simulation result, in which a posture (angle) error and a position error are taken on the vertical axis.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Object recognition apparatus 2 ... Solid object 3 ... Imaging device 4 ... Inclinometer 5 ... Information processing apparatus

Claims (3)

Calculating a pose of the three-dimensional object using a plurality of fixed points whose relative positional relation with the three-dimensional object is determined and whose relative positional relation is known;
An imaging device for imaging the three-dimensional object in which at least two to three fixed points are set;
An inclinometer for detecting each inclination around two horizontal axes of the imaging device;
The fixed points are extracted from the image captured by the imaging device, and the position information of the fixed points on the image and the angle information about the two axes output from the inclinometer are used as parameters. An object recognition apparatus comprising: a calculation unit that calculates a pose. The calculation unit calculates a pose using an iterative optimization method;
Using three fixed points, a straight line connecting two fixed points of the three fixed points is substantially vertical, and a straight line connecting the remaining one fixed point and one fixed point of the two fixed points is substantially horizontal. Set the position of each fixed point,
The calculation unit replaces the three-dimensional space coordinates of the fixed point that should be originally calculated from the angle formed by the vertical axis and the image plane of the imaging device, and sets the value of one angle information output from the inclinometer instead of the angle. The object recognition apparatus according to claim 1, wherein an assumption calculation is performed using the assumption calculation result as an initial value, and the iterative optimization is performed to calculate a three-dimensional spatial coordinate of a fixed point. An imaging device that captures a three-dimensional object, and an inclinometer that detects each inclination about two horizontal axes of the imaging device,
The fixed points are extracted from the captured image of the three-dimensional object in which two to three fixed points are set, the position information of the fixed points on the image, and the angle information about the two axes output from the inclinometer, An object recognition program for causing a computer to function as a calculation unit that calculates a pose of the three-dimensional object using as a parameter.