JP2013092407A - Three-dimensional coordinate acquisition device, camera attitude estimation device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve estimation accuracy of a camera attitude and reduce a processing load.SOLUTION: A three-dimensional coordinate acquisition device 100 acquires three-dimensional coordinates of a three-dimensional object that are used when the attitude of a camera is estimated from a photographed image of the three-dimensional object photographed by the camera. The three-dimensional coordinate acquisition device includes: a feature point detecting unit 111 that detects a feature point in an image plane from the photographed image; and a three-dimensional coordinate acquisition unit 113 that calculates an intersection of a straight line that passes through a focal point of the camera and the feature point in the image plane and the three-dimensional object, from an equation of the straight line and an equation of the three-dimensional object, and acquires the three-dimensional coordinates of the feature point.

Description

  The present invention relates to a three-dimensional coordinate acquisition device, a camera posture estimation device, and a program.

In recent years, AR (Augmented Reality) technology for processing a real-space image by a computer and adding more information has been used on various electronic devices (for example, a PC to which a WEB camera is connected, a mobile phone terminal with a camera). , Has come to be realized.
In the AR technique, it is necessary to estimate the camera posture (external parameter of the camera) with respect to the three-dimensional object in the camera image.
For estimating the camera posture, a technique using a reference marker or the like is known. However, a technique for estimating a camera posture with respect to a three-dimensional object without using a specific marker (markerless AR technique using a three-dimensional polygon model). ) Has also been proposed. For example, in Patent Documents 1 and 2, a 3D polygon model of a three-dimensional object is acquired in advance offline, and a camera that associates the 2D pixel coordinates in the camera image with the 3D coordinates of the 3D polygon model. A technique for determining a posture and estimating it as a camera posture is disclosed.
In general, in estimating the camera posture, the estimation accuracy is affected by whether or not exact three-dimensional coordinates corresponding to the two-dimensional pixel coordinates can be acquired. When a 3D polygon model is used, the 3D coordinates corresponding to one 2D pixel coordinate form a polygon model with a straight line passing through the point of the camera focus and one 2D pixel coordinate in the image plane. Therefore, the acquisition accuracy of the three-dimensional coordinates is affected by the accuracy of the three-dimensional polygon model itself. When a three-dimensional object (for example, a rectangular parallelepiped) whose surface is composed only of a plane is used, a highly accurate three-dimensional polygon model can be easily prepared, so that the camera posture can be estimated with relatively high accuracy. Can do.

Special table 2010-519629 JP 2011-118724 A

However, when using a three-dimensional object having a curved surface (a three-dimensional object whose surface is part or all of a curved surface. For example, a sphere, a cylinder, a cone, a torus) Errors such as deviation occur, and a highly accurate three-dimensional polygon model cannot be acquired. Therefore, when a three-dimensional object having a curved surface is used, there is a problem that the estimation accuracy of the camera posture deteriorates.
Furthermore, in addition to the problem related to camera posture estimation accuracy, when acquiring three-dimensional coordinates corresponding to two-dimensional pixel coordinates, it is necessary to specify one polygon (plane) corresponding to the two-dimensional pixel coordinates by calculation. Therefore, there is a problem that the processing load increases.
Note that the degree of error of the curved surface portion of the 3D polygon model depends on the number of planes constituting the curved surface portion. Therefore, in order to reduce the degree of error and increase the estimation accuracy of the camera posture, the curved surface portion is configured. Although it is necessary to greatly increase the number of planes, there is a problem that the processing load further increases.

  The present invention has been made in view of the above-described problems, and provides a technique for improving the estimation accuracy of the camera posture and reducing the processing load. In particular, the present invention provides a technique for improving the estimation accuracy of the camera posture and reducing the processing load when using a three-dimensional object having a curved surface.

  In order to solve the above problem, the three-dimensional coordinate acquisition apparatus according to one aspect of the present invention uses the three-dimensional object used when estimating the camera posture of the camera from a captured image of the three-dimensional object captured by the camera. A three-dimensional coordinate acquisition apparatus for acquiring a three-dimensional coordinate of a feature point detecting means for detecting a feature point in an image plane from the captured image, passing through a focus point of the camera and a feature point in the image plane And a three-dimensional coordinate acquisition means for calculating an intersection of the straight line and the three-dimensional object from the equation of the straight line and the equation of the three-dimensional object, and acquiring the intersection as a three-dimensional coordinate of the feature point. And Here, the equation of the three-dimensional object represents the surface of the entire three-dimensional object. For example, if the object is a sphere, the equation of the sphere is used. Point to.

  The three-dimensional coordinate acquisition apparatus may further include parameter calculation means for calculating the linear equation parameter and the three-dimensional object equation parameter using an internal parameter of the camera and a captured image of the three-dimensional object. It may be.

  In the three-dimensional coordinate acquisition apparatus, the three-dimensional coordinate acquisition unit acquires, as the three-dimensional coordinates, one intersection point close to the focal point of the camera among the plurality of intersection points when a plurality of the intersection points are calculated. You may make it do.

  In the three-dimensional coordinate acquisition apparatus, if the intersection point is not calculated from a feature information database that stores feature points of the three-dimensional object and a straight line that passes through the one feature point, the feature points are stored in the feature information database. It may be further provided with a deleting means for deleting from.

  In the above three-dimensional coordinate acquisition device, the three-dimensional coordinate acquisition unit uses a discriminant to determine whether the feature point is a point on a three-dimensional object when the equation for acquiring the intersection is a quadratic equation. May be determined.

  In the above three-dimensional coordinate acquisition apparatus, the solid object may have a curved surface in part or in whole.

  In the above three-dimensional coordinate acquisition apparatus, a part of the surface of the three-dimensional object is a curved surface, and the three-dimensional coordinate acquisition means uses the equation of the three-dimensional object for the curved surface portion of the three-dimensional object. Then, the three-dimensional coordinates of the feature points may be acquired using a three-dimensional polygon model for the planar portion of the three-dimensional object.

  In the three-dimensional coordinate acquisition apparatus, the three-dimensional coordinate acquisition unit acquires a position of a peripheral edge as viewed from the focal point of the camera as a three-dimensional coordinate of the feature point on the curved surface of the three-dimensional object. You may make it not.

  In the three-dimensional coordinate acquisition apparatus, the three-dimensional coordinate acquisition unit changes the threshold value of a determination formula for determining whether or not the intersection exists on the three-dimensional object, thereby changing the surface of the three-dimensional object. The position of the peripheral edge when viewed from the focal point of the camera may not be acquired as three-dimensional coordinates.

  In order to solve the above problems, a camera posture estimation device according to another aspect of the present invention is a camera posture estimation device that estimates a camera posture of a camera from a captured image of a three-dimensional object captured by the camera, A database unit that stores the two-dimensional pixel coordinates of the feature points of the three-dimensional object in the captured image captured in advance and the three-dimensional coordinates of the feature points of the three-dimensional object acquired in advance in association with each other. A feature point of the three-dimensional object is extracted from the captured image, and includes a collation unit that collates with the two-dimensional pixel coordinates in the database unit, and a camera posture estimation unit that estimates a camera posture based on the collation result, The database unit includes an equation of a straight line that passes through a focus point of the camera and a feature point in an image plane detected from the captured image, and an equation of the three-dimensional object. Et is calculated, the intersection between the straight line and the three-dimensional object, and to store the 3-dimensional coordinates of the feature points of the three-dimensional object.

  In the camera posture estimation apparatus, the three-dimensional object has a part of a surface that is a curved surface, and the database unit is calculated using an equation of the three-dimensional object for a curved surface portion of the three-dimensional object. Three-dimensional coordinates may be stored, and the three-dimensional coordinates calculated using a three-dimensional polygon model for the planar portion of the three-dimensional object may be stored.

  In order to solve the above problem, a program according to another aspect of the present invention provides a three-dimensional object of a three-dimensional object used when estimating a camera posture of the camera from a captured image of the object captured by the camera. A feature point detecting step for detecting a feature point in the image plane from the captured image, a straight line passing through the focal point of the camera and the feature point in the image plane, to the computer of the three-dimensional coordinate acquisition device that acquires the coordinates. A point of intersection between the straight line and the three-dimensional object is calculated from an equation and the equation of the three-dimensional object, and a three-dimensional coordinate acquisition step of acquiring the three-dimensional coordinate of the feature point is executed. .

  According to the present invention, it is possible to improve the estimation accuracy of the camera posture and reduce the processing load.

1 is a conceptual diagram of an AR system 1 including a camera posture estimation device 20 that is an embodiment of the present invention. 2 is an example of a functional block diagram of a camera posture estimation device 20. FIG. It is explanatory drawing for demonstrating a numerical formula model. It is an example of the functional block diagram of the three-dimensional coordinate acquisition apparatus 100 which is embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a conceptual diagram of an AR system 1 including a camera posture estimation device 20 according to an embodiment of the present invention. As shown in FIG. 1, the AR system 1 includes an imaging device 10, a camera posture estimation device 20, a display device 30, and an additional information database 90.

  The imaging device 10 is a device that acquires a captured image (also referred to as a camera image or a two-dimensional image). An example of the imaging apparatus 10 is a WEB camera as shown in FIG. However, the imaging device 10 is not an independent device such as a WEB camera, but is integrated with another device or a part of another device such as a camera module mounted on a mobile phone terminal with a camera. May be included. The imaging device 10 continuously acquires a camera image obtained by imaging the three-dimensional object (a camera image in which the three-dimensional object is captured). The imaging device 10 outputs a camera image to the camera posture estimation device 20 and the display device 30.

  The three-dimensional object used in the AR system 1 is a three-dimensional object having a curved surface (a three-dimensional object whose surface is partly or entirely a curved surface. For example, a sphere, a cylinder, a cone, or a torus). In the example shown in FIG. 1, a spherical globe is used, but a ball, a container for drinking water (for example, a can, a bottle, a plastic bottle), a printed matter (for example, an advertisement or a poster) attached to a cylinder, or the like is used. A similar AR system can be constructed.

The camera posture estimation device 20 acquires the camera image output from the imaging device 10. The camera posture estimation device 20 that has acquired the camera image estimates the relative posture of the camera with respect to the three-dimensional object, that is, the camera posture (external parameters of the camera) (details will be described later). The camera posture estimation device 20 that has estimated the camera posture outputs the camera posture (information for specifying the camera posture) to the display device 30.
In the AR system 1, when a plurality of three-dimensional objects are used, information for identifying the three-dimensional object used when estimating the camera posture is output to the display device 30 together with the camera posture.

The additional information database 90 is a storage device that stores additional information related to the three-dimensional object output to the display device 30. A PC HDD, a memory module of a portable terminal, and the like correspond to the additional information database 90.
Examples of additional information are CG, other images, and character information that are displayed superimposed on a three-dimensional object on the display screen of the display device 30. When the camera posture is estimated by the camera posture estimation device 20, additional information corresponding to the camera posture is read from the additional information database 90, output to the display device 30, and displayed on the display screen of the display device 30. It is displayed superimposed on the object.

  The additional information may be output to the display device 30 in response to a request from the display unit 30, or may be output to the display device 30 in response to control (output instruction) from the camera posture estimation device 20.

  The application example of the superimposed display differs depending on the type of the three-dimensional object. However, as shown in FIG. 1, when a globe is used as the three-dimensional object, weather information (for example, weather information) is displayed on the globe in the camera image. Overlaid display of figure, temperature distribution), geographical information (for example, information on the ground surface in a display mode in which altitude can be visually observed, information on reserve positions / reserves of various resources, past continent shape / predicted future continent shape) Application examples such as superimposing display of social information (for example, current country name, border, and border, country name update information, information indicating some power distribution / composition ratio) are assumed. Furthermore, an application example is also conceivable in which gesture recognition technology is further used to superimpose and display information (for example, country name, national flag, border line with the indicated position as the territory / territorial sea) according to the pointed position (indicated position). .

  The display device 30 is a device that posts camera images continuously acquired by the imaging device 10 to the user. An example of the display device 30 is a monitor as shown in FIG. However, the display device 30 is not an independent device such as the monitor shown in FIG. 1, but is integrated with other devices such as a display unit (display control unit and display) in a camera-equipped mobile phone terminal, or It may constitute a part of another device. Further, the display device 30 may be in the form of a head mounted display (HMD).

  When a monitor (display) is used as the display device 30, the additional information input from the additional information database 90 is superimposed and displayed on the camera image at the position corrected by the camera posture input from the camera posture estimation device 20. . When the display unit 30 uses a see-through display device 30 (for example, a see-through type HMD), only the additional information may be displayed in the field of view without displaying a camera image.

  It has been described that the camera module and the display unit mounted on the camera-equipped mobile phone terminal can be the imaging device 10 and the display device 30 of the AR system 1, but for example, a mobile terminal (notebook PC, mobile phone terminal) The entire AR system 1 may be configured on a portable game machine).

(Configuration of Camera Posture Estimation Device 20)
FIG. 2 is an example of a functional block diagram of the camera posture estimation device 20. As shown in FIG. 2, the camera posture estimation device 20 includes a database unit 100, a collation unit 200, and a camera posture estimation unit 300.

The database unit 100 includes a processing unit 110, a feature information database 120, and a three-dimensional coordinate database 130.
The feature information database 120 stores the coordinates (two-dimensional pixel coordinates) of feature information (information including feature points) of a three-dimensional object (a globe in the example of FIG. 1) in the camera image.
The three-dimensional coordinate database 130 stores the three-dimensional coordinates of the three-dimensional object. Specifically, the three-dimensional coordinate database 130 corresponds to the two-dimensional pixel coordinates (coordinates indicating the feature points of the three-dimensional object in the camera image) stored in the feature information database 120. (3D coordinates of feature points of a three-dimensional object) are stored.

The processing unit 110 executes various processes. For example, the processing unit 110 detects a feature point of the three-dimensional object from a camera image obtained by imaging the three-dimensional object (feature point detection unit), and stores the two-dimensional pixel coordinates of the feature point in the feature information database 120. deep. The processing unit 110 detects feature points using, for example, a Harris corner detector. Further, for example, the processing unit 110 acquires the three-dimensional coordinates of the feature points of the three-dimensional object (three-dimensional coordinate acquisition unit) and stores them in the three-dimensional coordinate database 130 in association with the two-dimensional pixel coordinates. In addition, the processing unit 110 provides information stored in the feature information database 120 or the three-dimensional coordinate database 130 to the collation unit 230.
Details of the database unit 100 will be described later.

  The collation unit 200 includes a feature point detection unit 210, a feature amount description unit 220, and a feature amount collation unit 230. The feature point detection unit 210 detects feature points in the camera image (including feature points of the three-dimensional object). The feature point detection unit 210 that has detected the feature points outputs the two-dimensional pixel coordinates of each feature point to the feature amount description unit 220.

The feature quantity description unit 220 calculates a local feature quantity at each feature point acquired from the feature point detection unit 210. The feature description unit 220 that has calculated the local feature outputs the calculation result to the feature verification unit 230. In addition, the feature quantity description unit 220 may calculate the local feature quantity in the same manner for the feature points recorded in the feature information database 120, and may store the local feature quantity in the feature information database 120, for example (the processing unit 110 may store the feature points). The process may be executed).
Note that the feature description unit 220 (or the processing unit 110) may calculate the local feature of feature points recorded in the database unit 100 in advance and store it in the database unit 100.

  The feature amount matching unit 230 compares the local feature amount acquired from the feature amount description unit 220 (that is, the local feature amount obtained from the camera image) with the local feature amount recorded in the feature information database 120, The feature point on the camera image and the feature point on the feature information database 120 are collated to match the feature point on the camera image with the feature point on the feature information database 120. For example, the feature amount matching unit 230 calculates a local feature amount such as SIFT or SURF for each feature point, and derives a feature point having a minimum Euclidean distance d between the local feature amounts as a feature point to be matched. .

When the matching between the feature point on the camera image and the feature point on the feature information database 120 is completed, the feature amount matching unit 230 uses the matched feature point on the camera image as the feature point on one side of the match. The two-dimensional pixel coordinates are acquired, and the three-dimensional coordinates associated with the feature points on the feature information database 120 that are the other-side feature points are acquired from the three-dimensional coordinate database 130.
The feature amount matching unit 230 outputs a match (set) between the two-dimensional pixel coordinates and the three-dimensional coordinates acquired as described above to the camera posture estimation unit 300 as a matching result.

The camera posture estimation unit 300 estimates a camera posture (in other words, a camera posture that matches the collation result) that explains the relationship between the collation results (a match between the three-dimensional coordinates and the two-dimensional coordinates) acquired from the collation unit 200.
Details of the camera posture estimation unit 300 will be described later.

  With the above configuration, the camera posture estimation device 20 estimates the camera posture from the camera image. Next, the database unit 100 and the camera posture estimation unit 300 will be described.

(Database unit 100)
The three-dimensional coordinate database 130 corresponds to the two-dimensional pixel coordinates (coordinates indicating the feature points of the three-dimensional object in the camera image) stored in the feature information database 120 (three-dimensional coordinates of the three-dimensional object). The processing unit 110 uses the geometric information of the three-dimensional object to calculate the three-dimensional coordinates of the three-dimensional object corresponding to the two-dimensional pixel coordinates of the feature point of the three-dimensional object. Acquired (three-dimensional coordinate acquisition means) and stored in the three-dimensional coordinate database 130. The geometric information of the three-dimensional object is an equation of the three-dimensional object. Hereinafter, the equation of the three-dimensional object is also referred to as a “mathematical model”.

In other words, the processing unit 110 detects feature points of the three-dimensional object from the camera image (feature point detection means), stores the two-dimensional pixel coordinates of the feature points in the feature information database 120, and uses the mathematical model to store the feature points. Three-dimensional coordinates corresponding to the two-dimensional pixel coordinates of the point are acquired (three-dimensional coordinate acquisition means) and stored in the three-dimensional coordinate database 130.
In other words, the processing unit 110 constructs (generates) a database (three-dimensional coordinate database 130, feature information database 120) referred to in feature point matching as described above.

  The perspective projection in the pinhole camera model is expressed by the following equation (1).

  FIG. 3 is an explanatory diagram for explaining the mathematical model. The mathematical model of the spherical solid object is represented by the following formula (2). In other words, the following equation (2) is an equation of a spherical solid object. In the following formula (2), p is a point on the sphere, c is the center of the sphere, and r is the radius of the sphere (see FIG. 3A).

  A straight line passing through the focal point and the feature point in the image plane is expressed by, for example, the following formula (3). In other words, the following equation (3) is the above linear equation. In the following formula (3), q is a point on a straight line, s is a focal point, and v is a direction vector of the straight line.

The three-dimensional coordinates of the feature points of the spherical solid object are obtained as the coordinates of the intersection of the mathematical model of the above formula (2) and the straight line of the above formula (3) (the straight line passing through the focus and the feature point in the image plane). be able to. In other words, the coordinates of the intersection can be acquired from the equation of the spherical solid object and the equation of the straight line passing through the focal point and the feature point in the image plane.
Specifically, the processing unit 110 represents the center c, which is a parameter of the mathematical model of the spherical solid object represented by the above equation (2) (that is, the parameter of the equation of the spherical solid object), and the above equation (3). A focal point s and a direction vector v, which are parameters of a straight line to be performed (that is, parameters of a linear equation passing through a focal point and a feature point in an image plane), p = q, and p (or q) is a coordinate of an intersection And obtained as the three-dimensional coordinates of the feature points of the spherical three-dimensional object (three-dimensional coordinate acquisition means).

  The processing unit 110 calculates mathematical model parameters and straight line parameters using the internal parameters of the camera and the camera image of the three-dimensional object (parameter calculation means). In the case of a spherical solid object, the processing unit 110 uses the parameters (center c) of the mathematical model of the spherical solid object of the above formula (2) and the straight line parameters (focal point s, direction vector v) of the above formula (3). calculate.

  Specifically, when the camera posture W = [R, t] (the origin is the center c of the sphere model) with respect to the spherical three-dimensional object at the time of imaging the spherical three-dimensional object, the following equation (4) Since (7) is established, the processing unit 110 can calculate the above parameters (center c, focus s, direction vector v).

  In the case where the processing unit 110 previously calculates the three-dimensional coordinates corresponding to the two-dimensional pixel coordinates of the feature points offline and registers them in the database, the camera posture can be ideally acquired from the camera image.

  When the camera posture is tracked for each frame, the database can be updated by acquiring three-dimensional coordinates using the feature point of the previous frame and the estimated value of the camera posture.

  When the camera posture is not given, the pixel coordinates of the three-dimensional object in the camera image (in the case of a sphere, the center of the ellipse (x0, y0) and the major axis 2a) are obtained, thereby obtaining the parameters of the mathematical model. , And straight line parameters can be obtained.

  For example, with c as the origin, v ′ in the above equation (7) is obtained from the focal length of the camera and the two-dimensional pixel coordinates of the feature point. When the three-dimensional object is rotated by θ, φ, and ψ with respect to the X axis, the Y axis, and the Z axis, the rotation matrix R is expressed by the following expression (8). And the direction vector v is obtained from the above equation (7).

  Next, a vector d between the center and the focal point of the sphere and its length are calculated from the position of the sphere in the camera image, the major axis 2a of the projected sphere (ellipse), the rotation matrix R, and the radius r of the sphere, and the focal point s. Ask for.

  Further, when a plurality of intersection points are calculated, the processing unit 110 acquires one intersection point close to the camera focus among the plurality of intersection points as a three-dimensional coordinate (three-dimensional coordinate acquisition unit). Thereby, the correct three-dimensional coordinate of the feature point of the three-dimensional object can be associated with the two-dimensional pixel coordinate of the feature point of the three-dimensional object.

  Further, the processing unit 110 may delete the feature point from the feature information database 120 when the intersection is not calculated when using a straight line passing through one feature point (deleting unit). That is, when no intersection point is obtained, it means that there is no feature point in the three-dimensional object, and therefore it may be deleted from the feature information database 120.

  In addition, when the equation for obtaining the intersection is a quadratic equation, the processing unit 110 determines whether the feature point is a point on the three-dimensional object using a discriminant (see the following equation (9)). (Three-dimensional coordinate acquisition means).

  Further, the processing unit 110 may not acquire the position on the curved surface of the three-dimensional object as the three-dimensional coordinates of the feature point at the periphery as viewed from the focal point of the camera (may be ignored). (3D coordinate acquisition means).

  In general, since a three-dimensional coordinate value having a large angle at which a normal line of a curved surface and a straight line have a large accuracy deteriorates, for example, in the case of a spherical three-dimensional object, a three-dimensional coordinate corresponding to a two-dimensional pixel coordinate value around a contour away from the center is an error. Becomes larger. Therefore, the processing unit 110 may improve the final camera posture estimation accuracy by not storing the three-dimensional coordinates that increase the error in the three-dimensional coordinate database 130.

  Specifically, the processing unit 110 masks a three-dimensional coordinate value having a large angle between the normal line and the straight line of the curved surface of the three-dimensional object (the angle at which the normal line of the curved surface intersects the straight line) (that is, as an intersection point). It is not calculated) and is not acquired as the three-dimensional coordinates of the feature points.

  Further, when the coordinate calculation formula for the intersection is a quadratic equation, the processing unit 110 may change the threshold value of the discriminant and simply realize the above-described masking.

The following equation (9) is an example of a discriminant for a sphere.

In the formula (9), the threshold value of D and αr ^2, 0 <α <excluded as a threshold below the 3-dimensional coordinate values close to the contour of the sphere by changing 1. That is, the processing unit 110 changes the threshold value of a determination formula (in the case of a sphere, the above formula (9)) for determining whether or not an intersection exists on the three-dimensional object, thereby changing the value on the curved surface of the three-dimensional object. The position of the peripheral edge when viewed from the focal point of the camera is not acquired as a three-dimensional coordinate (three-dimensional coordinate acquisition means).

  The processing unit 110 in the camera posture estimation device 20 registers the three-dimensional coordinate value acquired in this way in the three-dimensional coordinate database 130.

  Although an example of a spherical solid object has been described as an example of a three-dimensional object whose surface is partly or entirely curved, each mathematical expression can also be applied to other three-dimensional objects whose surface is partly or entirely curved. Using the model, the three-dimensional coordinates of each three-dimensional object can be acquired in the same manner as the spherical three-dimensional object.

  For example, in the case of a cylindrical solid object, the mathematical model of the side surface is represented by the following expression (10), and three-dimensional coordinates can be obtained using the same method as that of the spherical solid object (FIG. 3B). reference).

  When the axis of the cylindrical solid object is parallel to the y-axis, A in the above formula (10) is represented by the following formula (11).

  Further, for example, in the case of a conical solid object, the mathematical model of the side surface is represented by the following expression (12), and three-dimensional coordinates can be obtained using the same method as that of the spherical solid object.

  When the axis of the conical solid object is parallel to the y-axis, A in the above formula (12) is represented by the following formula (13).

  As described above, the database unit 100 uses each mathematical model to accurately represent all three-dimensional objects that can be expressed by a mathematical model, including solid objects whose surface is partly or entirely curved. Three-dimensional coordinates can be acquired. In particular, in the case of a three-dimensional object composed of a geometric shape such as a sphere, cylinder, cone, or torus (a three-dimensional object that can be expressed by a simple mathematical model), the processing load is small, and it is easy and precise. Three-dimensional coordinates can be acquired.

  The database unit 100 may acquire the three-dimensional coordinates using a three-dimensional polygon model in addition to the mathematical model. For example, the database unit 100 may acquire a three-dimensional coordinate by using a three-dimensional polygon model for a plane portion of a three-dimensional object and using a mathematical model for a curved surface portion. Combined with a 3D coordinate acquisition method using an existing 3D polygon model, it is possible to estimate the camera posture efficiently even for 3D objects with complicated shapes including both curved and flat surfaces. Become.

  In other words, when a 3D polygon model is used, it is necessary to calculate which polygon the 2D pixel coordinates correspond to, but by using a mathematical model for a portion of a curved surface that must be expressed by many polygons, Since the above-described calculation process is not necessary for the curved surface portion, the load can be reduced.

(Camera posture estimation unit 300)
The camera posture estimation unit 300 estimates a camera posture that explains the relationship between the three-dimensional coordinates and the two-dimensional coordinates from the match between the three-dimensional coordinates and the two-dimensional coordinates input from the collation unit 200.
Specifically, the camera posture estimation unit 300 estimates the camera posture W using the match between the three-dimensional coordinate X and the two-dimensional pixel coordinate x input from the collation unit 200 and the internal parameter A of the camera. For example, the camera posture estimation unit 300 estimates W that minimizes the reprojection error e in the following equation (14) by a least square algorithm.

  As described above, according to the camera posture estimation apparatus 20, even when the three-dimensional object includes a curved surface, the estimation accuracy of the camera posture can be improved and the processing load can be reduced.

  As shown in FIG. 2, the camera posture estimation device 20 includes a database unit 100, a collation unit 200, and a camera posture estimation unit 300, but each unit (database unit 100, collation unit 200, camera posture estimation unit 300). ) May be an independent device (ie, a separate unit). For example, the database unit 100 may be a separate body independent of the camera posture estimation device 20.

  Note that the processing unit 110 in the database unit 100 has processing functions (feature point detection means, three-dimensional coordinate acquisition means, parameter calculation means, and deletion means) as described above, and the three-dimensional object captured by the camera. The three-dimensional coordinates of the three-dimensional object used when estimating the camera posture of the camera from the camera image are acquired. Hereinafter, the device name of the database unit 100 as a separate body is referred to as a three-dimensional coordinate acquisition device.

  FIG. 4 is an example of a functional block diagram of the three-dimensional coordinate acquisition apparatus 100 according to the embodiment of the present invention. The three-dimensional coordinate acquisition apparatus 100 acquires the three-dimensional coordinates of the three-dimensional object used when estimating the camera posture. As illustrated in FIG. 4, the three-dimensional coordinate acquisition apparatus 100 includes a processing unit 110, a feature information database 120, a three-dimensional coordinate database 130, and a mathematical model storage unit 140. The processing unit 110 includes a feature point detection unit 111, a parameter calculation unit 112, a three-dimensional coordinate acquisition unit 113, a deletion unit 114, and a data transmission unit 119. Since the feature information database 120 and the three-dimensional coordinate database 130 have already been described, description thereof will be omitted.

  The mathematical model storage unit 140 stores a mathematical model of a three-dimensional object (that is, an equation of the three-dimensional object). In this embodiment, the mathematical model is acquired and stored in advance. However, the mathematical model may be acquired every time the three-dimensional coordinates are acquired, and the mathematical model is acquired every time the three-dimensional coordinates are acquired. In some cases, the three-dimensional coordinate acquisition apparatus 100 does not need to include the mathematical formula model storage unit 140.

  The feature point detection unit 111 detects feature points of the three-dimensional object in the camera image. Specifically, the feature point detection unit 111 detects a feature point in the image plane from the camera image. The parameter calculation unit 112 uses the internal parameters of the camera and the camera image of the three-dimensional object, and calculates the parameters of the straight line passing through the focal point of the camera and the feature point in the image plane, and the parameters of the mathematical model of the three-dimensional object. calculate.

  The three-dimensional coordinate acquisition unit 113 calculates the intersection of the straight line and the three-dimensional object from the above-described straight line equation and the mathematical model of the three-dimensional object (three-dimensional object equation), and uses it as the three-dimensional coordinates of the feature point. get. Note that, when a plurality of intersection points are calculated, the three-dimensional coordinate acquisition unit 113 acquires one intersection point close to the camera focus among the plurality of intersection points as a three-dimensional coordinate. In addition, when the equation for acquiring the intersection is a quadratic equation, the three-dimensional coordinate acquisition unit 113 determines whether or not the intersection exists on the three-dimensional object (for example, in the case of a sphere, the above equation (9 The three-dimensional coordinate acquisition unit 113 uses the mathematical model for the curved surface portion of the three-dimensional object. On the other hand, for the plane portion of the three-dimensional object, the three-dimensional coordinates of the feature points may be acquired using a three-dimensional polygon model.

  The three-dimensional coordinate acquisition unit 113 may not acquire the position of the peripheral edge as viewed from the focal point of the camera as the three-dimensional coordinates of the feature point on the curved surface of the three-dimensional object. Specifically, the three-dimensional coordinate acquisition unit 113 may not acquire the above-described position as the three-dimensional coordinate by changing the threshold value of the above-described determination formula.

  When the intersection is not calculated from a straight line passing through one feature point, the deletion unit 114 deletes the feature point from the feature information database 120.

  The transmission / reception unit 119 uses the information in the database (the feature information database 120, the three-dimensional coordinate database 130) constructed as described above, for example, in response to a request, as a camera posture estimation device (camera posture estimation without the database unit 100). Device 20).

  As described above, the three-dimensional coordinate acquisition apparatus 100 uses each mathematical model for any three-dimensional object that can be represented by a mathematical model and includes a solid object whose surface is partially or entirely curved. Accurate three-dimensional coordinates can be acquired. In particular, in the case of a three-dimensional object configured by a geometric shape such as a sphere, a cylinder, a cone, or a torus, the processing load is small, and precise three-dimensional coordinates can be easily acquired.

  Note that a program for executing each process of the camera posture estimation apparatus 20 according to the embodiment of the present invention is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read by a computer system. By executing, the processing related to the camera posture estimation apparatus 20 according to the embodiment of the present invention may be performed. Here, the “computer system” may include an OS and hardware such as peripheral devices. Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used. The “computer-readable recording medium” means a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, etc. This is a storage device.

  Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time. The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 ... AR system 10 ... Imaging device 20 ... Camera posture estimation apparatus 30 ... Display apparatus 90 ... Additional information database 100 ... Database part (three-dimensional coordinate acquisition apparatus) 110 ... Processing part 111 ... Feature point detection part 112 ... Parameter calculation part 113 ... 3D coordinate acquisition unit 114 ... Deletion unit 119 ... Data transmission unit 120 ... Feature information database 130 ... 3D coordinate database 200 ... Collation unit 210 ... Feature point detection unit 220 ... Feature quantity description unit 230 ... Feature quantity collation unit 300 ... Camera posture estimation unit

Claims (12)

A three-dimensional coordinate acquisition apparatus that acquires the three-dimensional coordinates of the three-dimensional object used when estimating the camera posture of the three-dimensional object from a captured image of the three-dimensional object captured by the camera,
Feature point detection means for detecting a feature point in the image plane from the captured image;
The intersection of the straight line and the three-dimensional object is calculated from the equation of a straight line passing through the focal point of the camera and the feature point in the image plane and the equation of the three-dimensional object, and the three-dimensional coordinates of the feature point A three-dimensional coordinate acquisition device, comprising: 2. The apparatus according to claim 1, further comprising: a parameter calculating unit that calculates a parameter of the straight line equation and a parameter of the three-dimensional object equation using an internal parameter of the camera and a captured image of the three-dimensional object. The three-dimensional coordinate acquisition apparatus described. The three-dimensional coordinate acquisition means includes
3. The intersection according to claim 1, wherein when a plurality of intersection points are calculated, one intersection point close to the focal point of the camera among the plurality of intersection points is acquired as the three-dimensional coordinates. 3D coordinate acquisition device. A feature information database for storing feature points of the three-dimensional object;
4. The apparatus according to claim 1, further comprising: a deletion unit that deletes the feature point from the feature information database when the intersection is not calculated from a straight line passing through the one feature point. The three-dimensional coordinate acquisition apparatus according to item 1. The three-dimensional coordinate acquisition means includes
When the equation for acquiring the intersection is a quadratic equation, it is determined whether or not the feature point is a point on the three-dimensional object using a discriminant. The three-dimensional coordinate acquisition apparatus according to any one of the above. The three-dimensional object is
6. The three-dimensional coordinate acquisition apparatus according to claim 1, wherein a part or all of the surface is a curved surface. The three-dimensional object is
A part of the surface is curved,
The three-dimensional coordinate acquisition means includes
For the curved surface portion of the three-dimensional object, the three-dimensional coordinates of the feature points are obtained using the equation of the three-dimensional object, while the three-dimensional polygon model is used for the planar portion of the three-dimensional object. The three-dimensional coordinate acquisition apparatus according to any one of claims 1 to 5, wherein the three-dimensional coordinates of the feature point are acquired. The three-dimensional coordinate acquisition means includes
The position on the curved surface of the three-dimensional object and the position of the peripheral edge when viewed from the focal point of the camera is not acquired as the three-dimensional coordinates of the feature points. The three-dimensional coordinate acquisition apparatus described in 1. The three-dimensional coordinate acquisition means includes
By changing a threshold value of a determination formula for determining whether or not the intersection exists on the three-dimensional object, the position on the curved surface of the three-dimensional object is a peripheral edge when viewed from the focal point of the camera. The three-dimensional coordinate acquisition apparatus according to claim 8, wherein the position is not acquired as three-dimensional coordinates. A camera posture estimation device that estimates a camera posture of a three-dimensional object imaged by a camera from a captured image of the camera,
A database unit that stores the two-dimensional pixel coordinates of the feature points of the three-dimensional object in the captured image captured in advance in association with the three-dimensional coordinates of the feature points of the three-dimensional object acquired in advance;
Extracting a feature point of the three-dimensional object from the captured image captured this time, and collating with the two-dimensional pixel coordinates in the database unit;
A camera posture estimation unit that estimates a camera posture based on the collation result,
The database part
The intersection of the straight line and the three-dimensional object calculated from the focal point of the camera and the equation of a straight line passing through the feature point in the image plane detected from the captured image and the equation of the three-dimensional object Is stored as a three-dimensional coordinate of the feature point of the three-dimensional object. The three-dimensional object is
A part of the surface is curved,
The database part
The three-dimensional coordinates calculated using the equation of the three-dimensional object for the curved surface portion of the three-dimensional object are stored, and the three-dimensional calculated using the three-dimensional polygon model for the plane portion of the three-dimensional object The camera posture estimation apparatus according to claim 10, wherein coordinates are stored. In a computer of a three-dimensional coordinate acquisition device that acquires the three-dimensional coordinates of the three-dimensional object used when estimating the camera posture of the three-dimensional object from a captured image of the three-dimensional object captured by the camera,
A feature point detection step of detecting a feature point in the image plane from the captured image;
The intersection of the straight line and the three-dimensional object is calculated from the equation of a straight line passing through the focal point of the camera and the feature point in the image plane and the equation of the three-dimensional object, and the three-dimensional coordinates of the feature point And a three-dimensional coordinate acquisition step acquired as follows.

Images