JP2000041173A - Deciding method for visual point position attitude, camera device and visual point position sensor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To uniquely decide a camera parameter to show the position attitude of a camera by acquiring the images of three marks existing within a prescribed plane through the camera, acquiring the image coordinates of those marks and then calculating and outputting a parameter to decide the camera position attitude based on the depth information on three marks and the image coordinates acquired from these depth information. SOLUTION: A camera parameter deciding device consists of an estimation module, a coordinate detection module and a parameter estimation module. The image coordinates of three land marks are acquired together with the depth information up to the land marks and a matrix U of C'=U'.W'-1 is acquired. Thus, the camera position is decided in AR (augmented reality). The camera parameter matrices U' and W' are shown in expressions I and II. In these expressions, the image coordinates of three marks are shown in (x1, y1), (x2, y2) and (x3, y3) with the depth information on the marks shown in z1, z2 and z3 and the world coordinate value of the marks shown in (XW1 YW1, O), (XW2, YW2, O) and (XW3, YW3 and O) respectively.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for determining a viewpoint position and orientation, a camera device, and a position and orientation sensor, and more particularly to an improvement in a method for determining a viewpoint position of a camera by three marks.

[0002]

2. Description of the Related Art In recent years, researches on mixed reality (MR) for the purpose of integrating a real space and a virtual space have been actively conducted. Among them, a technique for superimposing and displaying information of a virtual space on a real space is called “Augmented Reality” (AR: Augmented Reality).

[0003] AR realizing means can be roughly classified into two types. One method is to use a transmission-type HMD (Head-Mounted Display) to superimpose an image of a virtual object on a real world scene seen over a display surface, which is called an “optical see-through method”. The other is a method of superimposing and drawing a virtual object on an image shot by a video camera, which is called a “video see-through method”. In either method, 2
In order to realize a natural fusion of two spaces, it is necessary to address factors such as “alignment”, “coincidence of image quality”, and “synthesis of three-dimensional space” (expression of context and collision). Above all, “alignment” is the most basic and important factor for realizing AR.

[0004] Positioning in AR basically means measuring and estimating parameters such as the viewpoint (in the case of an optical see-through system) of a viewer and the position and orientation of a camera (in the case of a video see-through system). For this purpose, two methods are used. One is a "sensor-based" method using a three-dimensional position and orientation sensor such as a magnetic sensor or an ultrasonic sensor, and the other is an "image-based" method of alignment mainly used in video see-through AR. .

[0005]

Although the sensor-based alignment method is excellent in operation stability,
In many cases, accuracy is insufficient for use in AR. On the other hand, the image-based positioning method enables highly accurate positioning by directly using the actual image information to be merged for positioning. It is conceivable to apply various camera calibration methods that have been studied in the field of computer vision. But AR
In such a case, it is necessary to implement various kinds of processing under the restriction of real-time processing, and such an algorithm has a problem that an error easily occurs in landmark extraction / identification processing and operation becomes unstable.

[0006] A conventional method for positioning will be described. In the following, for simplification of description, it is assumed that the projection from the camera coordinate system to the image coordinate system is performed based on an ideal perspective projection model. That is,
It is assumed that factors such as image distortion, center shift, and aspect ratio have been measured in advance and have been removed at the stage of image coordinate extraction.

First, a basic form of camera parameter estimation will be described. It is assumed that a landmark Q _i (world coordinates Q _Wi = (X _Wi , Y _Wi , Z _Wi , 1)) in a three-dimensional space is imaged by a camera at image coordinates q _i = (x _i , y _i ). . Assuming a perspective projection for this camera, this projection is given by 34 transformation matrices C,

[0008]

(Equation 4) … (1) Where h _i is a parameter. Expanding equation (1),

[0009]

(Equation 5) … (2) is obtained. The parameter h _i can be eliminated from the third expression of (2). In addition, the world coordinates of the landmark (X _Wi ,
Y _Wi , Z _Wi ) are known, and the observation coordinates (x _i , y _i ) have been obtained on the image for that landmark, so 1
A pair of world coordinate values and observation coordinate values for a point landmark give the first and second two expressions of the expression (2).

Since the matrix C is 3 × 4, it has these 12 unknowns, ie, matrix elements. Since one landmark gives two equations, in order to determine the matrix C, six (i = 1, 2,... 6) or more (known) landmarks not on the same plane It just needs to be observed. How to find the matrix C is a problem of camera parameter estimation, that is, positioning.

It has been proposed to estimate camera parameters using depth information. Hereinafter, a method of estimating camera parameters using depth information will be described. The parameter h _i in equation (1) is proportional to the depth value Z _Ci of the landmark Q _i in the camera coordinate system, and using a certain constant k,

[0012]

(Equation 6) ... (3). Further, the value of k can be arbitrarily selected as long as the value satisfies the proportional relationship. Now, as a measure of depth for landmark Q _i ,

[0013]

(Equation 7) It is assumed that a value z _i that satisfies (4) is obtained. in this case,
By substituting z _i for h _i in equation (1), the following three equations are obtained for one landmark.

[0014]

(Equation 8) … (5) Here, the world coordinate values of four (or more) landmarks that are not on the same plane are (X _W1 , Y _W1 , Z _W1 ) (X _W2 , Y _W2 , Z _W2 ) (X _W3 , Y _W3 , Z _W3 ) (X _W4 , Y _W4 , Z _W4 )… (6), and in the image coordinate system, the coordinate values are (x ₁ , y ₁ , z ₁ ) (x ₂ , y ₂ , z ₂ ) (x ₃ , y ₃ , z ₃ ) (x ₄ , y ₄ , z ₃ )… (7)

[0015]

(Equation 9) … (8)

[0016]

(Equation 10) .. (9), Equation (5) can be expressed as U = CW... (10), and the matrix C is obtained by the following equation.

C = UW ⁻¹ (11) Here, the matrix W ⁻¹ is an inverse matrix of the matrix W represented by a set of known landmark world coordinates, and can be calculated in advance. . Therefore, the problem of estimating the camera parameters (C = {a _ij }) is conventionally determined by how the matrix U, that is, the image coordinates of the four landmarks
The result is to get (x _i , y _i ) and its depth measure z _i .

In order to obtain the landmark depth scale z _i , for example, Mellor (JP Mellor: Realtime camera c
alibration for enhanced reality visualization, Pro
c. CVRMed 95, pp.471-475, 1995.) proposed a method using information on the apparent size of landmarks. This method of Mellor makes use of the fact that the apparent size s _i of the landmark is inversely proportional to the distance from the viewpoint to the landmark.The reciprocal 1 / s _i of s _i obtained in this way is expressed by z in Equation (6). _By using it as _i , alignment using four landmarks is performed.

As described above, even when the landmark depth information is used for estimating camera parameters, four landmarks are required as described above. Camera parameters can also be estimated by imposing constraints on camera placement. Nakazawa et al. (Nakazawa, Nakano, Komatsu, Saito: Video Synthesizing System Based on Feature Points in Images and CG Images, Journal of the Institute of Image Information and Television Engineers, Vol.51, N
o.7, pp.1086-1095, 1997.) propose a method for estimating camera parameters using a plane of Z = 0. That is, based on the assumption that all landmarks are arranged on the plane of Z = 0 in the world coordinate system, the constraint that four landmarks are on the same plane is imposed, and the camera parameters are estimated. That is. Okuma et al. (Okuma, Kiyokawa, Takemura, Yokoya: Real-time estimation of camera parameters from real images for video see-through augmented reality, IEICE Technical Report, PRMU97-113, 1997.) By making it known, the alignment method that further simplified Nakazawa's method was realized.

[0020]

However, all of the above three prior art methods require four landmarks, and have a problem in real-time processing. Certainly, there are proposals using three landmarks (eg, Fisher), but this requires solving a complicated nonlinear equation, and there are multiple solutions, and it is difficult to obtain a unique solution. It is not an exaggeration to say that the camera parameters could not be determined.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned drawbacks of the prior art, and an object of the present invention is to provide a viewpoint position at which a camera parameter representing a camera position and orientation can be uniquely determined from three landmarks. It is an object of the present invention to propose a method for determining the position, a camera device, and a method for detecting the position and orientation of a camera.

[0022]

According to a first aspect of the present invention, there is provided a method for acquiring an image of three marks placed in a predetermined plane using a camera. Acquire the image coordinates of each of the three marks in the image,
The depth information of the three marks is obtained, and parameters for determining the position and orientation of the camera are calculated and output based on the image coordinates and the depth information obtained for the marks.

The present invention is applicable to various camera and sensor combinations. According to a preferred aspect of the present invention, the depth information of the three marks is detected based on a monocular camera and an output of a position and orientation sensor provided in the camera. . According to a third aspect of the present invention, the depth information of the three marks is obtained by a stereo camera and an output of a three-dimensional position and orientation sensor.

The mark depth information does not always need to be completely obtained. If the purpose is simply to display up to three beads, when the depth information of at least one mark is obtained, the depth information of the virtual mark is set as in claim 4, which is a preferred aspect of the present invention. I do. According to claim 5, which is a preferred aspect of the present invention, the three marks are selected as marks on the Z = 0 plane in world coordinates that are not on the same straight line.

According to claim 6 which is a preferable aspect of the present invention, the three marks are not on the same straight line,
When selecting a mark on an arbitrary plane other than Z = 0 in world coordinates, a transformation matrix from the arbitrary plane other than Z = 0 to the Z = 0 plane is obtained, and the landmark world transformed by this transformation matrix is obtained. It is characterized in that parameters of the camera are determined using the coordinates.

The above object is also achieved by a camera device according to claim 6. For example, a camera device that outputs position and orientation data together with an image includes a camera that captures images of three marks placed in a predetermined plane, and a unit that calculates coordinate values of the images of the marks captured by the camera. Means for obtaining depth information of the three marks, and means for calculating parameters for determining the position and orientation of the camera based on image coordinates and depth information obtained for the marks. Features.

The above object can also be achieved by a head position / posture sensor. The position and orientation sensor in this case is attached to the vicinity of the head and captures images of three marks placed in a predetermined plane, and an image of the mark captured by the camera. Means for obtaining coordinate values; means for obtaining depth information of the three marks; and calculating parameters for determining the position and orientation of the camera based on image coordinates and depth information obtained for the marks. Output means.

The object of the present invention is to set the image coordinates of the three marks to (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₄ ) and the depth information to z ₁ , z ₂ , z ₃ , the world coordinate values are (X _W1 , Y _W1 , 0),
If (X _W2 , Y _W2 , 0) and (X _W3 , Y _W3 , 0), the camera parameter matrix is

[0029]

[Equation 11]

[0030]

(Equation 12) Then

[0031]

(Equation 13) From the matrix C given by

[0032]

BRIEF DESCRIPTION OF THE DRAWINGS FIG.
An embodiment of the present invention will be described. The inventors of the present invention
The method of the embodiment is positioned as an extension of the method of Nakazawa et al. The method according to the present embodiment enables alignment using three landmarks, that is, enables estimation of camera parameters.

First, Nakazawa et al.'S method is systematized from the viewpoint of the inventor of the present invention. <Estimation Using Z = 0 Plane> Assuming that the Z coordinates of landmarks in the world coordinate system are all 0, a coordinate transformation matrix representing a projection relationship when observing the landmarks on an image is:
The third column of matrix C in equation (1) (the component related to the Z coordinate) was omitted.
It can be represented only by a 3x3 matrix, and it is represented as C. In this case, the projection of the landmark from the world coordinate system to the image coordinate system is simplified as compared with the equation (1).
It can be described as follows.

[0034]

[Equation 14] … (12) By expanding this equation,

[0035]

(Equation 15) … (13) is obtained. Substituting the third equation of (13) into the first and second equations to eliminate the parameter h _i , one landmark (X _Wi , Y
Two equations are obtained for _Wi ). Here, if a ₃₄ = 1, the unknowns of the matrix C are a ₁₁ , a ₁₂ , a ₁₄ , a ₂₁ , a
_21, from _{a 22, a 24, a 3} 1, be eight of a _32, four or more landmarks _{(X Wi, Y Wi) (} i = 1, 2, 3, 4) to observe the Then, the matrix C can be obtained.

The camera parameters C can be estimated by obtaining the third column components (a ₁₃ , a ₂₃ , a ₃₃ ) of the matrix C from the matrix C thus obtained. The procedure for obtaining the matrix C from the matrix C will be described in more detail below. Generally, a matrix C (3x4) representing a coordinate transformation from the world coordinate system to the image coordinate system is a perspective transformation matrix P (3x4 matrix) from the camera coordinate system to the image coordinate system, where f is the focal length of the camera.
And a coordinate transformation matrix M (4
x4 matrix) can be described as follows:

[0037]

(Equation 16) (14) On the other hand, the above-described matrix C can be similarly described by the following equation using the matrix P and the matrix M (4x3) in which the third column of the matrix M is omitted.

[0038]

[Equation 17] … (15) That is, since each element of the matrix C is obtained as described above,
If the focal length f of the camera is known, each element of the matrix M 'can be easily obtained from the matrix C'. Also, the matrix M
The third column of represents the z-axis of the camera coordinate system, which is
The first column (x-axis) and the second column (y
(Axis) can be obtained as a vector orthogonal to the two vectors represented by (axis). Therefore, the matrix M can be estimated from the matrix M, and the matrix M representing the camera parameters can be obtained by substituting the matrix M thus obtained into the equation (14). That is, the camera parameters C can be obtained by constraining the four landmarks on the Z = 0 plane.

<Estimation of Camera Parameter Matrix C Using Three Points> As shown in FIG. 1, three landmarks (Q ₁ , Q ₂ ,
The projection from the world coordinate system to the image coordinate system in Q ₃ ) can be described as in the following equation, as in equation (12).

[0040]

(Equation 18) (16) The parameter h _i in this equation is proportional to the depth value Z _{Ci of the} landmark Q _i (= Q ₁ , Q ₂ , Q ₃ ) in the camera coordinate system, and using a constant k,

[0041]

[Equation 19] ... (17) Further, the value of k can be arbitrarily selected as long as the value satisfies the proportional relationship. Now, as a measure of depth for landmark Q _i ,

[0042]

(Equation 20) It is assumed that a value z _i (z ₁ , z ₂ , z ₃ ) that satisfies (18) is obtained. In this case, the following three equations are obtained for one landmark by substituting z _i for h _i in equation (16).

[0043]

(Equation 21) … (19) When three or more landmarks that are not on the same line are observed,

[0044]

(Equation 22) … (20)

[0045]

(Equation 23) … (21), the relationship of equation (16) is

[0046]

(Equation 24) ... (22), the matrix C is a 3x3 matrix in which the third column (component relating to the Z coordinate) of the matrix C is omitted.
Is

[0047]

(Equation 25) … (23). Then, the camera parameter C can be obtained from the obtained C in the same manner as in the above-described method. That is, if the perspective transformation matrix from the camera coordinate system to the image coordinate system is P (3x4 matrix) and the coordinate transformation matrix M from the world coordinate system to the camera coordinate system is (4x4 matrix),

[0048]

(Equation 26) …(twenty four) … (25) and the matrix C (3x4)

[0049]

[Equation 27] … (26), and the matrix C is also

[0050]

[Equation 28] … (27) If the focal length f of the camera is known, the elements of the matrix M can be easily obtained from the matrix C as described above. And the third column of the matrix M is the matrix
It can be obtained as a vector orthogonal to the two vectors represented by the first column (x-axis) and the second column (y-axis) of M (that is, of the matrix M). Therefore, the matrix M can be estimated from the matrix M, and the matrix M representing the camera parameters can be obtained by substituting the matrix M thus obtained into the equation (26). That is, the camera parameters C could be obtained by constraining the three landmarks on the Z = 0 plane.

That is, the matrix W in equation (23)^-1Has three known
A set of landmark world coordinates, calculated in advance
be able to. Therefore, the camera parameter estimation question
The title is the matrix U, that is, the image coordinates of the three landmarks
And its depth scale z_iComes down to the problem of seeking. What
Contact, 3 landmarks Q_iAlways exists on one plane
However, as shown in FIG. 2, the plane is in the world coordinate system.
It may not be the Z = 0 plane. Even in such a case,
3 points Q_iZ = 0 flat from the plane where the landmarks are located
The coordinate transformation matrix N (4x4) to the surface always exists, and
You can ask. Therefore, such a coordinate transformation
World coordinates P of each landmark transformed by matrix N^N
_WiIs P^N _Wi= NQ_i (28), and must satisfy the expressions (16) to (27).
That is, the transformed coordinates P^N _WiEquations (16) through (27)
Let C be the camera parameter matrix obtained^(N)Then C = C^(N)By setting N ... (29), the correct camera parameter C is derived.
You.

[0052]

FIG. 3 shows the structure of a camera parameter determining apparatus according to an embodiment. As shown in FIG.
Depth estimation module 100 and coordinate detection module 20
0 and a parameter estimation module 300. As described above, the essence of the present invention is to obtain image coordinates of three landmarks and depth information to the landmarks,
This is to determine the camera position in AR by obtaining the matrix U in equation (23). In the apparatus of FIG. 3, for example, a three-dimensional position and orientation sensor (magnetic sensor) and one or more cameras can be provided to obtain depth information. Therefore, whether or not a three-dimensional sensor is connected to the present apparatus, or how many cameras are connected, or a target landmark is imaged to such an extent that depth information can be obtained. The operation of the present device differs depending on whether the device is present or not. Hereinafter, the operation of the present device will be described according to various modes of the input device.

The determining device of the embodiment can be realized by software or hardware.
The configuration in the figure is merely an example. <Positioning by Stereo> First Embodiment The first embodiment is a method of determining camera parameters when the apparatus shown in FIG. 3 has a stereo camera for inputting an image of a landmark.

In order to present a parallax image to the left and right eyes of the observer in the AR of the video see-through system, it is necessary to mount a stereo camera on the HMD and perform positioning with respect to the image of each camera. In the first embodiment, information obtained from these two cameras is used as a clue for positioning. When positioning is performed using a stereo camera, distance information z _i to the landmark can be obtained by determining the correspondence between the landmarks between images obtained by the two cameras.

For simplicity, the optical axes of the two stereo cameras are normalized so as to have an epipolar line that is parallel to each other, perpendicular to the base line, and parallel to the x-axis of the image coordinate system. Assume that The landmark Q _i is defined as a point q ^R _i = (x ^R _i , y ^R _i ) on the right image and a point q ^L _i =
(x ^L _i , y ^L _i ) (where y ^R _i = y ^L
_i ). At this time, as shown in FIG. 4, the parallax d between the corresponding points
_i (= x ^L _i -x ^R _i ) is inversely proportional to the depth value Z _{Ci of} Q _i .

[0056]

(Equation 29) … (30) Therefore, by obtaining the corresponding points of the three landmarks, the matrix U is

[0057]

[Equation 30] By placing ... (31), the parameters of the right camera, i.e., it is possible to obtain the coordinate transformation matrix C ^R. further,

[0058]

(Equation 31) (32), the parameters of the left camera, that is, the coordinate transformation matrix C ^L are

[0059]

(Equation 32) … (33) is easily obtained. Here, a ^R _jk represents each element of the coordinate transformation matrix C ^R of the right camera. Incidentally, even when the optical axes of the stereo cameras are not parallel to each other and are congested, if the perspective transformation matrix P (3x4) is known and the relative position between the cameras is given, the landmark Q The depth value Z ^R _Ci of _{i in} the right camera coordinate system can be easily obtained from the correspondence on the stereo image. Therefore, the matrix U is

[0060]

[Equation 33] .. (34), the coordinate transformation matrix C ^R of the right camera is obtained. <Alignment with Monocular Image and Sensor> Second Embodiment In the first embodiment, the present invention is applied to a system in which the apparatus shown in FIG. 3 has a stereo camera for inputting an image of a landmark. Was. The second embodiment is a method for determining camera parameters when the present invention is applied to a system having a monocular camera and a three-dimensional position and orientation sensor.

In order to compensate for the mutual drawbacks of the image-based and sensor-based registration methods, attempts have been made to perform registration using information on both the image and the sensor. This includes a concept of using sensor information to stabilize image-based alignment, and a concept of mainly capturing sensor-based alignment and correcting an error using image information. In the second embodiment, a method of correcting a positional shift in sensor-based alignment using monocular image information will be described below.

<Case where Three Points are Observed> 2-1-1 Example A situation is assumed where three landmarks are extracted on an image. As described above, if depth information of each landmark is available, equation (23) can be solved using the three landmarks. Here, since approximate camera position and orientation information can be used by the three-dimensional position and orientation sensor, depth information of each landmark is derived based on this information.

Now, it is assumed that the landmark Q _i (i = 1, 2, 3) is extracted as a point q _i = (x _i , y _i ) (i = 1, 2, 3) on the image. . At this time, the position and orientation M ^WC of the camera obtained from the three-dimensional position direction sensor (expressed as the coordinate transformation matrix 4x4 from the world coordinate system to the camera coordinate system), the camera coordinate of the landmark Q _i is

[0064]

(Equation 34) … (35). Utilizing the Z component Z ^(c) _Ci as the depth information of the landmark Q _i. As shown in FIG. 5, it is assumed that landmarks Q ₁ , Q ₂ and Q ₃ are observed on the image. At this time, the matrix U can be set as follows based on the image coordinates of each landmark and the depth information obtained by Expression (35).

[0065]

(Equation 35) … (36) The camera parameter matrix C obtained from this matrix U is expressed on the three landmarks with respect to the camera parameter matrix C ^(c) (= PM ^(WC) ) obtained from the three-dimensional position and orientation sensor output. Is corrected so as to remove the positional deviation.

<Case where Two Points Are Observed> 2-2nd Example Assume that two landmarks are extracted on the image. In this case, by virtually setting the third landmark, the camera parameters can be estimated in the same manner as described above. Assume that landmarks Q ₁ and Q ₂ are observed as shown in FIG. The third (virtual) landmark Q ₃ is Q _W1 , Q
It is assumed that there is a point Q _W3 on the Z = 0 plane that is not collinear with _W2 . The depth value Z ^(c) _Ci of the landmarks Q ₁ , Q ₂ , Q ₃ is ^calculated as
Determined by (35), further, estimates projected coordinates onto the image plane of the landmark Q ₃ a _{^{(x (c) 3, y}} (c) 3) by the following equation.

[0067]

[Equation 36] … (37)

[0068]

(37) (38) Using these, the matrix U is set as in equation (36). The camera parameter matrix C obtained in this manner is obtained by correcting the camera parameter matrix obtained from the three-dimensional position and orientation sensor output so as to remove a displacement on two landmarks. Become.

<Case where One Point is Observed> 2-3 Example Even if one landmark is extracted on the image, two virtual landmarks are set in the same manner as in the case of two points. By assuming this, it is possible to correct the displacement on the landmark. However, in the case of one point, as shown in FIG. 7, a point Q _C1 on the straight line passing through the origin of the camera coordinate system and the point q ₁ where Z = Z ^(c) _C1 is obtained, and Q ^(c) _C1 -By moving the camera position information in parallel only by Q _C1 (39), the displacement on the landmark can be corrected more easily.

<Positioning by Stereo and Three-Dimensional Sensor> Third Embodiment A method of integrating the above-described image-based positioning method and the sensor-based position shift correction method is proposed. The above-described method obtains a matrix U in equation (23) from the input image coordinates (x _i , y _i ) of three landmarks (including virtual) and depth information Q _i , and solves the matrix U to solve this. It was to estimate the matrix C representing the parameters. In the third embodiment,
By integrating these methods, an alignment method using both a stereo camera and sensor information is realized. This integration is realized by adaptively switching the method of estimating the depth value of each landmark according to the extraction state of the landmark on the left and right images. Hereinafter, a method of estimating camera coordinates for each situation where a landmark is extracted will be described.

<Extracting All Three Points with Binocular> Third Embodiment When all three points are extracted with both eyes, that is, when all three points are extracted with a stereo camera, each of the three points is extracted. The depth value of the landmark is estimated based on the stereo information. That is, the method of the first embodiment (Equations (30) to (33)) is applied as it is. <Two points are extracted with both eyes, and one point is extracted with one eye> ... Example 3-2 Two of the three points (Q ₁ , Q ₂ ) are obtained by a stereo camera.
If one point (Q ₃ ) is extracted with a single eye, that one point (Q ₃ )
Is not immediately obtained.

Therefore, the depth values Z _C1 , Z of the two points Q ₁ , Q _2
C2 is estimated based on the stereo information. On the other hand, the depth value Z ^(s) _C1 , Z of each landmark based on the sensor information
^(s) _C2 and Z ^{( s)} _C3 are estimated using equation (35). Furthermore, for landmarks Q ₁ and Q ₂ ,

[0073]

(38) The coefficient k _i that satisfies (40) is obtained, and the average value k _av is calculated. Using this coefficient k _av ,

[0074]

[Equation 39] The matrix U is obtained from Expression (36) using Z _C3 obtained by (41) as the depth value of Q ₃ . <1 point in both eyes, extracted with monocular two points> ... In this case 3-3 embodiment estimates based on the depth value Z _C1 Q ₁ at the stereo information. On the other hand, the depth values Z ^(s) _C1 , Z ^(s) _C2 , and Z ^(s) _C3 of each landmark based on the sensor information are estimated using Expression (35). In addition, for the landmark Q _1,

[0075]

(Equation 40)... coefficient k that satisfies (42)_avAnd calculate Q in the same manner as in equation (41)._Two, Q_ThreeDeep inside
The outgoing value is calculated and substituted into equation (35) to obtain a matrix U. <Extracting two points with both eyes> ... Example 3-4 In this case, Q₁, Q_TwoDepth value of Z_C1, Z_C2The stereo information
And depth value Z based on sensor information^(s)
_C1, Z^(s) _C2To the coefficient k_avIs calculated. Further, the second embodiment
The third (virtual) landmark Q_Three
Image coordinates (x^(s) _Three, y^(s) _Three) And depth value Z^(s) _C3And estimate
Z obtained by equation (41)_C3The Q_ThreeIs the depth value of. This
The matrix U is obtained by substituting these values into equation (35).

<One point is extracted with both eyes and one point is extracted with one eye>
Fifth Embodiment The depth value Z _C1 of Q ₁ is estimated based on stereo information, and a coefficient k _av is calculated from the depth value Z ^(s) _C1 based on sensor information. Further, based on the sensor information, Q ₂ 'depth value Z ^(s)
And _{C2, the} third of (virtual) landmark Q ₃ of the image coordinates ^{(x (s)}
_{3, y} ^(s) ₃ and depth value Z ^(s) _C3 are estimated, and Q ₂ ,
To calculate the depth value of Q _3. The matrix U is obtained by substituting these values into equation (35).

[0077] <extracting one point with both eyes> ... estimated based the 3-6 embodiment to Q ₁ camera coordinate Q _C1 in the stereo information,
The camera position is corrected by the method of the 2-3 embodiment. <Case where stereo information cannot be obtained> ... Example 3-7 Example Where stereo information cannot be obtained is a case where point m is extracted with a single eye, and the depth value of each landmark is determined based on sensor information. presume. That is, the method of the second embodiment is applied as it is.

<Control of Process Selection> As described above, the processing of the camera parameter determination device of the present invention differs depending on the presence or absence and number of sensors or cameras mounted.
Even if there is no change in the sensor or camera, it is adaptively required to take any one of the first to third embodiments depending on the number of landmarks detected as images. This selection is performed by the parameter estimation module 300, for example.

That is, the module 300 can know what device is attached to the determination device via an interface (not shown). Once the device type / number is known, the module 300 queries the coordinate detection module 200 to find out how many landmarks are captured in the image currently being captured from the camera. As a result, the depth estimation module 100
, An instruction to switch the processing algorithm (first to third embodiments) is issued.

<Experimental Results> An experiment was conducted to evaluate the effectiveness of the above-described alignment method. The experiment included a 6-DOF magnetic azimuth sensor (Polhemus Fastrak) and a small color C
An HMD equipped with two CD cameras (ELMO MN-421) was used. To generate the presentation video,
One SiliconGraphic workstation O2 was used. Landmark tracking image processing was performed by two Hitachi image processing boards IP5005 mounted on a PC.
The video from the camera is branched and input to O2 and the image processing board,
Data transfer from PC to O2 was performed by packet communication over Ethernet.

As landmarks in the real space, red marks were set at a plurality of points with known world coordinates. The image processing board binarizes and labels the input image according to a preset range of the mark color (in the YUV space), and extracts the coordinates of the center of gravity of each cluster at a video rate. The extracted coordinate data is transferred to O2, and the landmark is identified by comparison with the predicted observation position of each landmark obtained from the sensor information.

The update rate of the presented image in the constructed system was 10 Hz on average. The update rate when only the magnetic sensor was used was also 10 Hz, and it was confirmed that the effect of the alignment calculation on the performance of the entire system was almost negligible. In order to evaluate the proposed method quantitatively, we applied some alignment algorithms to the same situation and measured the change of displacement. The misalignment was measured by preparing a number of known three-dimensional position reference points other than the landmarks and calculating the average value of the magnitude of the misalignment at each point of the landmarks and the reference points. The experiments were (a) 3-point binocular, (b) 2-point binocular + 1 point monocular, (c) 1 point binocular + 2 point monocular, (d) 3
Landmark information and 3D of single point eye, (e) 2 point binocular, (f) 1 point binocular + 1 point monocular, (g) 2 point monocular, (h) 1 point binocular, (i) 1 point monocular The case where the position and orientation sensor information was used together and the case where (j) only the three-dimensional position and orientation sensor were used were performed.

FIG. 8 shows the input image (data A: right eye image) used in the experiment, and FIG. 8 shows the results of the fusion of the experiment under the conditions (a), (d), (e), and (j). This is shown in FIGS. 9 to 12. In the figure, “□” indicates an extracted landmark position. As a virtual object, a CG figure of the wire frame model was superimposed on a real cube and displayed. The values of the error under each condition are shown in FIG. In FIGS. 13 to 15, the horizontal axis represents the above-described methods (a) to (j), and the vertical axis represents the generated positional deviation. Further, FIGS. 14 to 15 show the results of similar experiments performed while changing the positional relationship between the observation target, the HMD, and the emitter of the magnetic sensor. Data B moves the HMD away from the emitter and closer to the observation target.
C is the result when the HMD is installed at a position distant from the observation target. Comparing the method using only a single eye, the method (d) using three feature points can be aligned with high accuracy, and the method (d) using two points (g) and one point (i) It can be seen that a marked improvement in accuracy is obtained as compared with the case where no correction is performed (j).

Looking at the effect of the alignment by the stereo information, data A (FIG. 13) and data C (15
In the figure), except for the case of one point (h), the result was smaller when no stereo information was used.
This is considered to be due to the influence of an error in feature point extraction by image processing or a relative position between cameras given as known. on the other hand,
In the situation of the data B, it can be seen that the use of stereo information improves the alignment accuracy. This is because the accuracy of distance information estimation by stereo is inversely proportional to the distance from the camera to the object, and the accuracy of distance information by stereo image processing was relatively good in the situation of data B where the observation target was nearby. It is supposed to be.

<Incorporation as Software> The present determination device detects the position of the viewpoint of the camera with high accuracy, and outputs a coordinate conversion parameter at the detected viewpoint, that is, a camera parameter. The output of the camera parameters is nothing but outputting the matrix elements of the coordinate transformation matrix C. When determining and outputting the matrix elements of the coordinate transformation matrix C using software, a processing routine for the determination can be incorporated in an application program for AR or MR. It may be incorporated as a ROM in the camera or in the camera body.
This is because it is more convenient for the matrix element determination process to be developed by the manufacturer of the HMD device or the position and orientation sensor than by the user.

When the present invention is applied to a personal computer or a workstation, it is incorporated as driver software. <Effects of Embodiments and Examples> In this specification, a stereo camera and a three-
An alignment method using a dimensional sensor was proposed. This method makes it possible to handle sensor-based and image-based registration methods in the same framework.

The method described in the second embodiment and the third embodiment is as follows.
It is based on the assumption that the information obtained by image processing is always the most reliable. However, as is clear from the experimental results, the accuracy of the distance information obtained by the stereo depends on the distance to the landmark. On the other hand, the accuracy of the three-dimensional position and orientation sensor changes according to the measurement range unique to the sensor. In the future, it will be important to evaluate the reliability of information in accordance with such image processing and sensor characteristics, and to study a positioning method that selects an optimal solution in accordance with the reliability.

Further, since the coordinate transformation obtained by this method does not maintain the orthogonality of the coordinate axes, an unnatural deformation may be applied to the virtual space. Dealing with such a situation is also an issue for the future.

[0089]

As described above, according to the present invention,
The position and orientation can be accurately detected by the three landmarks.

[Brief description of the drawings]

FIG. 1 is a view for explaining an arrangement relationship between one viewpoint and three landmarks in a position and orientation detection apparatus according to an embodiment.

FIG. 2 is a view for explaining a correction method when generalizing to an arbitrary plane other than Z = 0.

FIG. 3 is a diagram illustrating a configuration of an apparatus according to the embodiment.

FIG. 4 is a view for explaining the relationship between two viewpoint positions and one landmark.

FIG. 5 is a view for explaining the principle of determining camera posture parameters when three landmarks are used.

FIG. 6 is a view for explaining the principle of determining camera posture parameters when two points are used as a landmark and one point is imagined;

FIG. 7 is a view for explaining the principle of determining a camera posture parameter when two points are imagined using one landmark.

FIG. 8 is a perspective view of an object used in an experiment using the embodiment of the present invention.

FIG. 9 is a view for explaining that virtual landmarks are displayed on an experimental object in an overlapping manner in an experiment in which three landmarks are performed using a stereo camera.

FIG. 10 is a view for explaining that three landmarks are displayed so that a virtual figure is superimposed on an experimental object in an experiment performed using a monocular camera and a three-dimensional sensor.

FIG. 11 is a view for explaining that two landmarks are displayed with a virtual figure superimposed on an experimental object in an experiment performed using a stereo camera and a three-dimensional sensor.

FIG. 12 is a view for explaining that a virtual figure is displayed so as to be superimposed on a test object in an experiment performed using only a three-dimensional sensor.

FIG. 13 is a view for explaining positional displacements that occur when data A is changed to conditions a to j.

FIG. 14 is a view for explaining positional displacements that have occurred in data B when the conditions are changed to conditions a to j.

FIG. 15 is a view for explaining positional displacements that have occurred in data C when the conditions are changed to conditions a to j.

Claims (10)

[Claims] 1. An image of three marks placed in a predetermined plane is obtained using a camera, image coordinates of the three marks are obtained in the image, and depth information of the three marks is obtained. A method for calculating and outputting parameters for determining the position and orientation of the camera based on image coordinates and depth information obtained for the mark. 2. The viewpoint position and orientation determination according to claim 1, wherein depth information of the three marks is detected based on a monocular camera and an output of a position and orientation sensor provided in the camera. Method. 3. The method of determining a viewpoint position and orientation according to claim 1, wherein depth information of the three marks is obtained from an output of a stereo camera and a three-dimensional position and orientation sensor. 4. The method according to claim 1, wherein when depth information of at least one mark is obtained, depth information of a virtual mark is set. 5. The viewpoint position and orientation according to claim 1, wherein the three marks are selected as marks on the Z = 0 plane in world coordinates that do not lie on the same straight line. How to determine. 6. In the case where the three marks are selected as marks on an arbitrary plane which is not on the same straight line and which is not on the Z = 0 plane in world coordinates, Z is selected from the arbitrary plane which is not on the Z = 0 plane. The viewpoint position according to any one of claims 1 to 3, wherein a transformation matrix to a = 0 plane is obtained, and camera parameters are determined using the world coordinates of the landmark transformed by the transformation matrix. How to determine the posture. 7. A camera device for outputting position and orientation data together with an image, comprising: a camera for capturing images of three marks placed in a predetermined plane; and coordinate values of the images of the marks captured by the camera. Means for calculating depth information of the three marks; means for calculating parameters for determining the position and orientation of the camera based on image coordinates and depth information obtained for the marks; and A camera device comprising: 8. A head position / posture sensor, which is attached near the head and captures images of three marks placed in a predetermined plane, and an image of the mark captured by the camera. Means for obtaining coordinate values; means for obtaining depth information of the three marks; and calculating parameters for determining the position and orientation of the camera based on image coordinates and depth information obtained for the marks. And a means for outputting. 9. A storage medium for storing a program for causing a computer to execute the method for determining a viewpoint position and orientation according to claim 1. Description: 10. The image coordinates of the three marks are represented by (x₁,
y₁), (x_Two, y_Two), (x_Three, y _Four), Depth information z₁, z_Two, z_Three,
The world coordinate value is (X_W1, Y_W1, 0), (X_W2, Y_W2, 0), (X _W3, Y
_W3, 0), the camera's parameter matrix is:(Equation 2)Then,From the matrix C given by
Camera position and orientation detection method.