JP2003279310A - Apparatus and method for correcting position and attitude - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To identify the position and attitude of an image pickup device by simultaneously utilizing a plurality of feature points and correcting the position and the attitude of the image pickup device without distorting space. <P>SOLUTION: A signal indicating the position and the attitude of an HMD (Head Mount Display) is inputted from a sensor (S201). On the basis of a signal indicating a measurement result, viewing conversion is computed in the form of a conversion matrix (S202). Then the locations of the feature points at known locations in the actual world are converted from a world coordinate system into a camera coordinate system, and are computed into an image surface by perspective projection (S203). The image of an actual space is acquired from a camera (S204), and the feature points are detected from the picked-up image S205). On the basis of displacements between projected points of the feature points and detected points of the feature points, the correction of error of a sensor is processed (S206). <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a position / orientation correction device and a position / orientation correction method for correcting the position / orientation of an imaging device provided with an imaging unit for imaging a real space and a measuring unit for measuring the position / orientation in the world coordinate system. It is a thing.

[0002]

2. Description of the Related Art JP-A-11-084307 and JP-A-2000-0411
As already pointed out in Japanese Patent Publication No. 73 and Japanese Patent Application No. 2001-45536, as a method for obtaining the position and orientation of an imager such as a camera for capturing an image of a physical space (hereinafter, appropriately referred to as a camera), an image captured by the camera is In the method of measuring the position and orientation of the camera by using a position and orientation sensor such as a magnetic sensor without using the conventional method, there has been a problem that sufficient accuracy cannot be obtained to measure the position and orientation of the camera. Therefore, JP-A-11-084
In Japanese Patent Laid-Open No. 307, 2000-041173, and Japanese Patent Application No. 2001-45536, a position / orientation sensor is measured using a marker whose position is known in the real space or a feature point whose position is known in the real space. A method of correcting the resulting error has been proposed.

Japanese Patent Application Laid-Open No. 11-084307 and Japanese Patent Application 2001-45536
(1) The three-dimensional position of the marker or feature point on the world coordinate system is known in common, and projection conversion is performed using the camera position / orientation measurement value obtained by the position / orientation sensor. Projection of markers or feature points on the image plane 2
Calculate the two-dimensional position of the marker or the feature point on the image plane by calculating the two-dimensional position of the marker or feature point from the imaged image. (2) The marker obtained in (1) This is a method of obtaining the position and orientation of the camera by correcting the measurement value of the sensor so that the two-dimensional position of (1) matches or approaches the two-dimensional position of the marker obtained in (2).

In Japanese Patent Laid-Open No. 11-084307, one marker in the field of view is selected from a plurality of markers arranged in the physical space, and the marker is displayed on the image planes (1) and (2) above. There is disclosed a method of correcting the position and orientation of the camera so that the positions of 1 and 2 match. Also, Japanese Patent Application 2001-455
No. 36 describes a method for correcting the error of the sensor by using a plurality of feature points in the real space at one time. In this method, the error of the sensor is corrected by using a plurality of feature points on average, and the correction algorithm is disclosed in JP-A-11-08430.
It can be regarded as an extension of the algorithm used in Publication No. 7.

Further, in Japanese Patent Laid-Open No. 2000-041173, 3
By using the detection results of the point markers and the measured values of the sensor simultaneously as one, the projective transformation matrix from the physical space to the imaging surface of the camera is obtained by matrix calculation instead of the parameter of the position and orientation of the camera. Discloses a method of obtaining a projection matrix transformation corresponding to an external parameter of a camera. As a procedure, although the procedure of correcting the measurement value of the sensor with the marker is not used, the marker is used for the purpose of correcting the measurement value of the sensor.

Here, the external parameters of the camera are not the parameters such as the focal length, the principal point, the lens distortion, etc. which are inherent in the camera (called the internal parameters of the camera), but the position and orientation of the camera. Is a parameter that represents how the camera exists with respect to the outside world. In this Japanese Patent Laid-Open No. 2000-041173, the parameters such as the position and orientation of the camera are not estimated but estimated as a mapping expressed in the form of a 4 × 4 matrix. However, strictly speaking, the estimated matrix does not consist only of components caused by the position and orientation of the camera. That is, the orthogonal transformation is not always the case.

[0007]

According to the method disclosed in Japanese Patent Laid-Open No. 11-084307, a plurality of markers are arranged in the physical space, but the marker actually used for the correction calculation is selected from among them. I only use one. That is, even if the conditions under which a plurality of markers can be imaged at the same time are met, there is a problem that the information of the plurality of markers cannot be effectively used for correction. In addition, selecting one marker means that the marker used for correction is switched when the field of view is continuously changed. This causes a problem that the correction result greatly changes at that moment, and there is also a problem that the estimation result of the position and orientation of the camera does not have continuous values.

The method disclosed in Japanese Unexamined Patent Publication No. 2000-041173 uses a plurality of markers, but in principle, there is no guarantee that the obtained projective transformation matrix is orthonormal transformation. That is, the obtained mapping does not consist only of the components caused by the position and orientation of the camera, but also includes components that lose the orthogonality of the coordinate axes before and after the coordinate conversion. In this case, there is a problem that the space is distorted when the estimated mapping is used.

The method of Japanese Patent Application No. 2001-45536 is a method that addresses the above two problems. A process that uses a plurality of markers at the same time and loses orthogonality is not performed.
However, this method is a method of averaging a correction method that can be calculated with each marker based on the method disclosed in Japanese Patent Laid-Open No. 11-084307 for a plurality of feature points. Here, the projection position of the feature point in the real space on the image is
From the viewpoint of approaching the position of the feature point extracted from the image, the optimum correction calculation is not always obtained. In other words, when three or more characteristic points are captured, the obtained position and orientation of the camera does not minimize the deviation of the characteristic points on the image. In other words, even if the characteristic points captured by the camera are under the same condition, the correction result changes depending on the output of the 6-degree-of-freedom sensor having an error before correction. , It is a method that does not make full use of the information of feature points.

The present invention has been made in view of the above points, and a plurality of characteristic points are used at the same time, and the position and orientation of the image pickup device is corrected without distorting the space. The purpose is to identify the position and orientation of.

[0011]

In order to achieve the object of the present invention, for example, a position / orientation correction device of the present invention has the following configuration.

That is, a position / orientation correction apparatus for correcting a parameter indicating the position / orientation of an image pickup apparatus having an image pickup section for picking up an image of the real space and a measuring section for measuring the position / orientation in the world coordinate system, which exists in the real space. In an image including a plurality of feature points obtained by the image capturing apparatus capturing a plurality of feature points, a first position calculating unit that obtains positions of the plurality of feature points in the image, and the image capturing by the measuring unit. Based on the parameter indicating the position and orientation of the device, the second position calculating means for obtaining the positions of the plurality of feature points existing in the physical space on the image plane and the position by the first position calculating means are obtained. Each characteristic point,
And a correction unit configured to obtain a distance between corresponding feature points with respect to each set with respect to each feature point whose position is obtained by the second position calculation unit, and to correct the parameter using each distance. Is characterized by.

In order to achieve the object of the present invention, for example, the position and orientation correction method of the present invention has the following configuration.

That is, it is a position and orientation correction method for correcting a parameter indicating the position and orientation of an image capturing apparatus having an image capturing unit for capturing an image of the physical space and a measuring unit for measuring the position and orientation of the world coordinate system, which exists in the physical space. In an image including a plurality of feature points obtained by the imaging device capturing a plurality of feature points, a first position calculation step of obtaining positions of the plurality of feature points in the image, and the imaging by the measuring unit. The positions were obtained in the second position calculation step of obtaining the positions of the plurality of feature points existing in the physical space on the image plane based on the parameter indicating the position and orientation of the device, and in the first position calculation step. With each feature point and each feature point whose position was obtained in the second position calculation step,
And a step of calculating the distance between the corresponding feature points for each set and correcting the parameter using each distance.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will now be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment] FIG. 1 shows a functional configuration of a position / orientation correction device and an HMD according to the present embodiment. Ten
1 is a head-mounted device (HMD = Head Mount Display) in which a camera, a three-dimensional position / orientation sensor, and a display, which are movable in the physical space, are fixed to one another.
y). Reference numeral 101a is a three-dimensional position and orientation sensor that uses magnetism to measure the position and orientation, 101b is a camera capable of capturing an image of a physical space and obtaining an image, and 101c is a display capable of displaying an image. Here, a requirement necessary as a basic configuration in the present embodiment is that the three-dimensional position / orientation sensor 101a and the camera 101b have a fixed relationship with each other and can move in the physical space. The display 101c does not necessarily have to be fixed to the other two, or need not be built into the HMD 101. The three-dimensional position / orientation sensor 101a outputs a position / orientation signal, which is a measurement result, to the position / orientation measurement unit 102 described below, and the camera 101b outputs an imaging result to a feature point detection unit 103 and an image composition unit 107 described below. . display
Reference numeral 101c is a device that receives an image signal as an input and displays an image. The device 101c obtains an image signal from the image combining unit 107 and displays the content thereof.

Reference numeral 100 denotes a position / orientation correction device in this embodiment, which has a function of correcting the position / orientation of the HMD and outputting an image according to the corrected position / orientation to the HMD.
A position / orientation measuring unit 102 receives a signal indicating the 3D position / orientation output from the 3D position / orientation sensor 101a and generates a viewing transformation matrix representing the position / orientation of the camera 101b. This viewing conversion will be described later. 103
Is a feature point detection unit that has a function of inputting an image obtained by the camera 101b and detecting a feature point in the image in the real world (a marker artificially provided in advance or a natural feature may be used). Yes (at the same time, the positions of feature points in this image are also detected). 104 is a feature point projection calculation unit that converts the position of a feature point in the real world (the position in the world coordinate system described below) into the position in the camera coordinate system by the viewing conversion obtained by the position and orientation measurement unit 102, and It is a feature point projection calculation unit that calculates the perspective projection of the camera to calculate the position on the image plane. Detailed processing will be described later.

A position / orientation correction unit 105 compares the projected position of the feature point obtained by the feature point projection calculation unit 104 on the image plane with the position of the feature point obtained by the feature point detection unit 103, HM
The position / orientation correction unit corrects the position / orientation of D101 with higher accuracy. An image generation unit 106 includes a position / orientation correction unit 10
The virtual space is drawn using the data prepared in advance based on the signal indicating the corrected position and orientation obtained in 5, and an image is generated. Reference numeral 107 denotes an image synthesizing unit, which inputs a captured image from the camera 101b, and the image generating unit 1 receives the input captured image.
The image generated by 06 is drawn (combined). As a result, a combined image that matches the current position and orientation of the camera is displayed on the display unit 101c. The image generator 10
6 and the image composition unit 107 are constituent blocks for performing image composition in accordance with the estimated position and orientation of the camera, as one form of use of the correction result of the position and orientation of the camera, and are used in the position and orientation correction processing described later. It is not related.

The processing performed by the position / orientation correction apparatus having the above configuration will be described with reference to the flowchart shown in FIG. FIG. 2 is a flowchart of main processing performed by the position / orientation correction apparatus according to this embodiment.

First, the position and orientation of the HMD 101 is measured by the three-dimensional position and orientation sensor 101a, and a signal indicating the measurement result is sent to the position and orientation measuring unit 102 (step S20).
1). Next, the position / orientation measurement unit 102 calculates the viewing transformation in the form of a transformation matrix based on the signal indicating the measurement result. The viewing matrix will be described below with reference to FIG.

FIG. 3 is a schematic diagram of a camera in the physical space. As shown in the figure, x _c y _c plane imaging surface to the point of view of the camera 101b and the origin, the three-dimensional coordinate system at a vector to z _c-axis negative visual axis and the camera coordinate system, fixed to the real world The coordinate system thus set is the world coordinate system (in the figure, x, y, and z axis directions are x _w , y _w , and z _{w, respectively} ). Viewing transformation is coordinate transformation between these two coordinate systems, and refers to a transformation matrix that transforms a point in the world coordinate system into a point in the camera coordinate system. This step
The step S202 will be described in more detail below.

First, the three-dimensional position and orientation sensor 101a
Can measure the position and orientation of the camera in the world coordinate system
I shall. Turtle obtained by 3D position and orientation sensor
A three-dimensional vector representing the position of LA_conW, Posture
R 3 × 3 rotation transformation matrix_{c onW}Then the camera
Point X on the coordinate system_c= (X_c, Y_c, Z_c) In the world coordinate system
Upper point X_w= (X_w, Y_w, Z_w) Conversion formula
Is Can be expressed as Here, as the posture measurement values, the z-axis center and the x-axis
Roll / pitch / yaw angle that represents the rotation angle around the center and y-axis
When R is expressed as r, p, and y, respectively, R_{co nW}But
For example, yaw rotation, pitch rotation, roll rotation
If it is the result of adding rotation, From Can be calculated as Also, as the measured value of the posture
Rotation axis vector (r _x, R_y, R_z) (However, r
_x ^Two+ R_y ^Two+ R_z ^Two= 1) and rotation angle r_aObtained by
If yes, Can be calculated as

The viewing transformation is a transformation for transforming a point on the world coordinate system into a camera coordinate system, and therefore corresponds to the inverse transformation of the coordinate transformation expressed by the equation (1). That is, the point on the world coordinate system X _w = (x _w , y _w ,
z _w ) from the camera coordinate system X _c = (x _c , y _c , z _c ).
Conversion to Is expressed as The step of calculating this M _conw ⁻¹ is the step of _{obtaining the} viewing transformation. Therefore, the process of step S202 is to obtain a 4 × 4 conversion matrix for converting from the camera coordinate system to the world coordinate system based on the measured values of the position and orientation measured by the three-dimensional position and orientation sensor. This is a step of obtaining an inverse matrix. Here, to explain the subsequent steps, _Let R be the rotation component of M _conw ⁻¹ and T be the translation component. As a result, X _c = RX _w + T. ． ． It can be expressed as (5). As a supplement, these R and T are not the measured values of the position and orientation of the camera itself (the rotation component and the translation component of the inverse transformation).

Next, the positions of the characteristic points at known positions in the real world are converted from the world coordinate system to the camera coordinate system, and perspective projection calculation is further performed on the image plane (step S203). Here, the perspective projection is a transformation for projecting a three-dimensional point on the camera coordinate system to a two-dimensional coordinate U = (u _x , u _y ) on the image plane, and u _x = f _x × x _c / z _c u _y = f _y × y _c / z _c . ． ． This is a conversion represented by (6). Here, f _x and f _y are the focal lengths of the camera in the x-axis direction and the x-axis direction, respectively, as shown in FIG. The focal length of the camera physically has one value, but here, in order to simultaneously express the aspect ratios in the x-axis direction and the y-axis direction, two values are used as the focal length. In addition, the focal length is a value specific to the camera and is obtained in advance. This perspective projection conversion formula (6) is converted into (ux, uy) = P (xc, yc, zc) U = P ( _Xc ). ． ． Expressing as (7), the equation for converting the point on the world coordinate system on the screen is as follows from the equations (5) and (7): U = P (RX _w + T). ． ． It can be expressed as (8). That is, the projection calculation on the image plane performed in step S203 is a step of calculating the position on the image plane of a point on the world coordinate system using this equation (8).

Further, steps S201, S202, S2
While performing step 03, the image of the physical space imaged by the camera 101b is obtained (step S204), and the feature point detection unit 103 detects a feature point from the imaged image obtained by the camera 101b (step S205). For this, a marker having a closed area of a specific color is used, and the marker is obtained by color detection processing, or the real space is photographed in advance, and the area that becomes the feature point is prepared as a template, and template matching is performed. There are ways to do it.

The above steps S201, S202, S2
By the process of 03, the position of the feature point in the image based on the measurement value of the three-dimensional position and orientation sensor is obtained.
By the steps of 204 and S205, the position of the feature point in the captured image can be obtained. If there is no measurement error of the 3D position and orientation sensor, these points on the image plane will match, but in reality, the 3D position and orientation sensor has a measurement error. Do not match. This will be described with reference to FIG.

FIG. 5 shows steps S201, S202 and S2.
It is a figure which shows the position of the characteristic point in the image based on the measurement value of the three-dimensional position and orientation sensor obtained by the process of 03, and the position of the characteristic point in a captured image by the process of steps S204 and S205. In the figure, 500 is 3
An image in the real space viewed from the camera 510 in the position and orientation according to the measurement values obtained by the three-dimensional position and orientation sensor 101a. In the image 500, the x mark indicates the feature point projection calculation unit 10
4 shows respective feature points projected on the image by the above projection calculation. On the other hand, 501 is an image in the real space seen from the camera 520 in the actual position / orientation, and black circles in the image 501 indicate respective characteristic points seen from the camera 520.
Reference numeral 503 denotes an image in which the images 501 and 502 are overlapped. In the processing for correcting the position and orientation described later, the measurement value is corrected so that the distance between each x mark and each black circle in the image 503 is minimized. Processing.

Since the position and orientation of the camera obtained by the three-dimensional position and orientation sensor are different from the true position and orientation of the camera,
The positions of the projected points of the characteristic points (points indicated by x) and the detection points of the characteristic points (points indicated by black circles) do not match as shown in FIG. Therefore, the position / orientation correction unit 105 performs processing for correcting the error of the three-dimensional position / orientation sensor based on the deviation on the image plane (step S206). Specifically, the position / orientation correction unit 105 corrects the viewing transform using the steepest descent method so that the error is minimized. This step S20
Step 6 will be described in detail below with reference to FIG.

FIG. 6 is a flow chart showing details of the processing in step S206. First, step S20
It is determined whether there are three or more feature points detected from the image in step 5 (step S601), and if there are three or more feature points, the process proceeds to step S602. On the other hand, if there are three or more, the calculation for correcting the error cannot be performed, and thus the process proceeds to step S207.

When there are three or more characteristic points detected from the image, the characteristic points detected from the image are associated with the projected points of the characteristic points calculated in step S203 in step S205. This means that when n (n ≧ 3) feature points have been detected in step S205 and the position on these images is V _i , the closest projection point obtained in step S203 And find the point U _i
This is the step of storing (step S602). Further, in step S602, the point on the world coordinate system of the feature point that is the projection source of the projection point is also set as X _i . Then, the process proceeds to step S603.

Before describing the steps after step S603, the principle will be described below in order to explain the processing contents of the steps performed from here. The rotation component of equation (5) is
It can be expressed by the rotation axis and the rotation angle as shown in the equation (3), and the rotation angle is the length of the rotation axis ((ω _x ,
ω _y , ω _z ) = (r _a r _x , r _a r _y , r _a r _z )) can be expressed in three values. Therefore, the parameter of 6 degrees of freedom including the rotation component and the translation component is vector S = [ω
_x ω _y ω _z t _x t _y t _z ]. As a result, the viewing conversion formula (5), which is the coordinate conversion by rotation and parallel movement in the three-dimensional space, is converted into X _c = Q (S, X _w )
When expressed as, conversion formula (8) including projection conversion
U _i = N ′ (Q (S, X _i )) = F (S, X _i ). ． ． It can be expressed as (9).

On the other hand, regarding the position of the feature point detected on the image,
Is obtained based on the measurement values of the 3D position and orientation sensor.
6-degree-of-freedom parameter S of the viewing transformation and true power
Viewing transformation that can be obtained from the position and orientation of the camera
If the difference in the 6-degree-of-freedom parameter of
The 6-degree-of-freedom parameter of the wing transformation is S + ΔS.
),     V_i= N '(Q (S + ΔS, X_i)) = F (S + ΔS, X_i). ． ． (10) The formula holds. Measurement error in 3D position and orientation sensor
If there is no ΔS is 0, then U_iAnd V_iIs the same point
However, since the sensor generally has an error, U_iAnd V _iIs the same
It's not the same point. To correct the error of this sensor
That is U_iAnd V _iBased on the error between
In other words, S + ΔS is estimated. In other words
For example, the distance between these two points, that is, the error vector on the image plane
Le, ΔU_i= U_i-V_i        ． ． ． (11) Error of the length of However, the minimum ΔS or S + ΔS can be obtained.
You should come. To apply the steepest descent method,
Ω on the right side_x, Ω_y, Ω_z, T_x, T_y, T_zPartial differentiation with
Find the function Ω in equation (9)_x, Ω_y, Ω_z, T_x，
t_y, T_zEach part has a solution that is partially differentiated by 6 variables
The Jacobian matrix is Is. Here, from the vector A with the number of j elements,
The Jacobian matrix for vector B is a matrix with j × k elements.
Riko J_baIs displayed. u_xAnd u_ySomething
Partial differentiation with the variable p of Therefore, each element in the three columns on the right side of Expression (11) is As the position of the feature point on the camera coordinate system and the focal length.
Can be requested. Similarly, the left side of equation (11)
Each element in the three columns is Is. Where ∂x_c/ ∂ω_x, ∂x_c/ ∂ω_y, ∂
x_c/ ∂ω_z, ∂y_c/ ∂ω_x, ∂y_c/ ∂ω_y, ∂y
_c/ ∂ω_z, ∂z_c/ ∂ω_x, ∂z_c/ ∂ω_y, ∂z_c
/ ∂ω_zIs not described in detail, but the features on the world coordinate system
It can be calculated from the position of the point and the rotation axis rotation angle.
Then, the left three columns of equation (15), that is, equation (11), is the world
Obtained from the position of the feature point on the coordinate system and the rotation angle of the rotation axis
be able to.

Further, from equation (11), The formula holds. This is for a small change in S
And J obtained based on the value of S at that time_usUsing
Therefore, it is possible to calculate a minute change amount of U.
I'm tasting. From this, the actual error amounts ΔU and ΔS are
When ΔS is small, It becomes a relationship. Here, the points used for this correction calculation
Consider that there are n points. Equation (17) is
Since it consists of points, the vector of n points is further vertical
Using the vector arranged in The formula holds. [ΔU in equation (18)_us1  Δ
U_us2  ． ． ． ΔU _n] And [ΔJ_us1  ΔJ
_us2  ． ． ． ΔJ_usn] Respectively Then, the error becomes the minimum with the same weight at all points.
S is a generalized inverse matrix (pseudo inverse matrix) of Φ, ΔS = (ΦΦ^t)^-1Ψ. ． ． (20) Can be asked as

This corresponds to a calculation process for obtaining the next S when S is present in the problem of obtaining S by the steepest descent method. In other words, it can be applied as a calculation of one process with the iterative optimization process of the steepest descent method. That is, when there is a certain initial value S ₀ , the positions X ₁ , X _2, ..., X _n of the invariant feature points on the world coordinate system and the error vector [Δu ₁ Δu ₂ . ． ． Based on Δu _n ], the following can be obtained using the equation (20). Using this result, S ₁ = S ₀ + ΔS, and [Δu ₁ Δ
u ₂ . ． ． Δu _n ] is recalculated, and the equation (2
0) is used to determine ΔS. Furthermore, using this result,
S ₂ = S ₁ + ΔS, and the same calculation is performed again [[Δu ₁ Δu ₂ . ． ． Delta] u _n] until approaches zero, or is repeated until almost no difference between the _{S k-1} and _{S k} when repeated k times. By this iterative process, the 6-degree-of-freedom parameter of the camera viewing transformation that minimizes the error between the projected position and the detected position of the feature point on the image plane can be obtained as

Up to this point, the principle of the steps after step S603 has been described. Continuing from this, step S6
The steps after 03 will be described using the above formula. The 6-degree-of-freedom parameter of the viewing transformation matrix obtained in step S202 is set to S described above (step S
603). Next, based on the position X _i (i = 1, 2, ..., N) of the feature point to be used on the world coordinate system and the current value of S, equations (13) and (15), That is, the formula (1
J _usi which is 1) is obtained (step S605). Up to this point, U _i and J _usi have been obtained. Then, since _{V i} is obtained in step S205, _{V i} and _{U i} and J
ΔS is calculated from _usi by using the equation (20) (step S606).

Next, it is judged whether or not ΔS is sufficiently small (for example, whether or not it is smaller than a predetermined threshold value θ) (step S607), and it is sufficiently small (S is a sixth-degree-of-freedom parameter with a small error. If so, the process advances to step S207 to perform the process as described above. On the other hand, Δ
If S is not sufficiently small, the process proceeds to step S608, and S + ΔS is newly replaced with S (step S60
8) The process proceeds to step S604. Then, U _i is recalculated using the updated S (step S60
4). Then, the processing from step S604 to step S608 is repeated until ΔS becomes sufficiently small.

As described above, the correction result is not obtained as the correction amount indicating the difference such as the correction matrix and the correction value, but the 6-degree-of-freedom indicating the direct coordinate conversion of the parallel movement component and the rotation component of the viewing transformation. It is obtained as the parameter S. Next, the processing after step S207 shown in FIG. 2 will be described.

In step S204, the captured image obtained from the camera 101b is input to the image composition section 107 (step S207) and written in a display image buffer (not shown) in the image composition section 107. Next, the image generation unit 106
An image of a virtual space or a virtual object is drawn by using the viewing transformation obtained in step S206 (step S208). At this time, by overwriting on the display image buffer in the image composition unit 107, it is possible to obtain an image in which the virtual space or the virtual object is fused with the real space as the background. When the drawing of the virtual space or the virtual object in the display image buffer is completed, the image combining unit 107 displays the combined image in the display unit 10.
1c, and the display unit 101c displays this composite image (step S209).

Here, the processing from step S207 to step S209 is a step for synthesizing the image of the virtual space or the image of the virtual object with the image of the real space as an example of effectively utilizing the obtained viewing transformation of the camera. However, the viewing transform of the camera may be used for other purposes. Of course, if the viewing transform is obtained, it can be easily converted into the position and orientation of the camera in the world coordinate system by calculating the inverse transform, so it can also be used for applications that use the position and orientation of the camera. It goes without saying that it is possible.

The above steps S201 to S2
The series of steps up to 09 are processing steps in a state of the camera and the attached three-dimensional position and orientation sensor at a certain time. The series of processing steps can be performed continuously at consecutive times to estimate the position and orientation of the camera following the movement of the camera.
Step S210 is a process for interrupting this continuous process.

As described above, the position / orientation correction apparatus and the position / orientation correction method according to the present embodiment can correct the error in the measurement result obtained by the sensor for measuring the three-dimensional position and orientation by using the feature points in the real space. it can. Specifically, the position and orientation of the camera are corrected by simultaneously using a plurality of feature points, without distorting the space, and by correcting the position and orientation of the camera so that the distance between the corresponding feature points is minimized. Can be identified.

[Second Embodiment] In the present embodiment, FIG.
An embodiment in which the process in step S206 is changed in the flowchart shown in FIG.
The functional configuration of the position / orientation correction device in this embodiment is the same as that in the first embodiment (FIG. 1). Hereinafter, the processing in step S206 according to this embodiment will be described below with reference to FIG. 7. FIG. 7 shows step S206.
3 is a flowchart showing details of the processing in FIG.

First, it is determined whether or not there are three or more feature points detected from the image in step S205 (step S701), and if there are three or more feature points, the process proceeds to step S7.
Proceed to 02, if there are not three or more, the process is step S.
Proceed to 709. Step S702 to Step S70
Each processing up to 8 is performed from step S602 to step S
Since it is the same as each processing up to 608, description thereof will be omitted.

The step S709 as a feature in this embodiment is a process when three or more feature points are not obtained. When the number of feature points is one, the feature projected on the image plane is Only the rotation component or the parallel movement component of the measurement value of the sensor is corrected so that the position of the point and the position of the feature point in the captured image match, and when the number of feature points is two, The average position of these two points is obtained, and the same process is performed for one point at this average position, or the measurement value of the sensor is corrected by the method in the case of the above-mentioned one point at each point. The calculated result is once calculated, and the two correction results obtained are averaged and corrected.

[Other Embodiments] The object of the present invention is to
Storage medium (or recording medium) recording a program code of software that realizes the functions of the above-described embodiments
It is needless to say that it is also achieved by supplying the above to a system or apparatus, and the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. It goes without saying that a case where some or all of the actual processing is performed and the functions of the above-described embodiments are realized by the processing is also included.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted in the computer or the function expansion unit connected to the computer, based on the instruction of the program code. Needless to say, this also includes a case where a CPU or the like included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

[0047]

As described above, according to the present invention, by using a plurality of feature points at the same time and correcting the position and orientation of the image pickup apparatus without distorting the space, the position and orientation of the image pickup apparatus can be corrected. Can be identified.

[Brief description of drawings]

FIG. 1 is a block diagram showing a functional configuration of a position and orientation correction device and an HMD according to an embodiment of the present invention.

FIG. 2 is a flowchart of main processing performed by the position / orientation correction device according to the embodiment of the present invention.

FIG. 3 is a schematic diagram of a camera in a physical space.

FIG. 4 is a diagram illustrating a focal length.

FIG. 5 shows the positions of feature points in the image based on the measurement values of the three-dimensional position and orientation sensor obtained in steps S201, S202, and S203, and step S204,
It is a figure which shows the position of the characteristic point in a captured image by the process of S205.

FIG. 6 is a step S2 in the first embodiment of the present invention.
It is a flowchart which shows the detail of the process of 06.

FIG. 7 is a step S2 in the second embodiment of the present invention.
It is a flowchart which shows the detail of the process of 06.

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ Identification code FI theme code (reference) H04N 7/18 H04N 7/18 K (72) Inventor Kazuki Takemoto 3-30-2 Shimomaruko, Ota-ku, Tokyo Issue Canon Inc. F-term (Reference) 2F065 AA04 AA37 BB05 BB27 CC00 EE00 FF04 FF65 FF67 JJ03 JJ19 JJ26 PP01 QQ00 QQ24 QQ28 QQ38 SS02 SS13 5B057 AA20 BA02 CA01 CA08 CA12 CA16 CB01 DB16 DB01 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 CD02 DC36 5C022 AB62 AB68 AC01 AC69 5C054 CG07 FD03 FE13

Claims (9)

[Claims] 1. A position / orientation correction device for correcting a parameter indicating a position / orientation of an imaging device, the device being provided in the real space, the imaging device having an imaging unit for imaging the physical space and a measuring unit for measuring the position / orientation in the world coordinate system. In an image including a plurality of feature points obtained by the imaging device capturing a plurality of feature points,
First to obtain the positions of the plurality of feature points in the image
Position calculating means, and second position calculating means for calculating positions on the image plane of a plurality of feature points existing in the physical space based on a parameter indicating the position and orientation of the imaging device by the measuring unit, The distance between the corresponding feature points is obtained for each set by each of the feature points whose position has been obtained by the first position calculating means and each feature point whose position has been obtained by the second position calculating means. A position and orientation correction device, comprising: a correction unit that corrects the parameters by using respective distances. 2. The position / orientation correction apparatus according to claim 1, wherein the correction means corrects the parameter using a steepest descent method. 3. The position / orientation correction apparatus according to claim 2, wherein the correction means corrects the parameters by using the steepest descent method until the respective distances become equal to or less than a predetermined value. 4. The position / orientation correction apparatus according to claim 2, wherein the correction means corrects the parameter by using the steepest descent method until the change in the parameter becomes equal to or less than a predetermined value. 5. The position / orientation correction device according to claim 1, wherein the number of the plurality of feature points existing in the physical space is three or more. 6. A position / orientation correction method for correcting a parameter indicating the position / orientation of an imaging device, which comprises an imaging unit for imaging the physical space and a measuring unit for measuring the position / orientation in the world coordinate system, which exists in the physical space. In an image including a plurality of feature points obtained by the imaging device capturing a plurality of feature points,
First to obtain the positions of the plurality of feature points in the image
And a second position calculation step of obtaining positions on the image plane of a plurality of feature points existing in the physical space based on a parameter indicating the position and orientation of the imaging device by the measurement unit. The distance between the corresponding feature points is calculated for each set by each of the characteristic points whose positions are obtained in the first position calculating step and each of the characteristic points whose positions are obtained in the second position calculating step. And a correcting step of correcting the parameter using the respective distances. 7. A program for causing an information processing device to function as the position / orientation correction device according to claim 1, when read by the information processing device. 8. A program for causing the information processing apparatus to execute the position-orientation correction method according to claim 6 by reading the information processing apparatus. 9. A computer-readable storage medium that stores the program according to claim 7 or 8.