JP2003344018A - Unit and method for image processing as well as program and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To correct a measuring error of a camera eye point by an attitude sensor and to correct a cumulative error of an azimuth direction generated especially due to the passage of time so as to realize an MR without any dislocation. <P>SOLUTION: First, a template image is created (S501). Then, an image I<SP>t</SP>is photographed (S502). A sensor output at this time is acquired (S503). Then, a model view matrix M<SP>t</SP>is calculated based on the sensor output (S504). Then, a correction matrix ΔM<SP>t</SP>is calculated (S505). By using the calculated correction matrix ΔM<SP>t</SP>, the model view matrix ΔM<SP>t</SP>is corrected (S506), and a CG is plotted and displayed by using the corrected model view matrix (S507). <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus for outputting the posture or position / orientation of a measurement target, a method therefor, a program, and a storage medium.

[0002]

2. Description of the Related Art In recent years, mixed reality (hereinafter referred to as "MR" (M
Research on “ixed Reality”) has been actively conducted.

In the MR, an image of a virtual space (for example, a virtual object drawn by computer graphics (hereinafter, referred to as CG) or character information) is added to an image of a real space photographed by a photographing device such as a video camera. Video see-through method with superimposed display and HM worn by user on head
There is an optical see-through method in which an image in the real space is optically transmitted to a D (Head-Mounted Display) while the image in the virtual space is superimposed and displayed on the display screen.

[0004] The applications of MR include the use of medical assistance for presenting to a doctor as if the patient's internal state is seen through, and the use of work assistance for displaying the assembly procedure of the product on the actual product in a factory. , A new field that is completely qualitatively different from the existing VR is expected.

What is commonly required for these applications is a technique for aligning the real space and the virtual space, and many efforts have been made in the past.

The problem of position alignment in the MR of the video see-through method results in the problem of accurately obtaining the position and orientation of the viewpoint of the photographing apparatus. Optical see-through MR
It can be said that the problem of position alignment in (1) is similarly a problem of obtaining the position and orientation of the viewpoint of the user.

Conventional MR systems (especially M indoors)
In the R system), as a method of solving these problems, it is general to derive the position and orientation of these viewpoints using a position and orientation sensor such as a magnetic sensor or an ultrasonic sensor.

On the other hand, in the conventional outdoor MR system, a gyro sensor (strictly speaking, a plurality of gyro sensors for measuring angular velocities in the three-axis directions and accelerations in the three-axis directions are used to derive the attitudes of these viewpoints. Although it is a three-axis attitude sensor configured by combining a plurality of acceleration sensors for measurement, in the present specification, this is referred to as a gyro sensor for convenience).

[0009]

However, when the viewpoint posture is obtained by using the gyro sensor, there is a drift error in the gyro sensor even if a high-precision gyro sensor is used, and therefore there is a lapse of time. Along with this, an error gradually occurs in the measurement value in the azimuth direction. Further, since the gyro sensor can only measure the posture, it cannot follow the change in the position of the viewpoint. That is, a position shift occurs between the real space and the virtual space as time passes and the position of the viewpoint changes.

The present invention has been made in view of the above problems, and an object thereof is to measure the posture or position / orientation of a viewpoint, and in particular, to correct an error in an azimuth direction component that occurs over time. With the goal.

[0011]

In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention has the following configuration.

That is, an image pickup device for obtaining a picked-up image including a measurement target, a posture sensor for measuring a posture of an image pickup viewpoint of the image pickup device, and a posture and / or a position of the measurement target are output to the posture sensor. Storage means for storing calculation information for calculating based on the detection position of the index detected by the detection means, the detection means for detecting the position of the index in the captured image captured by the imaging device And updating the calculation information stored in the storage means, and calculating the posture and / or position of the measurement target based on the measurement value and the calculation information updated by the updating means. And updating means for updating position information in three directions in the camera coordinate system of the image pickup apparatus.

In order to achieve the object of the present invention, for example, the image processing method of the present invention has the following configuration.

That is, a photographed image from the image pickup device is inputted, a measurement result is inputted from a posture sensor for measuring the posture of the image pickup device at the image pickup viewpoint, and the index of the index in the photographed image photographed by the image pickup device is inputted. It is characterized in that the position / posture of the image pickup device is obtained based on the position detection result and the detection position of the index detected by the detection means.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention will now be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment] In this embodiment, an image processing apparatus for realizing presentation of an MR space without misalignment by correcting a posture measurement error of a camera viewpoint by a posture sensor will be described.

FIG. 2 shows the configuration of a conventional image processing apparatus for superimposing an image of a virtual object on an image of a real object according to the attitude of an HMD equipped with an attitude sensor.

The HMD 200 shown in the figure is of a video see-through type, and is a display 201 for displaying an image.
And a camera 202 (observer viewpoint camera) that captures an image of the real space from the viewpoint position of the observer wearing the HMD 200.
And a posture sensor 203 (for example, a gyro sensor here) that measures the posture of the camera 202 from the viewpoint. Further, the image processing device 210 includes the attitude sensor 203.
Sensor input module 21 for inputting sensor output from the
1, an image input module 213 for inputting a captured image of a physical object from the camera 202, a sensor input module 211
Based on the viewpoint orientation of the camera 202 input from the camera 202 and the viewpoint position of the camera 202 obtained by another method.
Based on the information indicating the position / orientation of the viewpoint calculated by the viewpoint position / orientation calculation module 212, which generates information indicating the position / orientation of the viewpoint (for example, a 4 × 4 model view matrix M ^t ). The image generation module 214 is configured to generate an image in which the image of the virtual object is superimposed and drawn on the image of the physical object, and provides a presentation image to the display 201. In this case, as time passes, an image including a positional shift caused by the accumulation of the error of the sensor output is displayed on the display 201.

In this embodiment, it is assumed that the viewpoint position is previously held in the viewpoint position / orientation calculation module 212 as a fixed value. Generally, when the distance to the observation target object (real object, virtual object) in the MR space to be observed is relatively large with respect to the actual movement amount of the viewpoint position, even if there is some error in the viewpoint position. It has the property of not significantly affecting the position shift on the image. In particular, when the object to be observed is present at a long distance such as an outdoor MR system and the observer is stopped at one place, it is effective to set the viewpoint position to a fixed value in this way. Is. Of course, another position sensor (for example, GPS) that measures the viewpoint position of the camera 202
May be further attached to the HMD 200, and the output thereof may be input as the viewpoint position.

In the present embodiment, for the above reason, the error in the viewpoint position is sufficiently small as a relative value,
It is assumed that the positional shift on the image due to the error of the viewpoint position is sufficiently negligible.

Next, FIG. 3 shows the configuration of the image processing apparatus in this embodiment in which the HMD 200 is connected. The same parts as those in FIG. 2 are designated by the same reference numerals, and the description of the same parts as those in FIG.

The image processing apparatus 310 shown in FIG.
In the image processing device 210 shown in FIG.
15 is added, and the viewpoint position / orientation calculation module 2 is further added.
12 has a configuration in which 12 is changed to a viewpoint position / orientation calculation module 312. This correction value calculation module 21
Reference numeral 5 calculates a correction value (correction matrix ΔM ^t ) by a correction value calculation process described later based on the captured image input from the image input module 213 and the viewpoint orientation input from the sensor input module 211, and the viewpoint position / orientation calculation is performed. Output to module 312. The viewpoint position / posture calculation module 312 uses the viewpoint posture of the camera 202 input from the sensor input module 211 and the camera 2 obtained by another method.
02 based on the viewpoint position and the correction value input from the correction value calculation module 215, the attitude correction process described below is executed to correct the position / orientation information (model view matrix M ^t ) calculated based on the sensor output. , The corrected viewpoint position / orientation information (corrected model view matrix M $ ^t ) is generated.

Next, the basic principle of the correction value calculation processing in the correction value calculation module 215 will be described.

The correction value calculation process is basically a reality that can be used as an index for alignment of image features of a landmark (for example, a corner of a building or the roof of a house) imaged on an image. It is performed based on the observation predicted position of the landmark on the image predicted based on the sensor output using the object (a part thereof) and the observed position on the image of the landmark actually detected by the image processing. . Therefore, the most important point of the correction value calculation process is how accurately and stably the observation position of the landmark is detected from the image.

In the present embodiment, landmark detection is performed by template matching using a landmark template image.

Generally, when image features are extracted from an image by template matching, the rotation of the image features on the image plane becomes a problem. The rotation of the image feature occurs when the camera or the object to be photographed rotates in the roll direction in the camera coordinate system. For example, FIG. 14 (a)
When the landmark L is detected from the photographed image I shown in FIG. 14B using the template image T shown in FIG. 14B, when the search processing that does not consider the rotation of the image feature is performed, the land is stably obtained. The mark cannot be detected.
On the other hand, as shown in FIG. 14C, the template image T is rotated (45 in the example in the figure) in consideration of the rotation of the image feature.
By preparing a plurality of template images T ′ to which each rotation is added and performing a search process for each template image, it is possible to detect the landmarks that cope with the rotation of the image features. However, the amount of calculation increases in proportion to the number of templates, which results in processing with a very high calculation load.

In this embodiment, the attitude sensor 203
The posture of the viewpoint of the camera 202 is measured by. As for the value in the azimuth direction among these measured values, there is an accumulation of error with the passage of time as described above. Has been acquired. Therefore, as shown in FIG. 14D, a converted image I ′ obtained by adding rotation to the captured image I is generated based on the rotation angle of the camera 202 in the roll direction measured by the orientation sensor 203, and this image I is generated. By performing the search process with the template image T on the above, it becomes possible to detect the landmark that does not depend on the rotation of the image feature.

Further, in this embodiment, the measured values by the attitude sensor 203 are obtained for the attitudes of the other two axes, and the attitude correction value calculation module 215 in the previous frame performs the processing in the previous frame. Attitude correction value is obtained. Therefore, as shown in FIG. 14E, the rough position p of the landmark on the photographed image I is predicted based on those values, and a region near the predicted position (target image extraction region in the same figure)
By performing the above-described rotation process only on the target image R, the target image R that is the target of the landmark search process is created (FIG. 14F), and the search range can be limited.

Therefore, it becomes possible to perform landmark detection by template matching at high speed and stably.

Next, FIG. 4 shows a specific configuration of the correction value calculation module 215.

The correction value calculation module 215 will be described later.
Image I⁰Template to generate a template image based on
The rate image generation module 403 and the image at time t
Image I ^tAnd the attitude of the camera 202 (roll^t) And based on
Target image generation module
404, using target image and template image
Calculate the degree of similarity and detect the position of the landmark.
Response search module 402, and detected lander
According to the position of the
(Correction matrix ΔM described later^t) And output this
It is configured by the positive value update module 401.

Next, each variable used in this embodiment will be described.

The i-th (i = 1,2,3 ,,) run
Mark is L_i ・ Landmark L_iThe position (known) in world coordinates of
P_i= (X_i, Y_i, Z_i, 1)^T ・ Set the default position of the camera to (X⁰, Y⁰, Z⁰) -The default appearance of the camera used to generate the template image
Power (roll⁰, Pitch⁰, Yaw⁰) ・ Model view matrix at the default position and orientation of the camera
(The conversion matrix from the world coordinate system to the camera coordinate system) is M⁰ ・ The focal length (known) of the camera is f -Projective transformation matrix of camera (from camera coordinate system to image coordinate system
Transformation matrix) (known) to S ・ The captured image in the default position and orientation of the camera is I⁰ ・ Landmark L_iImage I⁰The shooting position above is p
_i ⁰= (X_i ⁰h_i ⁰, Y_i ⁰h_i ⁰, H_i ⁰)^T ・ Landmark L_iTemplate image to search for
To T_i ・ The template image size (default) is N × N ・ The coordinate range of the template image is xs_T, Xe_T, Y
s_T, Ye_T(However, xs_T= Ys_T= -N / 2 decimal
The value with the part cut off. xe_T= Ye_T= Xs_T+ N-1) ・ The image taken at time t is I^t ・ Attitude measurement value by the sensor at time t (roll
^t, Pitch^t, Yaw^t) ・ Posture measurement value (roll^t, Pitch^t, Yaw^t)
Model view matrix calculated from (from the world coordinate system to the camera
Transformation matrix to coordinate system) M^t ・ Image I^tLandmark L above_iPredicted position of
P_i ^t= (X_i ^th _i ^t, Y_i ^th_i ^t, H_i ^t)^T ・ Image I^tLandmark L actually detected above_iA picture of
Image position is p $_i ^t= (X $_i ^t, Y $_i ^t) ・ Image I^tFrom Landmark L_iSearch to detect
R is the target image to be processed_i ^t ・ Landmark search range in the x direction (default) is ± m ・ Landmark search range in the y direction (default) is ± n -The size of the target image is N'xN "(where N '=
N + 2m, N ″ = N + 2n) ・ The range of coordinates of the target image is xs_R, Xe_R, Ys
_R, Ye_R(However, xs_R= Xs_T-M, xe_R= Xe
_T+ M, ys_R= Ys_T-N, ye_R= Ye_T+ N) ・ R on the target image_i ^tLandmark L in_iof
Let the detected coordinates be (j_i ^t, K_i ^t) ・ Detected coordinates of each landmark (j_i ^t, K_i ^t) Representative
The value is (j^t, K^t) -Corrected update value of camera attitude calculated at time t
Δroll, Δpitch, Δyaw -Correction update value of camera position calculated at time t
Δx, Δy, Δz -Model view matrix M calculated at time t^tSupplement
The correction matrix to correct is ΔM^t -Already calculated by the processing so far (at time t-1
The correction matrix (calculated) is ΔM^t-1 ・ Correction matrix ΔM^t-1Correction matrix ΔM^tTo update to
Correction update matrix ΔM ′ of ^t ・ M^tCorrection matrix ΔM^tCorrected model corrected by
Lubu Matrix is M $^t ・ M^tCorrection matrix ΔM^t-1Corrected by
Let the model view matrix be M '^t Based on the above settings, the posture measurement error in this embodiment
The difference correction processing will be described below according to the flow of processing.
It

<Creation of Template Image> First, the camera for photographing the real space is set to a predetermined position and orientation, and the image I ₀ is set.
To shoot. FIG. 1 shows an example of the image I ₀ . L _{1 to} L ₄ in the figure are landmarks, and the frame portions indicated by T _{1 to} T ₄ are regions extracted as template images corresponding to the respective landmarks.

Next, the model view matrix M ⁰ is calculated. Since the calculation method for calculating the model view matrix from the position and orientation of the camera is a known method, the description thereof is omitted here.

Further, p _i ⁰ is calculated for each landmark (L _{1 to} L _{4 in} FIG. 1) in the photographed image by the following formula.

P _i ⁰ = SM ⁰ P _i and then for each landmark the template image T
_i (the image of the portion indicated by T _{1 to} T ₄ in FIG. 1) is created by the method described below.

When roll ⁰ is 0, a rectangular area of size N × N centering on (x _i ⁰ , y _i ⁰ ) is extracted from image I ₀ and used as a template image T _i. . In the template image T _i , if the center of the image is represented by coordinates (0,0), this process can be described as follows.

T _i (j, k) = I ⁰ (x _i ⁰ + j, y _i ⁰ + k) where j = xs _{T to} xe _T and k = ys _{T to} yes _T.

On the other hand, when roll ⁰ is not 0, (x
_i ⁰ , y _i ⁰ ) as the center, and an N × N rectangular area is -r
Extract a rectangular region rotated by ol ⁰ . That is, for each pixel of j = xs _{T to} xe _T and k = ys _{T to} yes _T , T _i (j, k) = I ⁰ (x _i ⁰ + jcos (−roll).
⁰ ) -ksin (-roll ⁰ ), y _i ⁰ + jsin
A template image T _i such that (−roll ⁰ ) + kcos (−roll ⁰ )) is created.

<Model view matrix M at each time^tof
Calculation> Sensor output at time t (posture (roll^t，
pitch^t, Yaw^t)) And the default position of the camera
(X⁰, Y⁰, Z⁰) Based on the model view matrix M
^tTo calculate. Model view row from camera position and orientation
Since the calculation method for calculating the columns is a known method, here
Is omitted.

<Correction Value Calculation Processing: Model View Matrix M ^t
Calculation of Correction Matrix ΔM ^t for Correcting> Model View Matrix M
a description will be given of a method of calculating the correction matrix ΔM ^t to correct a ^t.

First, the model view matrix M ^t is corrected by using the correction matrix ΔM ^t−1 already calculated in the processing so far to obtain M ′ ^t . If this process is the first (t
= 0), the correction matrix ΔM ^t−1 is a unit matrix.

M ′ ^t = ΔM ^t−1 M ^t Next, p _i ^t is calculated for each landmark according to the following formula.

P _i ^t = SM ′ ^t P _{i Since} this method is a known method, detailed description will be omitted. Also, landmarks result of obtaining the coordinates of each landmark, the coordinates are outside the range of the coordinates of the image I ^t are excluded from further processing.

Next, create a target image _R ^{i t} for each landmark. Specifically, the image ^{I t} is a local coordinates in the image ^{_{(x i t, y i t}} ) and extracts a rectangle rotated by -Roll ^t the rectangular area N '× N "around the. That is, j = xs _{T to} xe _T , k
The following conversion is performed for each pixel of = ys _{T to} yes _T.

R_i ^t(J, k) = I^t(X_i ^t+ Jco
s (-roll^t) -Ksin (-roll^t), Y_i
^t+ Jsin (-roll^t) + Kcos (-roll
^t)) Next, for each landmark, the target image R_i ^tAnd ten
Plate image T_iMatch the target image
The position of the landmark above (j_i ^t, K_i ^t)
Meru. The specific processing of the method to obtain is described below.
It

[0048] First, the target image _R ^{i t} on the coordinates (j,
The similarity e (j, k) between the N × N rectangular area centered at k) and the template image T _i is calculated. The calculation of the degree of similarity is performed by, for example, cross-correlation or SSD (Sum of Square).
d Difference) and the like, but any known template matching method may be used. This similarity e (j, k) is calculated for all j and k (where j = -m to m, k = -n to n) to maximize the similarity e (j, k). Let k be (j _i ^t , k
_i ^t ).

Then, it was calculated for each landmark.
(J_i ^t, K_i ^t) From the representative value (j^t, K ^t) Is calculated
To do. The representative value is calculated for each landmark, for example.
Metta (j_i ^t, K_i ^t) Average value and median value
By doing. It should be noted that it was calculated for each landmark (j_i
^t, K_i ^t), The degree of similarity e at the time of detection
(J_i ^t, K_i ^t) Is greater than a predetermined threshold
Reliability is low because only the peaks are used to calculate the representative value.
The detection result can be excluded. In this case, the similarity e
(J_i ^t, K_i ^t) Is a landmark above the threshold
If the number of is less than or equal to a predetermined number, the correction value at time t
The arithmetic processing may be terminated.

Then, the correction matrix ΔM ^t is updated based on the landmark detection result.

First, the corrected update value Δroll of the attitude of the camera
l, Δpitch, Δyaw are obtained as follows.

Δroll = 0 Δpitch = arctan (k ^t / f) Δyaw = arctan (j ^t / f) Since it is assumed that the position of the camera is fixed, all correction update values Δx, Δy, and Δz of the position are It becomes 0.

Next, the above postures Δroll, Δpitc
As a model view matrix defined by h, Δyaw, and positions Δx, Δy, Δz, a correction update matrix ΔM ′ ^t
To calculate. Since the calculation method for calculating the model view matrix from the position and orientation of the camera is a known method, the description thereof is omitted here.

Then, the correction matrix ΔM obtained so far
^An updated correction matrix ΔM ^t is calculated from ^t−1 and the correction update matrix ΔM ′ ^t according to the following formula.

ΔM ^t = ΔM ′ ^t ΔM ^t−1 <position / orientation correction process: corrected model view matrix M $ ^t
Calculation> Model view matrix M $ after correction at time t
^t can be calculated according to the following equation.

By drawing and displaying CG using M $ ^t = ΔM ^t M ^t and the corrected model view matrix M $ ^t , the positional deviation in the azimuth direction with the passage of time can be obtained even with the gyro sensor. Can be reduced.

Flowcharts of the correction processing in this embodiment described above are shown in FIGS. 5 to 7, and will be described below.

FIG. 5 is a flow chart of the main processing of the above-mentioned correction processing.

First, a template image is created (step S501). FIG. 6 shows a flowchart of a specific process of creating a template image.

First, the image I ⁰ is input from the camera fixed in the predetermined position and orientation (step S601). Next, the model view matrix M ⁰ is calculated based on the position and orientation of the camera at this time (step S602). Then for all i (in other words, for all landmarks) p
_i ⁰ is obtained (steps S603 and S604). Next, a template image is created. The method of creating is as described above, and pixel values are calculated for all j and k within the range described above for each landmark, and the template image T
_It is stored in the coordinate (j, k) of _i (steps S606 to S608).

[0061] After generating the template image according to the process shown in FIG. 6 above, returning to FIG. 5, to photograph an image ^{I t} (step S502). Further, the sensor output at this time is also acquired (step S503). Incidentally, step S5
The order of the processing of 02 and S503 is not limited to this, and the order may be reversed, or they may be synchronized and performed simultaneously.

Next, the model view matrix M ^t is calculated based on the sensor output (step S504). Then, the correction matrix ΔM ^t is calculated (step S505). A flowchart of a specific process in calculating the correction matrix ΔM ^t is shown in FIG. 7 and will be described below.

First, the model view matrix M ^t is corrected with the correction matrix ΔM ^t−1 to obtain the model view matrix M ′ ^t (step S701). And then for all i,
In other words, p _i ^t is calculated for all landmarks (steps S702 and S703). The calculated p
To be eligible for processing described later if _{the i} ^t is outside the range of the image I ^t.

Next, the target image R for each landmark
calculating a _i ^t (step S704 to S706). And performs matching of the target image _R ^{i t} and the template image _{T i,} each j, for each k similarity e (j, k) is calculated (step S707, S708). Then, (j, k) that maximizes the similarity e (j, k) is (j _i ^t , k).
_i ^t ) (step S709). Step S above
The processing from 707 to step S709 is calculated for all i, in other words, for all landmarks (step S710).

Then, the average value of the obtained (j _i ^t , k _i ^t ) is calculated to calculate (j ^t , k ^t ) (step S71).
1). Further, the correction value of the position and orientation of the camera is obtained (step S712), the correction update matrix ΔM ′ ^t is obtained (step S713), and the correction matrix ΔM ^t is finally obtained (step S714).

When the correction matrix ΔM ^t is calculated according to the processing shown in FIG. 7, the model view matrix M ^t is corrected using the calculated correction matrix ΔM ^t (step S506).

Then, the CG is drawn and displayed using the corrected model view matrix M $ ^t (step S507).

As described above, with the image processing apparatus and method according to the present embodiment, it is possible to correct the posture measurement error of the camera viewpoint by the posture sensor and realize MR without misalignment.

[Second Embodiment] In the first embodiment,
The correction process is performed in a single loop (drawing loop). In this case, the drawing frame rate cannot be sufficiently obtained due to the calculation load of the image processing. Alternatively, if the image processing is made simple (with a small amount of calculation) in order to secure the drawing frame rate, it is not possible to obtain sufficient accuracy of correction.

Therefore, in this embodiment, the drawing loop and the correction calculation loop are separated and operated at independent update cycles (for example, the drawing loop is 60 Hz and the correction calculation loop is 1 loop / second). Further, the image processing apparatus used in the first embodiment is used as an apparatus that executes the processing of this embodiment.

<Drawing Loop> Basically, the processing according to the flowcharts shown in FIGS. 5 and 6 is executed, but in step S505, the latest correction matrix ΔM ^s transmitted from the correction calculation loop described later is obtained. This is set to ΔM ^t .

<Correction Calculation Loop> FIG. 8 shows a flowchart of the processing of the correction calculation loop. First, the image I ^{s at} time s and the model view matrix M ^s at that time are input from the drawing loop (step S801). Then, the correction matrix ΔM ^s is calculated in the same manner as the processing in step S505 described in the first embodiment (step S80).
2). Then, the calculated correction matrix ΔM ^s is transmitted to the drawing loop (step S803). Then, the above processing is executed until the end is permitted (step S804).

In the present embodiment, the drawing loop and the correction calculation loop are divided and executed in one image processing apparatus (for example), but the present invention is not limited to this, and the processing of each loop is performed individually. You can run it on your computer.
Then, the respective computers are set in a communicable state, and the respective processing results can be transmitted and received between the respective computers. By doing so, the number of processes handled by one computer is reduced, and thus more rapid processes are possible.

[Third Embodiment] In the second embodiment, in the process of correcting the model view matrix, the correction is performed by a simple product operation of the obtained correction matrix ΔM ^t and the model view matrix M ^t by the sensor. Although the subsequent model view matrix M $ ^t is obtained, the correction matrix does not always represent the correction information appropriate for the current frame (time t) because the correction matrix is updated with a gap compared to the drawing cycle. I can't say that

Therefore, in this embodiment, in step S505 in the second embodiment, the correction matrix ΔM ^t suitable for the time ^t is calculated using the past correction matrix obtained from the correction calculation loop.

First, the correction matrix Δ obtained at time s
Expanding M ^s , the correction value Δya in the azimuth direction of the camera posture
to calculate the w ^s and the pitch direction of the correction value Δpitch ^t. Since a method of obtaining individual rotation components from the model view matrix is known, description thereof will be omitted here. Time s-
Also performs the same processing in one, the correction value of the camera posture at time ^t Δyaw t and Derutapitch ^t
Is calculated as follows.

Δyaw ^t = Δyaw ^s + (Δyaw ^s −
Δyaw ^s−1 ) × Δst / Δs Δpitch ^t = Δpitch ^s + (Δpitch ^s −
Δpitch ^s−1 ) × Δst / Δs where Δst is the elapsed time from time s to time t, Δ
Let s represent the elapsed time from time s-1 to time s.

Then, the obtained correction value Δyaw^tAnd Δp
itch^tUsing the correction matrix ΔM ^tAsk for. That conclusion
As a result, the correction matrix calculation method in this embodiment is applied.
By doing so, correction appropriate for the current frame (time t)
The matrix can be calculated.

In this embodiment, the correction value is extrapolated by the linear prediction of the first order as shown in the above equation, but the method of predicting the correction value is not limited to this, and the method of predicting the second order is not limited to this. It is also possible to use linear prediction or other prediction methods.

[Fourth Embodiment] In the present embodiment, the first
A method of performing the correction more accurately than the embodiment of FIG.

First, of the variables used in this embodiment,
Differences from the above embodiment will be described.

A mode based on the sensor output at time t
Rotation component R of the Delview matrix^t · The model view matrix flatness based on the camera's default position
Line shift component T^t ・ Image I^tLandmark L above_iDetection position p
$_i ^t= (X $_i ^t, Y $_i ^t) ・ Landmark L_i"Image I^tTurtle on projected point "
Position pc in La coordinate system_i ^t ・ Landmark L_iModel view matrix obtained from
Correction update matrix (rotation component in azimuth direction) ΔR_i’^t ・ Landmark L_iCorrection in the yaw direction obtained from
Update value Δyaw_i ^t ・ Yaw direction correction required from all landmarks
Update value Δyaw^t ・ Correction matrix of model view matrix (rotation component in azimuth direction)
  ΔR^t -Correction matrix ΔR already calculated in the processing so far
^t-1(Identity matrix in the first loop) ・ Correction matrix ΔR^t-1Model view corrected by
Matrix rotation component R ' ^t ・ Correction matrix ΔR^t-1Model view corrected by
Matrix M ’^t ・ Correction matrix ΔR^t-1Correction matrix ΔR^tTo update to
Correction update matrix (rotation component in azimuth direction) ΔR ′^t Based on the above settings, the correction method in the present embodiment
9 and 10 showing a flowchart of processing of the same method.
Will be explained.

FIG. 9 is a flowchart of the main processing in this embodiment. Since the processing from step S901 to step S903 is the same as the processing from step S501 to step S503 in the first embodiment, description thereof will be omitted.

Next, the rotation component R of the model view matrix
^t and the translation component T ^t are calculated (step S90).
4). Specifically, the rotation component R ^t is the sensor output (the posture of the camera obtained from the sensor) (roll ^t , pitch ^t ,
It is determined by a known method based on yaw ^t ). On the other hand, the translation component T ^t is obtained by a known method based on the viewpoint position of the camera.

Then, the correction matrix ΔR ^t is obtained (step S905). A flow chart of a specific process for obtaining the correction matrix ΔR ^t is shown in FIG. 10 and will be described below.

[0086] First, so far the process already matrix R ^t in the correction matrix [Delta] R ^t-1, which is calculated by the corrected as follows, obtaining the matrix R ^'t.

R ′ ^t = R ^t ΔR ^t−1 Next, the matrix M ′ ^t is obtained as follows using the obtained matrix R ′ ^t (step S1001).

M ′ ^t = R ′ ^t T ^{t Since} the processing from step S1002 to step S1010 is the same as the processing from step S702 to step S710, respectively, description thereof will be omitted here.

[0089] Next, Motoma' was _{^{(j i t, k i t}} ) by using the image ^I position of each landmark on the _{^{t p $ i t = (x}} $
_i ^t , y $ _i ^t ) is calculated (step S1012).
The calculation is performed by the following formula.

[0090] _{^{x $ i t = x i t}} + j i t cos (-ro
ll ^t ) −k _i ^t sin (−roll ^t ) y $ _i ^t = y _i ^t + j _i ^t sin (−roll ^t ) + k
_i ^t cos _(-roll ^t) and then "projection point onto the image ^{I t"} of each landmark
Calculating a position _pc ^{i t} in the camera coordinate system (step S1013).

[0091] pc_i ^t= (X $_i ^t, Y $_i ^t, -F, 1)^T At this time, if a is a scaling parameter, pc
_i ^t・ A = R '^tΔR _i’^tT^tP_iIs established. this
By solving the equation, Δyaw_i ^tTo calculate. How to
It is shown below. However, in the following, Inv (M) is the matrix M
The inverse matrix is shown.

P $_i ^t= (X $_i ^t, Y $_i ^t, Z $_i
^t, 1)^T= Inv (R '^t) Pc _i ^t P ’_i= (X '_i, Y ’_i, Z '_i, 1) = T^tP_i Putting it, P $_i ^t= ΔR_i’^tP ’_i/ A
so, X $_i ^t= {Cos (Δyaw_i ^t) X '_i-Sin
(Δyaw_i ^t) Z '_i} / A Z $_i ^t= {Sin (Δyaw_i ^t) X '_i+ Cos
(Δyaw_i ^t) Z '_i} / A By solving this, Δyaw_i ^t= Arctan {(Z $_i ^t・ X '_i-X
$_i ^t・ Z '_i) / (X $_i ^t・ X '_i+ Z $_i ^t・
Z '_i)} (Step S1014). This step S101
4 processing for all i, that is, all landmar
(Step S1015). And asked
All Δyaw_i ^tAverage value Δyaw^tAsk for
(Step S1016).

Then, the correction update matrix ΔR ′ ^t is obtained using the obtained correction update value Δyaw ^t (step S101).
7). A method of calculating the model view matrix for rotating the coordinate system in the azimuth direction at an arbitrary angle (here, Δyaw ^t ) is known, and therefore its description is omitted. This correction update matrix Δ
The correction matrix ΔR ^t to be calculated is calculated using R ′ ^t as follows (step S1018).

[0094] After calculating the correction matrix [Delta] R ^t according to the process shown in ^{ΔR t = ΔR t-1 ΔR} 't above 10, returning to FIG. 9, the model view matrix by using the calculated correction matrix [Delta] R ^t M $ ^t is calculated (step S9)
06). The calculation is performed according to the following formula.

M $ ^t = R ^t ΔR ^t T ^t Then, similarly to the first embodiment, the CG is drawn and displayed using the calculated model view matrix (step S90).
7).

[Fifth Embodiment] In the first to fourth embodiments, the position of the viewpoint is known, and only the posture (direction, angle) is corrected. As described above, if the distance to the observation target object is relatively large with respect to the amount of movement of the viewpoint position, it is effective to set the viewpoint position to a fixed value, but if that assumption does not hold, Positional displacement occurs due to movement. Therefore, in this embodiment, a method of correcting the viewpoint position will be described. However, in the present embodiment, it is assumed that the movement amount ΔTz in the Z-axis direction (depth direction, direction perpendicular to the imaging surface) in the camera coordinate system is always 0. Further, it is assumed that the sensor obtains a correct value for the rotation component. If this assumption holds, the position can be corrected by detecting at least one landmark. Here, the setting in the present embodiment is shown below.

Rotation component of the model view matrix based on the sensor output at time t R ^t · Translation component of the model view matrix based on the default position of the camera at time t T ^t · Correction matrix of the model view matrix (world Translation component in coordinate system) ΔT ^t · Correction matrix of model view matrix obtained from landmark L _i (translation component in world coordinate system) ΔT _i
^t -correction matrix ΔT already calculated in the processing so far
^{t-1 (start} unit matrix in the loop) and correction matrix [Delta] T ^t-1 'corrected model view matrix M by ^t · correction matrix [Delta] T ^{t-1' t} · Model translation component T of the corrected model view matrix by Correction update matrix of view matrix (translation component in camera coordinate system) ΔTc ^t · Landmark L _i , correction update value in x-axis direction (camera coordinate system) ΔTx _i ^t · Landmark L _i , determined from corrected and updated value .DELTA.Ty i _t ^· all landmarks in the y-axis direction (camera coordinate system) is determined from the corrected and updated value Delta] Tx t ^· all landmarks in the x-axis direction (camera coordinate system), the y-axis direction 11 and 12 showing the flowchart of the processing of the correction method in the present embodiment based on the setting of the correction update value ΔTy ^t or more of (camera coordinate system).
Will be explained.

FIG. 11 is a flowchart of the main processing in this embodiment. The processing from step S1101 to step S1104 is the same as each processing from step S901 to step S904 in the fourth embodiment, and thus the description thereof is omitted here.

Next, the correction matrix ΔT ^t is obtained (step S1105). A flowchart of a specific process for obtaining the correction matrix ΔT ^t is shown in FIG. 12 and will be described below.

First, the matrix T ^t is corrected by the correction matrix ΔT ^t−1 already calculated in the processing so far, and the matrix T ′ ^t is calculated.
And matrix M ′ ^t are obtained as follows (step S1
201).

T ′ ^t = ΔT ^t−1 T ^t M ′ ^t = R ^t T ′ ^t Each process from the next step S1202 to step S1211 is step S1002 in the fourth embodiment.
Since each process from step S1012 to step S1012 is the same, the description thereof is omitted here.

Next, in step S1212, the land marker
Black L_iUpdate value ΔTx for_i ^t, ΔTy_i ^tCalculate
Put out.

ΔTx _i ^t = f · Zc _i ^t (x $ _i ^t −x _i ^t ) ΔTy _i ^t = f · Zc _i ^t (y $ _i ^t −y _i ^t ) where Zc _i ^t is the camera coordinate system. At the z coordinate of the landmark at, the third component of M ′ ^t P _i has that value.

Correction correction value ΔTx above_i ^t, ΔTy_i ^t
For all i, that is, for all landmarks
Obtained (step S1213), then all corrections obtained
Update value ΔTx_i ^t, ΔTy_i ^tAverage value ΔTx^t, ΔT
y^tIs calculated (step S1214). And asked
Average value of correction update value ΔTx^t, ΔTy^tUsing x direction
Towards ΔTx^t, Ty in the y direction^tTo the coordinate system
Correction update matrix ΔTc to be applied^tIs calculated (step S12
15). A coordinate transformation matrix that applies arbitrary parallel movement to the coordinate system
The method of calculation is publicly known, and the description thereof is omitted. That
Correction matrix ΔT ^tIs calculated as follows (step
S1216).

[0105] After calculating the ^{ΔT t = Inv (R t)} ΔTc t R t ΔT t-1 or more correction matrix [Delta] T ^t in accordance with the processing shown in FIG. 12, referring back to FIG. 11, using the calculated correction matrix [Delta] T ^t To calculate the model view matrix M $ ^t (step S
1106). The calculation is performed according to the following formula.

M $ ^t = R ^t ΔT ^t T ^t Then, as in the first embodiment, the CG is drawn and displayed using the calculated model view matrix (step S11).
07).

[Sixth Embodiment] In the fifth embodiment, Δ
Since Tz was always assumed to be 0, accurate alignment could not be performed when the viewpoint moved forward and backward with respect to the line-of-sight direction. In the present embodiment, two or more landmarks are always observed to cope with the case where ΔTz is not zero.

The flowchart of the correction processing in this embodiment is basically the same as that in the fifth embodiment shown in FIGS.
However, the contents of the processing in steps S1214 and S1215 are different. Hereinafter, each processing in steps S1214 and S1215 in the correction processing in the present embodiment will be described.

Letting ΔTx ^t , ΔTy ^t , and ΔTz ^t be the correction update values of the camera coordinate system in the x-, y-, and z-axis directions, respectively.
Landmark imaging predicted position p _i ^t and detection position p $ _i ^t
In between, the following formula holds for each landmark.

[0110] _{^{ΔTx t + x $ i t ·}} f · ΔTz t = f ·
_{^{Zc i t (x $ i t}} -x i t) ΔTy t + y $ i t · f · ΔTz t = f · Zc i t (y
$ _I ^t -y _i ^t) Thus, built the following simultaneous equations for a plurality of landmarks, by solving this, unknown corrected and updated value ΔT
x ^t , ΔTy ^t , and ΔTz ^t are calculated (step S12).
14).

[0111]

[Equation 1]

Then, the calculated ΔTx^t, ΔTy^t, ΔT
z^tUsing the correction update matrix ΔTc ^tBy a known method
(Step S1215). And step S121
Update matrix ΔTc obtained in 6^tUsing Fifth Embodiment
Correction matrix ΔT^tAsk for.

[Seventh Embodiment] In the first to sixth embodiments, only rotation or parallel movement can be corrected. In this embodiment, both corrections are performed. As a basic method, after the rotation is corrected, the parallel movement is corrected. However, the present invention is not limited to this, and the correction may be performed in the reverse order, or the parallel movement correction (or the reverse order may be performed) after the rotation correction may be repeated a predetermined number of times. It may be repeated until the error becomes smaller than the set threshold value or the fluctuation of the error due to the correction becomes smaller than the threshold value.

The settings used in this embodiment will be described below.

The rotation component R ″ ^{t of} the model view matrix corrected by the correction matrix obtained in the intermediate stage of processing R ″ ^t · The model view matrix M ″ ^t corrected by the correction matrix obtained in the intermediate stage of processing The correction process in the present embodiment will be described based on the settings.

FIG. 13 shows a flowchart of the main processing in this embodiment. The flowchart shown in the figure is a flowchart obtained by adding a process (step S1306) for calculating the correction matrix ΔT ^t to the flowchart shown in FIG. 9 of the fourth embodiment, and calculates the correction matrix ΔR ^t . The processing performed (step S1305) is also different. Hereinafter, a process of calculating the correction matrix ΔT ^t (step S1306) and a process of calculating the correction matrix ΔR ^t (step S1305) in the present embodiment will be described. Descriptions of other parts are omitted.

Correction matrix ΔR in step S1305
The flow chart of the specific process for calculating ^t is basically the same as that of FIG. 10 of the fourth embodiment, but in this embodiment, the matrix R ′ ^t and M ′ ^t are calculated in step S1001. The addition matrix T ′ ^t is calculated.

R '^t= R^tΔR^t-1 T ’^t= ΔT^t-1T^t M ’^t= R '^tT ’^t Then, in the subsequent processing (for example, S1014), the
Fixed value T at 10 ^tAs an alternative to the derived T ′
^tTo use.

On the other hand, the flow chart of the concrete processing for correcting the correction matrix ΔT ^t in step S1306 is basically the same as that of FIG. 12 of the fifth embodiment,
In this embodiment, in step S1201, the matrix ^{R t} using the correction matrix [Delta] R ^t obtained in step S1305
Is corrected and the matrices R ″ ^t and M ″ ^t are obtained according to the following equations.

R ″ ^t = R ^t ΔR ^t M ″ ^t = R ″ ^t T ′ ^{t Further} , the processing in the present embodiment is the same as ΔTx _i ^t , ΔTy _{i in} step S1212 in the flowchart of FIG.
^t is ^calculated as follows.

ΔTx _i ^t = f · Zc _i ^t (x $ _i ^t −x _i ^t ) ΔTy _i ^t = f · Zc _i ^t (y $ _i ^t −y _i ^t ) where Zc _i ^t is the camera coordinate system. At the z coordinate of the landmark at, the third component of M ″ ^t P _i has that value.

In the process of this embodiment, the correction matrix ΔT ^t is calculated in step S1216 in the flowchart of FIG. 12, but it is calculated according to the following formula.

[0123] [Delta] T ^t = the ^{Inv (R "t) ΔTc t} R" t ΔT t-1 and more correction matrix [Delta] T ^t is completed, the process returns to the flowchart shown in FIG. 13, the model view matrix M ^{$ t} in step S1307 Is calculated as follows.

M $ ^t = R ^t ΔR ^t ΔT ^t T ^{t Further} , the above-described processing for obtaining the correction matrices ΔR ^t and ΔT ^t (steps S1305 and S1306) may be repeated a predetermined number of times as described above.

[Eighth Embodiment] In the first to seventh embodiments, the position of the landmark in the world coordinate system is known, but other methods can be adopted. That is, to the position of the landmark on the image I ⁰ may be specified directly in the initial position and orientation, the image I ⁰ at the initial position and orientation
It is also possible to extract a feature point having an image feature (for example, an edge portion or a portion having a strong texture) that is remarkable (easy to track) from above and use this position as the position of the landmark.

Here, consider a case where the image feature imaged at the image coordinates (x _i ⁰ , y _i ⁰ ) is designated or detected by manual input or image processing and used as the landmark L _i . The camera coordinates of the landmark Pc _i
Assuming that ⁰ = (x _i ⁰ , y _i ⁰ , f, 1), the world coordinates are P _i = Inv (M ⁰ ) Pc _i ⁰ using the inverse matrix of the model view matrix M ⁰ in the initial position and orientation. Can be defined, and the methods described in the first to third embodiments can be applied as they are.

However, since the depth information of the landmark position cannot be obtained, the correction using the depth information of the landmark position (the position correction described in the fifth and subsequent embodiments) cannot be performed. Absent.

[Modification 1] In the above embodiment, M
Although the posture (or position / orientation) of the viewpoint of the camera in the R system is measured, the applicable range of the present invention is not limited to this, and the posture (or position / orientation) of the viewpoint of the camera is not limited to this.
It goes without saying that it can also be used for any purpose of measuring.

[Modification 2] In the above embodiment, the posture (or position / orientation) of the viewpoint is measured in the video see-through MR system. However, even in the optical see-through MR system, the present invention is used. The attitude (or position / orientation) can be measured by the image processing device. In this case, the posture sensor is attached to the HMD, and the HMD is fixed so as to be fixed at a position where the relative posture (or position / posture) relationship with the viewpoint position of the observer, which is the measurement target, is known. Attach the camera to. Then, the attitude (or position / orientation) of the camera is calculated by a method similar to that of the above-described embodiment, and the value is further converted to calculate the attitude (or position / orientation) of the observer's viewpoint. Further, the applicable range of the present invention is not limited to the measurement target object, and the posture (or position / orientation) of any measurement target object is similarly measured by mounting the camera and the posture sensor. can do.

[Modification 3] In the above embodiment, the template image generation module 430 generated the template image based on the image I ⁰ taken in the predetermined position and orientation. However, the template image is not necessarily the image I ^0.
The template image may not be generated based on the above, and a template image stored in advance may be used, or a template image obtained by any known method such as a dynamic template updating method may be used.

[Modification 4] In the above embodiment, the target image generation module 404 extracts only the region near the predicted position of the landmark as the target image. However, the target image of template matching (that is, the target image) is It is not always necessary to extract a region near the predicted position of the landmark. For example, as shown in FIG. 14D, an image I ′ obtained by rotating the entire input image is set as a target image common to each landmark, and the predicted position of each landmark is calculated in the image I ′. The correspondence search may be performed in the vicinity, or the correspondence search may be performed on the entire area of the image I ′.

[Fifth Modification] In the above embodiment, landmark detection by template matching is used as a means for measuring the posture or position / posture, but the landmark detection method in the image processing apparatus of the present invention is not limited to this. As long as the purpose of detecting the position of the landmark in the image by template matching is not limited to the measurement of the posture or position / posture, it can be applied.

[Other Embodiments] An object of the present invention is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to execute the system or apparatus. It is needless to say that this is also achieved by the computer (or CPU or MPU) of the device reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. Do some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

Further, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted in the computer or the function expansion unit connected to the computer, based on the instruction of the program code, Needless to say, this also includes a case where a CPU or the like included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the above-mentioned storage medium, the storage medium stores the program code corresponding to the above-described (the flowchart shown in at least one of FIGS. 5 to 13). become.

[0136]

As described above, according to the present invention, it is possible to correct the measurement error of the camera viewpoint by the attitude sensor, especially the correction of the accumulation error in the azimuth direction which occurs with the passage of time, and the MR having no displacement can be obtained. Can be realized.

[Brief description of drawings]

FIG. 1 is a diagram showing an initial image I ⁰ .

FIG. 2 is a diagram showing a configuration of a conventional image processing apparatus.

FIG. 3 is a diagram showing a configuration of an image processing apparatus according to the first embodiment of the present invention.

FIG. 4 is a diagram showing a specific configuration of a viewpoint position / orientation correction value calculation module 215.

FIG. 5 is a flowchart of main processing according to the first embodiment of the present invention.

FIG. 6 is a flowchart of a specific process when creating a template image.

FIG. 7 is a flowchart of a specific process in calculating a correction matrix ΔM ^t .

FIG. 8 is a flowchart of processing of a correction calculation loop according to the second embodiment of the present invention.

FIG. 9 is a flowchart of main processing according to the fourth embodiment of the present invention.

FIG. 10 is a flowchart of specific processing for obtaining a correction matrix ΔR ^t .

FIG. 11 is a flowchart of main processing in the fifth embodiment of the present invention.

FIG. 12 is a flowchart of specific processing for obtaining a correction matrix ΔT ^t .

FIG. 13 is a flowchart of main processing according to the seventh embodiment of the present invention.

FIG. 14 is a diagram illustrating a basic principle of template matching in the present invention.

   ─────────────────────────────────────────────────── ─── Continued front page    F term (reference) 2F065 AA04 AA20 AA31 BB05 BB15                       CC14 CC16 DD03 EE00 FF04                       FF26 FF65 FF67 JJ03 QQ25                       QQ31 QQ38 QQ42 RR07 RR09                       UU05                 5C054 AA05 DA01 FC12 FE14 GB16                 5L096 CA02 FA67 FA69 HA08 JA03                       JA09

Claims (4)

[Claims] 1. An imaging device that acquires a captured image including a measurement target, a posture sensor that measures a posture of an imaging viewpoint of the imaging device, and a posture and / or a position of the measurement target as an output of the posture sensor. Based on the detection position of the index detected by the detection unit, a storage unit that stores calculation information to be calculated based on, a detection unit that detects the position of the index in the captured image captured by the imaging device And update means for updating the calculation information stored in the storage means, and calculate the posture and / or position of the measurement target based on the measurement value and the calculation information updated by the update means. An image processing apparatus comprising: a calculating unit, wherein the updating unit updates position information in three directions in a camera coordinate system of the imaging device. 2. A captured image from an imaging device is input, a measurement result from an orientation sensor that measures the orientation of the imaging viewpoint of the imaging device is input, and the index of the index in the captured image captured by the imaging device is input. An image processing method, characterized in that the position / orientation of the image pickup device is obtained based on the measurement result and the detected position of the index detected by the detection means. 3. A program for causing a computer to execute the image processing method according to claim 2. 4. A computer-readable storage medium storing the program according to claim 3.