JP2003203252A - Image display device and its method and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To easily acquire correction information for converting the attitude of a sensor in a sensor coordinate into the attitude of a measuring object in a world coordinate. <P>SOLUTION: First of all, an image generation part 150' renders a virtual object based on the attitude determined beforehand and outputs it on a display screen 110 in a correction information calculation mode. A user adjusts the attitude a head part so that the displayed virtual object has a correct position relation with a real space. A correction information calculation part 310 calculates unknown correction information following the output of an attitude sensor 120 at the point of time when the correct position relation is attained, and stores the information in a memory 140'. Then, in the case where position displacement occurs between the virtual object and the real space when the attitude of the head part is changed in a normal mode, a correction information modification part 400 modifies simultaneously RySV and RyTW which are uncertain correction information by input of a modification command by a modification information input part 410, and outputs the result to a memory 140'. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image display device and method for displaying an image of a virtual space fused with a real space based on an output value of a posture sensor. More specifically, a superimposed image is generated by superimposing an image of the virtual space on the real image captured by the image capturing device, and the superimposed image is displayed on the display screen, or the real space is optically transmitted to the display screen. Meanwhile, the present invention relates to an image display device that displays an image of a virtual space on the display screen, a method thereof, and a storage medium.

[0002]

2. Description of the Related Art In recent years, mixed reality (hereinafter referred to as "MR" (Mi
xed Reality)) is being actively researched. In the MR, an image of a virtual space (for example, a virtual object drawn by computer graphics (hereinafter, referred to as CG), character information, etc.) is superimposed on an image of a real space shot by a shooting device such as a video camera. There are a video see-through method for displaying and an optical see-through method for superimposing an image in a virtual space on the display screen while optically transmitting the real space on the display screen of the display device.

Such applications of MR include navigation for displaying the name and guidance on a building or the like in an actual city, and C for a building to be built at a planned construction site of the building.
New fields, such as landscape simulation in which G images are superimposed and displayed, are qualitatively different from the conventional virtual reality are expected. A common requirement for these applications is how to perform accurate alignment between the real space and the virtual space, and many efforts have been made in the past.

The problem of registration in MR of the video see-through method is consequent to the problem of obtaining the three-dimensional position and orientation of the viewpoint of the photographing device in the world coordinate system set in the real space (hereinafter simply referred to as the world coordinate system). It Further, the problem of alignment in the MR of the optical see-through method can be said to be a problem of similarly obtaining the three-dimensional position and orientation of the user's viewpoint in the world coordinate system.

As a method for solving these problems, a three-dimensional position / orientation sensor such as a magnetic sensor or an ultrasonic sensor is used to make the three points of the photographing device or the user's viewpoint (hereinafter simply referred to as a viewpoint) in the world coordinate system. It is common practice to acquire the dimensional position and orientation.

Further, in a situation where a fixed value can be used for the position of the viewpoint such as when the distance to the object is sufficiently large outdoors, it is configured by a combination of a gyro sensor and an accelerometer. It is common practice to obtain a three-dimensional posture of a viewpoint using a three-dimensional posture sensor and then acquire the three-dimensional position and posture of the viewpoint based on the obtained posture.

The output value output from the three-dimensional posture sensor is, for example, in the case of the posture sensor TISS-5-40 manufactured by Tokimec Co., Ltd., the opposite direction of gravity is the Y axis, and the X-axis is defined by this Y axis. It is the three-dimensional posture of the sensor itself in the sensor coordinate system, which is the coordinate system defined as the −Z axis in the front direction of the sensor when the sensor is activated on the Z plane. In this way, the output value output by the three-dimensional posture sensor is generally not the three-dimensional posture of the viewpoint in the world coordinate system that is the measurement target. That is, the sensor output value cannot be used as it is as the three-dimensional posture of the viewpoint in the world coordinate system, and some coordinate conversion needs to be performed. Specifically, coordinate conversion for converting the attitude of the sensor itself into the attitude of the viewpoint and coordinate conversion for converting the attitude in the sensor coordinate system into the attitude in the world coordinate system are necessary. In this specification, data for performing coordinate conversion between the sensor output value and the three-dimensional posture of the viewpoint in the world coordinate system will be referred to as correction information.

FIG. 1 is a diagram showing a configuration of a general image display device for presenting an optical see-through type mixed reality.

The display screen 110 is of a see-through type and is fixed to the head mounting portion 100 together with the posture sensor 120. When a user (not shown) wears the head mounting portion 100 so that the display screen 110 is positioned in front of the eyes, the display screen 11
Through the 0 optical system (not shown in FIG. 1), the real space in front of the display screen can be observed. Attitude sensor 12
0 measures the posture of the sensor itself in the sensor coordinate system and outputs the posture measurement value with 3 degrees of freedom. The attitude sensor 120 internally has an inclinometer (not shown) capable of measuring the gravity direction of the earth, and as described above, one axis of the sensor coordinate system (in this example, the Y axis) is It is set in the opposite direction of gravity.

The posture information output unit 130 converts the measurement value input from the posture sensor 120 according to the correction information stored in the memory 140, calculates the posture of the user's viewpoint in the world coordinate system, and outputs this as posture information. To do. The image generation unit 150 generates a virtual image corresponding to the posture of the user's viewpoint according to the posture information input from the posture information output unit 130, and outputs this. The display screen 110 receives a virtual image from the image generator 150 and displays it. With the configuration described above, the user (not shown) sees the virtual image displayed on the display screen 110 by superimposing it on the image of the real space through the display screen 110.

Next, referring to FIG. 2, the posture information output unit 13
A method of calculating the orientation of the user's viewpoint in the world coordinate system in 0 will be described.

In the figure, the orientation of the sensor coordinate system 210 in the world coordinate system 200 is R _TW , and the sensor coordinate system 210 is
The attitude of the attitude sensor 120 at R _ST , and the attitude sensor 1
The relative posture of the user's viewpoint 220 viewed from 20 is R _VS , and the posture of the user's viewpoint 220 in the world coordinate system 200 is R _VW .

Where R is a 4 × 4 matrix, and R_BA
Describes the posture of the object B in a certain coordinate system A. this
In other words, the coordinate system defined by the object B from the coordinate system A
The coordinate conversion matrix to B, and the coordinate P in the coordinate system A_A
= (X_A, Y_A, Z_A, 1)^TIs the coordinate P in coordinate system B_B
= (X_B, Y_B, Z_B, 1)^TConversion formula to convert to P_B= R
_BAP_AIs defined. That is, world coordinate system 2
Posture R of the user's viewpoint 220 in 00_VWIs the world
Coordinate P in standard 200_W= (X_W, Y_W, Z_W, 1)^T
Is the coordinate P in the viewpoint coordinate system 230 of the user._V= (X_V，
Y_V, Z_V, 1)^TCoordinate transformation matrix (P_V=
R_VWP_W) Can also be paraphrased.

The matrix R is defined by a rotation matrix Rx defined by a rotation angle θ about the X axis, a rotation matrix Ry defined by a rotation angle (azimuth angle) φ about the Y axis, and a rotation angle ψ about the Z axis. It is configured by the product of the defined rotation matrix Rz, and the relation of R = RzRxRy is established. Each of these matrices is represented as follows.

[0015]

[Equation 1] At this time, R _VW can be represented by R _VW = R _VS · R _ST · R _TW (Equation A).

Since the Y-axis of the sensor coordinate system 210 is set in the direction opposite to gravity, the Y-axis of the world coordinate system 200 is set.
By defining the axis perpendicular to the ground surface, the world coordinate system 2
00 and the direction of the Y axis of the sensor coordinate system 210 can be matched. At this time, Rx _TW and Rz _TW, which are components of R _TW around the X axis and the Z axis, are unit matrices, respectively, and R _TW is equivalent to a rotation matrix Ry _TW defined by a rotation angle φ _TW around the Y axis. Become. Therefore, the above (formula A) is transformed into R _VW = _RVS · R _ST · Ry _TW (formula B).

Of these, R_STFrom the posture sensor 120
Input to the force information output unit 130, R_VWIs the posture information output unit 1
30 is output to the image generation unit 150, and R_VSAnd R
y_TW(In other words, R_VSRotation about three axes that define
Angle θ_VS, Φ_VS, Ψ_VSAnd Ry _TWRotation angle φ that defines_TW)
But R_STR_VWCorresponds to the correction information required to convert to
To do. The posture information output unit 130 outputs from the posture sensor 120.
R input_STAnd R stored in the memory 140_VS
And Ry_TWBased on (Equation B) using _VWCalculate
Then, this is output to the image generation unit 150.

In order to perform accurate alignment between the real space and the virtual space, it is necessary to set accurate correction information in the memory 140 by some means. Only when the correct correction information is given, the display of the virtual image accurately aligned in the real space is realized.

As one of the known methods for setting the correction information, the user or the operator uses the input means (not shown) to store the respective values of θ _VS , φ _VS , ψ _VS and φ _TW. There is a method of interactively increasing and decreasing and adjusting each value by trial and error until accurate alignment is achieved.

However, in this method, since it is necessary to adjust the four parameters at the same time, there is a problem that the work is complicated and the adjustment takes time.

As one method for reducing this complexity,
There is a method proposed by Japanese Patent Application No. 2001-050990 by the present applicant. The correction information setting method will be described below.

FIG. 3 is a block diagram showing the configuration of an image display device when this correction information setting method is incorporated in the image display device of FIG. As shown in FIG.
In addition to the above configuration, the correction information calculation unit 310 and the instruction information input unit 32
0 and the switching unit 330 are added. Also,
The components of the image display device corresponding to the posture information output unit 130 and the memory 140 of FIG. 1 have different functions from those of the image display device of FIG. 1. Therefore, in the image display device of FIG. These are designated as 130 'and memory 140'.

The correction information is calculated by moving the posture of the user's viewpoint 220 to a posture Ry ⁰ _VW which is a predetermined posture in the world coordinate system and has a visual axis parallel to the horizon at that time. It is performed by acquiring the output R ⁰ _ST of the posture sensor 120 at. In addition to the correction information, the memory 140 ′ holds the above-mentioned predetermined viewpoint posture Ry ⁰ _VW (or the rotation angle φ ⁰ _VW about the Y axis that defines it).

The switching unit 330 receives an input from a user or an operator (neither is shown in FIG. 3),
The mode of the posture information output unit 130 'is set to the normal mode or the correction information calculation mode.

When the mode is the normal mode, the posture information output unit 130 'has the posture information output unit 13 described with reference to FIG.
Similar to 0, R _VW is calculated from R _ST input from the posture sensor 120 using the correction information, and this is output to the image generation unit 150 as posture information.

On the other hand, when the mode is the correction information calculation mode, Ry ⁰ _VW is input from the memory 140 ′ and is output to the image generation unit 150 as posture information. The instruction information input unit 320 receives an input from a user or an operator and transmits an instruction to execute the correction information calculation process to the correction information calculation unit 310. More specifically, in the correction information calculation mode, the user or the operator establishes a correct positional relationship between the virtual image displayed on the display screen 110 and the image of the physical space optically observed through the display screen 110. The posture of the viewpoint 220 is adjusted so that Then, at the time when it is determined that both images are sufficiently overlapped (that is, when the viewpoint 220 is determined to be in the posture Ry ⁰ _VW ), for example, by pressing a specific key, the instruction information input unit 320 Input to.

The correction information calculation unit 310 inputs an instruction to execute the correction information calculation process from the instruction information input unit 320, and at that time (that is, the user or the operator determines that the viewpoint 220 is located in the posture Ry ⁰ _VW). receives the output R ⁰ _ST of the orientation sensor 120 in the time), the posture Ry ⁰
_The correction information is calculated based on _VW and the posture R ⁰ _ST .

In the above method, the rotation component about the y-axis of the inverse matrix R _SV (representing the relative attitude of the attitude sensor from the user's viewpoint) of the attitude R _VS which is one of the correction information. It is necessary that Ry _SV (or the rotation angle φ _SV around the Y axis that defines it) is known by some method and already stored in the memory 140 ′.

At this time, the relation of Ry ⁰ _VW = R _VS · R ⁰ _ST · Ry _TW (Equation C) is established between the data processed by the correction information calculation unit 310 from (Equation B). . Here, if the (formula C) is modified, Ry ⁰ _VW = (Rz _SV Rx _SV Ry _SV ) ⁻¹ R ⁰ _ST Ry _TW (Formula D).

The correction information calculation unit 310 has a memory 140 '.
Ry ⁰ _VW and Ry _SV are input from the sensor 120, and R ⁰ _ST is input from the sensor 120, and unknown correction information Rz _SV and Rz are input based on (Equation D).
Calculate x _{S V} and Ry _TW . The procedure is shown below.

When the equation (D) is further modified, it becomes Rz _SV Rx _SV Ry _SV Ry ⁰ _VW = Rz ⁰ _ST Rx ⁰ _ST Ry ⁰ _ST Ry _TW (Equation E). Here, since each of the left side and the right side of (Equation E) is the product of the rotation components around the Z, X, and Y axes,
An identity is established for each rotation component around the Z, X, and Y axes.

First, the identities of the rotational components around the Z and X axes are as follows. That is, Rz _SV = Rz ⁰ _ST (Equation F) Rx _SV = Rx ⁰ _ST (Equation G) From this, Rz _SV and Rx _SV can be obtained.

On the other hand, the identity of the rotation component around the Y axis is as follows. That is, Ry _SV Ry ⁰ _VW = Ry ⁰ _ST Ry _{TW From} this, Ry _TW = Ry _SV Ry ⁰ _VW Ry ⁰ _ST ^-1 (Equation H) and Ry _TW can be obtained.

The correction information calculation unit 310 is based on the above processing.
Correction information Rz_SV, Rx_SV, Ry _TWIs calculated and
Further R from these values_VS(= (Rz_SVR x_SVR
y_SV)^-1) Is calculated and R_VSAnd Ry_TW(Or
Rotation angle θ that defines them_SV, Ψ_SVAnd φ_TW) Memory 1
Output to 40 '.

Derivation of Ry _SV may be carried out by trial and error using a measured value of a protractor or the like as a clue, or may be carried out by using any other measuring means.

As described above, if only Ry _SV of the correction information is obtained in advance, other unknown correction information can be easily derived and accurate alignment can be realized.

[0037]

However, the correction information Ry
It is generally difficult to measure _SV accurately in advance. Thus, in practice, the rough setting of Ry _SV, derivation of other correction information, the fine adjustment of Ry _SV, derivation of other correction information, the fine adjustment of Ry _SV, the task of ..., the correct positioning You have to repeat it until it is realized,
There has been a problem that acquisition of correction information requires a complicated operation and time.

The present invention has been made in view of the above problems, and it is an object of the present invention to more easily obtain correction information for converting a sensor measurement value into a viewpoint posture in the world coordinate system.

[0039]

In order to achieve the object of the present invention, for example, an image display device of the present invention has the following configuration. That is, an image display device that displays a virtual image according to the posture of the viewpoint of the user observing the display screen on the display screen, and a posture sensor that outputs a posture measurement value, and a predetermined posture that is stored in advance in a memory. A notification indicating that the first display means for generating the virtual image based on the information and displaying the virtual image on the display screen, the posture of the viewpoint, and the posture indicated by the predetermined posture information match is given. If
Calculation means for calculating correction information based on the posture measurement value from the posture sensor and the predetermined posture information, and a viewpoint obtained using the posture measurement value and the correction information obtained from the posture sensor. Second display means for generating a virtual image based on the posture information and displaying the virtual image on the display screen, and correcting the correction information according to a predetermined operation while displaying the virtual image by the second display means. And a correction means for doing so.

In order to achieve the above object, the image display method of the present invention is a virtual space corresponding to the posture of the viewpoint of the user observing the display screen while optically transmitting the real space to the display screen. An image display method for displaying the image on the display screen, and a switching step of switching between a first mode and a second mode; and in the first mode, correction information and an attitude sensor input. A posture that calculates the posture of the viewpoint based on the posture measurement value of three degrees of freedom and outputs the posture as posture information, and in the second mode, outputs a predetermined posture as the posture information of the viewpoint. An information output step, an image generation step of generating the image based on the posture information output by the posture information output step, and an instruction that the posture of the viewpoint matches the predetermined posture Instruction information input step of inputting indication information, a correction for calculating the correction information based on the attitude measurement value at the time when the instruction information is input and / or before and after the time, and the predetermined attitude An information calculation step, a correction information input step of inputting correction information for correcting the correction information, and a correction information correction step of correcting the correction information according to the correction information are provided. Further, an image display method according to another aspect of the present invention generates a superimposed image in which an image of a virtual space corresponding to the viewpoint posture of the image capturing device is superimposed on a real image captured by the image capturing device,
An image display method for displaying this on a display screen, comprising:
And a switching step of switching between the second mode and the second mode, and in the first mode, the posture of the viewpoint is calculated based on the correction information and the posture measurement value of three degrees of freedom input from the posture sensor. And output this as posture information,
In the second mode, a posture information output step of outputting a predetermined posture as posture information of the viewpoint,
An image generation step of generating the superimposed image based on the posture information output by the posture information output step, and an instruction of inputting instruction information indicating that the posture of the viewpoint matches the predetermined posture. An information input step, a correction information calculation step of calculating the correction information based on the attitude measurement value at the time when the instruction information is input and / or before and after the time point, and the predetermined attitude; A correction information input step of inputting correction information for correcting the correction information and a correction information correction step of correcting the correction information according to the correction information are provided.

[0041]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

In the following embodiments, the case where the image display device and the method of the present invention are applied to a simulation device and a method for performing a landscape simulation of a building to be constructed will be described.

FIG. 4 shows the apparatus configuration of the simulation apparatus of this embodiment. As shown in the figure, the apparatus configuration of the present embodiment is similar to the configuration of the image display apparatus shown in FIG. 3 except that the correction information correction section 400 and the correction information input section 410 are included.
Has been added. In addition, the image generation unit 15
0'is assumed to have a function of generating a CG image of a building to be constructed as a virtual image. In addition, the head mounting portion 1
00, display screen 110, posture sensor 120, posture information output unit 130 ′, memory 140 ′, correction information calculation unit 31
0, the instruction information input section 320, and the switching section 330 have the same operations as those of the image display apparatus shown in FIG. 3, and thus detailed description thereof will be omitted.

In this embodiment, the correction information φ _SV held in advance in the memory 140 'does not have to be an accurate value. The outline of the correction information acquisition process in this embodiment will be described. (1) First, the correction information φ roughly set
Based on _SV , correction information R _VS and Ry _TW (or rotation angles θ _SV , ψ _SV and φ _TW defining them) are calculated by the correction information calculation method described in FIGS. 2 and 3,
(2) After that, it is realized by finely adjusting the uncertain correction information Ry _SV and Ry _TW (or the rotation angles φ _SV and φ _TW defining them) by the method described below.

Fine adjustment of the rotation angles φ _SV and φ _TW will be described. Correction information input unit 410, the input of the user or operator (both not shown), the command (increase command) to the value of the correction information phi _SV is increased by a certain value or, alternatively, a constant value the value of the correction information phi _SV decreased The command (decrease command) to be output is output to the correction information correction unit 400.

The correction information correction section 400 has a memory 140 '.
The values of the correction information φ _SV and φ _TW are input from the correction information input unit 410, the correction information described below is added to the correction information according to the increase / decrease command transmitted from the correction information input unit 410, and the result is output to the memory 140 ′. To do.

That is, when the increase command is input, the value of φ _SV is increased by a constant value Δφ, and the value of φ _TW is also increased by Δ.
Increase by φ. φ _SV ← φ _SV + Δφ φ _TW ← φ _TW + Δφ (Equation I)

Similarly, when the decrease command is input,
The value of φ _SV is reduced by Δφ, and the value of φ _TW is also reduced by Δφ. φ _SV ← φ _{SV −} Δφ φ _TW ← φ _{TW −} Δφ (Equation J)

Here, the value of the correction information φ _SV and the value of φ _TW are manipulated by the same amount of value because the value of (equation H) is set between these values.
Therefore, the relationship of φ _TW = φ _SV + φ ⁰ _VW −φ ⁰ _ST (Equation K) is established.

FIG. 5 is a flow chart for explaining the processing procedure of the image display device according to the present embodiment. The program code according to the flowchart is stored in a memory such as a RAM or a ROM (not shown) in the apparatus of the present embodiment, read by a CPU (not shown), and executed.

First, in step S1010, it is determined whether the current mode is the correction information calculation mode.
The switching of the modes is performed by the switching unit 330. For example, the switching unit 330 detects a predetermined key operation, determines whether the correction information calculation mode or the normal mode is instructed, and the determination result is the posture information output unit 13.
0 ′ and the correction information calculation unit 310 are notified.

When the mode is the correction information calculation mode, in step S1140, the correction information calculation unit 310 determines whether the instruction information is input from the instruction information input unit 320. When the instruction information has not been input, the correction information calculation unit 310 does not execute the processing shown in steps S1150 to S1180. Then, the process skips to step S1190, and the posture information output unit 130 '
Predetermined posture information is input from the memory 140 '. Then, in step S1090, this is output to the image generation unit 150 ′ as viewpoint orientation information.

In step 1100, the image generator 15
0'inputs the posture information of the viewpoint from the posture information output unit 130 'and sets it as the drawing parameter of the virtual object. Further, in step S1110, a virtual object is drawn using the set drawing parameters to generate a CG image. Then, in step 1120, the generated CG image is output to the display screen 110. Thus, the virtual object according to the predetermined posture is displayed on the display screen 110. This process is repeatedly performed during the correction information calculation mode.

On the other hand, the user or the operator adjusts the posture of the head so that the virtual object displayed on the display screen 110 has a correct positional relationship with the real space, and when they match, the instruction information input section 320 causes the instruction information to be input. Enter.
In this, for example, the instruction information input 320 detects a predetermined key operation by the user, and notifies the correction information calculation unit 310 of that.

When the instruction information is input, the correction information calculation unit 310 inputs the measurement value from the attitude sensor 120 in step S1150. Next, step S116.
At 0, the predetermined posture and correction information Ry _SV (or the rotation angle φ _SV that defines it) is input from the memory 140 ′. Then, in step S1170, the unknown correction information is calculated according to the above-described (formula F), (formula G), and (formula H), and in step S1180, the calculated correction information is output to the memory 140 ′.

After that, the normal mode is set by the switching unit 330, and the uncertain correction information Ry _SV and Ry _{TW are set.}
(Or the rotation angles φ _SV and φ _TW that define them) are finely adjusted. The normal mode in which such fine adjustment is possible will be described below.

When the mode is set to the normal mode by the switching unit 330, the process proceeds from step S1010 to step S1020. In step S1020, it is determined whether correction information is input from correction information input unit 410.

If the correction information has not been input, the process proceeds to step S1060 and the posture information output unit 130 '.
However, while inputting the measurement value from the posture sensor 120,
In step S1070, the correction information is input from the memory 140 '. In step S1080, the posture information output unit 130 'calculates the posture of the viewpoint based on the input measurement value and correction information. Then, step S109
0, the image generation unit 1 uses this as viewpoint posture information.
Output to 50 '.

The image generating section 150 'has step S110.
At 0, the posture information of the viewpoint is input from the posture information output unit 130 ′, at step S1110 a virtual object is drawn based on this posture information to generate a CG image, and at step 1120 the CG image generated is displayed on the display screen 1
Output to 10. Thereby, the virtual object according to the posture of the viewpoint is displayed on the display screen 110. This process is repeated during the normal mode.

The user or operator confirms whether the virtual object displayed on the display screen 110 and the real space are correctly aligned when the posture of the head is changed, and the positional shift is caused by the change in the posture. If it occurs, input the correction information by the correction information input unit 410,
Fine-tune the correction information.

When the correction information is input, the correction information correction section 400 executes the following steps S1030 to S1050.

First, in step S1030, the correction information Ry _SV and Ry _TW (or the rotation angles φ _SV and φ _TW defining them) are input from the memory 140 '.
Next, in step S1040, the correction information is corrected according to the above (formula I) and (formula J), and in step S1050, the corrected correction information is stored in the memory 14
Output to 0 '. Since the corrected correction information is immediately reflected in the calculation of the posture in the posture information output unit 130 ', the user or the operator interactively corrects it while seeing the result of the fusion of the image of the real space and the image of the virtual space. It is possible to enter information. Therefore, fine adjustment of the correction information can be easily performed so that correct alignment is realized. The correction information is a command to increase or decrease the predetermined amount, but it may be possible to instruct the increase amount or the decrease amount.

The posture information output unit 130 'and the memory 14
0 ′, the image generation unit 150 ′, the correction information calculation unit 310, the instruction information input unit 320, the switching unit 330, the correction information correction unit 400, and the correction information input unit 410 are, for example, one general-purpose computer (notebook computer, etc.). ). Further, in the present embodiment, the posture sensor 120 is, for example, the posture sensor TI of Tokimec Co., Ltd.
Although it is configured to include SS-5-40, other attitude sensors may of course be used.

The switching unit 330 and the instruction information input unit 32
It is desirable that 0 and the input by the correction information input unit 410 are performed through the same user interface, and for example, a keyboard or a control pad can be used.

Although the above-mentioned embodiment is for realizing the MR of the optical see-through system, the present invention is not limited to the optical see-through system and can be applied to the MR of the video see-through system. In that case, the measurement target of the posture that was the user's viewpoint 220 in the above embodiment is
It becomes the viewpoint of the image pickup device (for example, a video camera). Also,
In the case of the video see-through method, the image generation unit 150 ′ is
The CG image is superimposed and drawn on the image of the physical space taken by the image pickup device, and this is displayed on the display screen 110.

<Modification 1> In the above embodiment, the CG image of the virtual object to be displayed is used as a clue for the alignment, but it is necessary to confirm that the alignment between the virtual space and the real space is performed. Any geometric information may be used as long as it is a visual cue. For example, the topography and the shape of the real building may be modeled, and the wire frame model may be superimposed and drawn at the same position in the real space for alignment. It goes without saying that it is possible to display the wire frame model of the physical object and the virtual object to be displayed in combination.

<Modification 2> In the above embodiment, the image display device and method of the present invention are applied to the landscape simulation, but needless to say, it is applicable to MR systems other than the landscape simulation. is there.
Further, the present invention is not limited to the MR system and can be used for various purposes of measuring the position and orientation of the viewpoint using the orientation sensor.

<Modification 3> In the above embodiment, the attitude sensor TISS-5-40 manufactured by Tokimec Co., Ltd. is used as the attitude sensor, but the invention is not limited to this application. As long as one axis of the sensor coordinate system is set in association with the world coordinate system, it can be applied to any sensor.

<Modification 4> In the above embodiment, the correction information calculation unit 310 inputs the output of the posture sensor 120 at the time when the instruction information is input as R ⁰ _ST , and uses this for the calculation of the correction information. In step S1170, the output of the posture sensor 120 at the time when the instruction information is input does not necessarily have to be R ⁰ _ST . For example, the measurement values (for example, the average value thereof) of the posture sensor 120 before and after the time when the instruction information is input may be used.
Alternatively, the viewpoint 220 may be kept stationary in the posture Ry ⁰ _VW for a certain period of time, and R ⁰ _ST may be calculated based on the average value of the output of the posture sensor 120 during that time, or the viewpoint 220 may be positioned in the posture Ry ⁰ _VW . R ⁰ _ST may be acquired by any method as long as the output of the attitude sensor 120 can be obtained during the operation.

[Other Embodiments] An object of the present invention is to supply a storage medium (or recording medium) recording a program code of software for realizing the functions of the above-described embodiments to a system or apparatus, and to supply the system or apparatus with the storage medium. It is needless to say that this is also achieved by the computer (or CPU or MPU) of the device reading and executing the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention. Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instructions of the program code. Do some or all of the actual processing,
It goes without saying that the processing includes the case where the functions of the above-described embodiments are realized.

Furthermore, after the program code read from the storage medium is written in the memory provided in the function expansion card inserted in the computer or the function expansion unit connected to the computer, based on the instruction of the program code, Needless to say, this also includes a case where a CPU or the like included in the function expansion card or the function expansion unit performs a part or all of the actual processing and the processing realizes the functions of the above-described embodiments.

When the present invention is applied to the above storage medium, the storage medium stores the program code corresponding to the above-described flowchart (shown in FIG. 5).

As described above, according to the present invention, it is possible to easily obtain the correction information for converting the orientation of the sensor in the sensor coordinate system into the orientation of the measurement target in the world coordinate system.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a configuration of a general image display device that presents an optical see-through type mixed reality.

FIG. 2 shows a world coordinate system 2 in an attitude information output unit 130.
10 is a diagram illustrating a method of calculating the posture of the user's viewpoint 220 in 00. FIG.

FIG. 3 is a diagram showing a configuration of an image display device including a correction information calculation unit.

FIG. 4 is a block diagram showing a configuration of an image display device according to an embodiment of the present invention.

FIG. 5 is a flowchart illustrating control in the image display device according to the embodiment of the present invention.

Claims (10)

[Claims] 1. An image display device for displaying a virtual image according to a posture of a user's viewpoint observing a display screen on the display screen, wherein the posture sensor outputs a posture measurement value, and the image sensor is stored in advance in a memory. The first display means for generating the virtual image based on predetermined posture information and displaying the virtual image on the display screen indicates that the viewpoint posture matches the posture indicated by the predetermined posture information. When a notification is made, a calculation means for calculating correction information based on the posture measurement value from the posture sensor and the predetermined posture information, and a posture measurement value and the correction information obtained from the posture sensor. Second display means for generating a virtual image on the display screen based on the posture information of the viewpoint obtained by using the second display means, and a predetermined operation while displaying the virtual image by the second display means. Depending on The image display apparatus characterized by comprising a correction means for correcting a serial correction information. 2. The first and second display means display an image of a virtual space on the display screen while optically projecting an image of the real space on the display screen. 1. The image display device according to 1. 3. The first and second display means displays the virtual image by superimposing the virtual image on the real image captured by the image capturing device on the display screen. Image display device. 4. The calculation means calculates the correction information based on the posture measurement value at the time when the instruction information is input and / or before and after the time and the predetermined posture information. The image display device according to claim 1. 5. The output value of the posture sensor is information representing a posture of the sensor itself in a sensor coordinate system, and the correction information is a first posture for converting the posture of the sensor itself into the posture of the viewpoint. Second coordinate conversion information and converting the attitude in the sensor coordinate system into the attitude in the world coordinate system
The coordinate conversion information of the azimuth of the first coordinate conversion information and the conversion information of azimuth of the second coordinate conversion information are corrected at the same time. The image display device according to any one of claims 1 to 4, wherein: 6. An image display method for displaying on a display screen an image of a virtual space according to the posture of the viewpoint of a user observing the display screen while optically transmitting the real space on the display screen, A switching step of switching between the first mode and the second mode; and in the first mode, the posture of the viewpoint is determined based on the correction information and the posture measurement value of three degrees of freedom input from the posture sensor. A posture information output step of calculating and outputting this as posture information and outputting a predetermined posture as the posture information of the viewpoint in the second mode, and a posture information output by the posture information output step An image generation step of generating the image based on the instruction information; an instruction information input step of inputting instruction information for instructing that the posture of the viewpoint matches the predetermined posture; A correction information calculating step of calculating the correction information based on the posture measurement value before and / or after the time point when the force is applied and the predetermined posture, and correction information for correcting the correction information. An image display method comprising: a correction information inputting step for inputting the correction information; and a correction information correction step for correcting the correction information according to the correction information. 7. An image display method for generating a superimposed image, which is obtained by superimposing an image of a virtual space according to a posture of a viewpoint of the image capturing device on a real image captured by the image capturing device, and displaying the superimposed image on a display screen. A switching step of switching between the first mode and the second mode, and in the first mode, based on the correction information and the posture measurement value of three degrees of freedom input from the posture sensor, A posture information output step of calculating a posture and outputting the posture information as posture information, and in the second mode, outputting a predetermined posture as posture information of the viewpoint, and a posture output by the posture information output step. An image generation step of generating the superimposed image based on information; an instruction information input step of inputting instruction information instructing that the posture of the viewpoint matches the predetermined posture; Correction information calculation step of calculating the correction information based on the posture measurement value at the time when the display information is input and / or before and after the time point and the predetermined posture, and for correcting the correction information An image display method comprising: a correction information input step of inputting the correction information of 1. and a correction information correction step of correcting the correction information according to the correction information. 8. The posture measurement value is information representing a posture of the sensor itself in a sensor coordinate system, and the correction information is a first coordinate transformation for converting the posture of the sensor itself into a posture of the viewpoint. Second information for converting the posture in the sensor coordinate system into the posture in the world coordinate system
The correction information correction step includes correcting the conversion information regarding the azimuth angle of the first coordinate conversion information and the conversion information regarding the azimuth angle of the second coordinate conversion information at the same time. The image display method according to claim 6 or 7. 9. A computer-readable storage medium that stores a program code for causing a computer to execute the image display method according to any one of claims 5 to 7. 10. A control program for causing a computer to execute the image display method according to claim 5.