JPH1184307A - Head-mounted optical device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately measure the posture of a worker by following up his movement even when the three-dimensional position of the worker is moved by providing a device with a first optical system integrated in a main body and set for guiding an image to an eyeball and a camera fixed to the main body and having a visual field in front of his head. SOLUTION: Respective HMDs 210 are fixed to the head of a player by bands. Besides, a magnetic sensor 220 and the CCD camera 240 are fixed to the head of the player. When the player obliquely looks down in order to view the surface of a table, the surface of the table, a virtual pack, a real mallet and a virtual goal can be viewed in the visual field through the HMDs 210. When the head of the player is horizontally moved within a horizontal two-dimensional plane or the tilting movement, the yaw movement or the rolling movement thereof is executed, the change thereof is detected by the sensor 220 and also observed as the change of the image picked up by the camera 240 in accordance with the change of the posture of the head.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a head-mounted optical device used for, for example, a head-mounted display, and more particularly, to an image acquisition means such as a camera for accurately detecting a head position of a wearer. The present invention relates to an optical device having the same.

[0002]

2. Description of the Related Art In recent years, mixed reality (hereinafter, referred to as "MR" (Mixed
Reality) has been actively studied.
Conventionally, MR is a virtual reality (hereinafter abbreviated as VR) that can only be experienced in a situation separated from the real space.
For the purpose of coexistence of the real world and the real space, it is attracting attention as a technology for enhancing VR.

[0003] Applications of MR include medical assistance for presenting to a physician a transparent view of the inside of a patient's body,
New fields that are completely different in quality from conventional VRs are expected, such as the use of work assistance to display the assembly procedure of a product on a real product in a factory. Applications of MR include the use of medical assistance, which presents to a physician as if looking through the inside of a patient's body, and the use of work assistance, which displays the assembly procedure of a product on a real product at a factory. A new field that is qualitatively completely different from VR is expected.

[0004] A common requirement for these applications is a technique for removing the "shift" between the real space and the virtual space. The “displacement” can be classified into a displacement, a time displacement, and a qualitative displacement, and among them, many efforts have been made to eliminate the displacement, which is the most basic requirement. Video see-through (Vid
In the case of the MR of the eo-See-Through) type, the problem of the alignment results in a problem of accurately determining the three-dimensional position of the video camera.

An optical see-through (MR) MR using a transflective HMD (Head Mount Display)
The problem of alignment in the case of
Although it is a problem to determine the three-dimensional position, three-dimensional position and orientation sensors such as a magnetic sensor, an ultrasonic sensor, and a gyro are generally used as the measurement method, but the accuracy of these is not always sufficient, and the Causes positional deviation.

On the other hand, in the case of the video see-through system,
A method of directly performing positioning on an image based on image information without using such a sensor is also conceivable. Although this method can directly deal with a position shift and thus can perform position alignment with high accuracy, there are problems such as a lack of real-time performance and reliability. In recent years, attempts have been reported to achieve accurate alignment by using both a position and orientation sensor and image information to compensate for the disadvantages of both.

As one attempt, “Dynamic Registrati
on Correction in Video-Based-Augmented Reality Sys
tems '' (Bajura Michael and Ulrich Neuman, IEEE Compute
r Graphics and Applications 15, 5, pp. 52-60, 199
5) (hereinafter referred to as the first document) proposed a method of correcting a position shift caused by an error of a magnetic sensor by image information in a video see-through type MR.

[0008] "Superior Augmented Reality Reg"
istration by Integrating Landmark Tracking and Mag
netic Tracking '' (State Andrei et al., Proc. of SIGGRAPH
96, pp. 429-438, 1996) (hereinafter referred to as the second document)
We further developed this method and proposed a method to compensate for the ambiguity of position estimation by image information using sensor information. In the above-mentioned conventional example, when a video see-through type MR is constructed using only the position and orientation sensor, in consideration of the fact that a displacement of the sensor causes a displacement on an image, the displacement is represented by image information. In order to detect a landmark, a landmark whose three-dimensional position is known is set as a clue in the real space.

Assuming that no error is included in the sensor output, the coordinates Q _{I of the} landmark actually observed on the image
When, the observation predicted coordinate P _I landmarks that Dasa led from the three-dimensional position of the camera positions and landmarks obtained based on the sensor output should be the same. But,
In practice, the camera position obtained based on the sensor output is not accurate, so that Q _I and P _I do not match. This P _I and Q _I
The displacement indicates the displacement between the virtual space and the real space at the landmark position, and therefore, by extracting the landmark position from the image, the direction and magnitude of the displacement can be calculated.

As described above, by quantitatively measuring the displacement on the image, the camera position can be corrected so as to eliminate the displacement. The simplest alignment method that uses an azimuth sensor and an image is considered to be correction of a sensor error using a single landmark, and the camera position is translated or rotated according to the displacement of the landmark on the image. An approach has been proposed by the first document.

FIG. 1 shows the position using one landmark.
The basic concept of displacement correction will be described. Below, the camera
The internal parameters of the filter and remove the effects of distortion
Images are taken by the ideal imaging system
Assume that The viewpoint position of the camera is C, run on the image
Q: Domark's observation coordinates_IThe landmark of the real space
Place Q_IThen the point Q_IAre points C and Q_IA straight line connecting_QUp
Exists. Meanwhile, given by the position and orientation sensor
From the camera position, landmarks in the camera coordinate system
Position P_CAnd the observation coordinates P on the image_ICan be inferred.
In the following, from the point C to the point Q_I, P_ITo the three-dimensional vector
Each v_l, V_TwoNotation. In this method, the corrected
Landmark observation coordinate P^' _IIs Q_IMatches
(I.e., the corrected land in the camera coordinate system)
Mark prediction position P ^' _CIs a straight line l_QRidiculous)
By correcting the relative position information between the camera and the object,
The displacement is corrected.

[0012] Consider that the displacement of the landmark is corrected by rotating the camera position. This can be realized by modifying the position information of the camera so that the camera rotates by the angle θ formed by the two vectors v _l and v ₂ . In the actual calculation, a vector v _ln , v _2n obtained by normalizing the above vectors v _l , v ₂ is used, and the outer product v _ln × v _2n is used as a rotation axis, and the inner product v _1n · v _2n is used as a rotation angle. Rotate the camera around C.

It is considered that the displacement of a landmark is corrected by a relative translation of a camera position. This can be realized by translating the object positions in the virtual world by v = n (v ₁ −v ₂ ). Here, n is a scale factor defined by the following equation.

[0014]

(Equation 1)

Here, | AB | is a symbol indicating the distance between points A and B. The same correction can be made by modifying the position information of the camera so that the camera moves in parallel by −v. This is because this operation is equivalent to relatively moving the virtual object by v. The above two methods are two-dimensionally matching positional deviations on landmarks, and cannot correct a camera position to a three-dimensionally correct position. However, when the sensor error is small, a sufficient effect can be expected. Further, the calculation cost for the correction is very small, and this method is excellent in real time.

[0016]

However, the technique disclosed in the above document is limited to the video see-through system, and is not suitable for application to an optical see-through mixed reality apparatus. This is because in the optical see-through system, the problem of alignment becomes much more important than in the video see-through system.

[0017]

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and has as its object to propose an optical device having a camera that moves following the movement of an operator. And To achieve the above object, the present invention provides an optical device mounted on an optical see-through head,
A body and a first integrated into the body for guiding an image to an eyeball;
And a camera fixed to the main body and having a field of view in front of the head.

According to a preferred aspect of the present invention, a three-dimensional position and orientation sensor for detecting a three-dimensional position and orientation,
A liquid crystal display panel for displaying the image. According to a preferred aspect of the present invention, the three-dimensional position sensor is a magnetic sensor that detects a change in a magnetic field, and the magnetic sensor is fixed to the main body while being spatially separated from the liquid crystal display unit. .

A head mounted optical device according to a preferred aspect of the present invention is an optical see-through type optical device, which is incorporated in the main body and guides light from an external environment to an eyeball. By having a system and a display unit for displaying the image, light of the image from the first optical system and light of the environment from the second optical system enter the eyeball. It is characterized by the following.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A system for presenting a mixed reality presentation method and an HMD applied to an air hockey game device according to the present invention will be described below. The air hockey game is a competitive game in which an opponent exists. Usually, a puck is floated by supplying compressed air from the lower part, the puck is beaten, and the puck is scored when the puck is put into the opponent's goal. In this game, the player with the highest score wins.
In the air hockey game to which the MR according to the present embodiment is applied, the puck is superimposed on a table in a real environment as a virtual three-dimensional image, presented to a player, and played with a real mallet.

<Structure of Game Apparatus> FIG. 2 is a view of the game apparatus portion of the system according to the present embodiment as viewed from the side. Mixed reality air hockey game is played on table 1
The two opponents 2000, 3000 face each other with a mallet (260L, 260R) in between. The two opponents 2000 and 3000 wear head-mounted displays 210L and 210R (hereinafter abbreviated as HMDs) on their heads. The mallet of the present embodiment has an infrared light emitter at its tip. In the present embodiment, the mallet position is detected by image processing. However, if the mallet has features in the shape and color, it can be detected by pattern recognition using those features.

The HMD 210 of the embodiment is of a see-through type as shown in FIG. Two opponents 2000, 300
0 indicates that the surface of the table 1000 can be observed even when the HMDs 210L and 210R are mounted. HMD
A three-dimensional virtual image is input to 210 from an image processing system described later. Therefore, the opponents 2000, 3000
The three-dimensional image displayed on the display screen of the HMD 210 is superimposed on the image of the real space passed through the optical system of the HMD 210 (not shown in FIG. 2).

FIG. 3 shows an image viewed from the HMD 210L by the left player 2000. The two players hit a virtual puck 1500. To hit the puck 1500, a real mallet 260L held by the player 2000 is used. Player 2000 holds mallet 260L in his hand. The goal 1200R can be seen just before the opponent player 3000. Image processing system (third
(Not shown in the figure) generates a three-dimensional CG and displays it on the HMD 240L so that the goal 1200R can be seen near the opponent.

On the other hand, the player 3000 also has the HMD 210
Goal 1200L near the player 3000 via R
Will be seen. The pack 1500 is also generated by an image processing system (not shown) and displayed on each HMD. <HMD with Magnetic Sensor> FIG. 4 shows the configuration of the HMD 210. The HMD 210 is disclosed in, for example,
The magnetic sensor 220 is attached to the main body of the HMD of No. 3551 via a column 221. In the figure, reference numeral 211 denotes an LCD display panel. The light from the LCD display panel enters the optical member 212, is reflected by the total reflection surface 214, is reflected by the total reflection surface of the concave mirror 213, passes through the total reflection surface 214, and passes through the eyes of the observer. Reaches

In the present embodiment, the magnetic sensor 220 is
An lhemus magnetic sensor, Fastrack, was used. Since the magnetic sensor is vulnerable to magnetic noise, the magnetic sensor is separated from the display panel 211 and the camera 240, which are noise sources, by the support 221. The configuration in which the magnetic sensor and / or camera is attached to the HMD shown in FIG. 4 is not limited to the HMD of the optical see-through type. It can be attached to the HMD for the purpose of accurately detecting the head position and posture.

In FIG. 2, each HMD 210 is fixed to the player's head by a band (not shown).
A magnetic sensor 220 as shown in FIG. 4 and a CCD camera 240 as shown in FIG.
(240L, 240R) are respectively fixed.
The field of view of the camera 240 is set in front of the player. In the case of the air hockey game, since the user looks at the upper surface of the table 1000, the camera 240 also captures an image of the surface of the table 1000. The magnetic sensor 220 (220L, 220R)
0 senses the change in the alternating magnetic field.

When the player faces obliquely downward to look at the surface of the table 1000, the field of view through the HMD 210 shows the surface of the table 1000 and the above-described virtual puck 1
500, real mallet 260 (260L, 260
R), virtual goal 1200 (1200L, 1200
R) is visible. When the player horizontally moves the head in a horizontal two-dimensional plane, or performs a tilting motion, a yaw motion, and a rolling motion, the change is first detected by the magnetic sensor 220, and
It is observed as a change in the image captured by the CCD camera 240 with the change in the posture of the head.

<Plurality of Markers> Each mallet 260 has an infrared light emitter at its tip, and the position of the mallet can be known by a CCD camera 230 that detects the infrared rays. CCD camera 24
0 outputs an image called a marker image.

FIG. 5 shows an example of a marker arranged on the table 1000. In FIG. 5, 5 indicated by a circle
One landmark or marker (1600 to 1604) indicates a marker used for auxiliary detection of the head position of the player 2000, and five landmarks or markers (1650 to 1654) indicated by squares indicate the player 30.
10 shows a marker used for auxiliary detection of the head position of 00. By arranging a plurality of markers in this way, the position of the head, particularly the posture, determines which marker is visible. In other words, the CCD attached to each player
If a marker in the image captured by the camera 240 can be specified and its position in the image can be detected, it can be used for correcting the output signal of the magnetic sensor for detecting the head posture of the player.

A group of markers (1600 to 1) respectively assigned to two players (2000 and 3000)
608) and the marker groups 1650 to 1658) are respectively colored in different colors. In the present embodiment, the marker for the left player (# 1 player) is colored red, and the marker for the right player (# 2 player) is green. This is to make it easy to distinguish markers in image processing.

A major feature of the present embodiment is that a plurality of markers are arranged. The multiple placement ensures that at least one marker is within the field of view of the CCD camera 240 as long as the player acts on the table 1000 within the operating range of the present air hockey game. FIG. 6 illustrates a situation in which the image processing range for detecting a marker moves with the movement of the head when the player moves the head in various ways. As shown in the figure, one image contains at least one marker. In other words, the number of markers, the interval between markers, and the like should be set according to the size of the table 1000, the viewing angle of the camera 240, and the size of the player's movement range based on the nature of the game. In this case, the farther away from the player, the wider the field of view becomes, so the interval between the markers may be widened. This is because the distance between adjacent markers in the image is the same as the distance between distant markers in the image. This is to prevent unnecessary imaging of a plurality of markers in the same frame.

<MR Image Generation System> FIG.
2 shows a configuration of a system for generating and presenting a three-dimensional image in the game device shown in FIG. This image generation / presentation system displays a three-dimensional virtual image (pack 1500, goal 120 in FIG. 3) on each of the display devices of the HMD 240L of the left player 2000 and the HMD 240R of the right player 3000.
0) is output. The generation of the three-dimensional virtual image is performed by the image generation units 5050L and 5050R. In the present embodiment, each of the image generation units 5050 is
The computer system ONYX2 manufactured by SiliconGraphics was used.

The image generation unit 5050 inputs the pack position information and the like generated by the game state management unit 5030 and the information on the corrected viewpoint position and head direction generated by the two correction processing units 5040L and 5040R. Game state management unit 5030 and correction processing units 5040L, 5040R
Each consisted of a computer system ONYX2.

The CCD camera 230 fixed above the center of the table 1000 fits the entire surface of the table 1000 into the field of view. Mallet information acquired by the camera 230 is input to the mallet position measurement unit 5010. This measuring unit 5010 is also O2 manufactured by SiliconGraphics.
It consisted of a computer system. Measurement unit 501
0 detects the mallet position of the two players, that is, the position of the hand. Information on the position of the hand is input to the game state management unit 5030, where the game state is managed. That is, the game state and the progress of the game are basically all determined by the position of the mallet.

A position / posture detection unit 5000 constituted by a computer system O2 manufactured by Silicon Graphics Inc. inputs the outputs of the two magnetic sensors 220L and 220R, detects the viewpoint position and head posture of each player, and executes a correction processing unit. Output to 5040L and 5040R. On the other hand, the CCD cameras 240L and 240R fixed to the heads of the players obtain marker images, and the marker images are processed by the marker position detection units 5060L and 5060R, respectively.
The position of the marker within the field of view of each camera 240 is detected. Information on the marker position is sent to the correction processing unit 5
040 (5040L, 5040R).

Here, two marker position detecting units 5060
(5060L, 5060R) were configured by an O2 computer system. <Mallet Position Measurement> FIGS. 8 to 10 are flowcharts showing a control procedure for measuring the mallet position.

In an air hockey game, a player does not advance his own mallet to the area of another player. Therefore, the process of searching for the mallet 260L (260R) of the left player 2000 (right player 3000) can be performed by concentrating the process on the image data I _L (image data I _R ) of the left field as shown in FIG. Good.
It is easy to divide the image acquired by the CCD camera 230 at the fixed position into two as shown in FIG.

Accordingly, in the flowchart of FIG. 8, the mallet 26 of the player # 1 (player 2000)
Regarding the search for 0L, in step S100, the player #
The search for mallet 260R of player 2 (player 3000) is performed in step S200. Therefore, for convenience, the right player's mallet is searched (step S20).
0) will be described as an example.

First, at step S210, the TV camera 23
A multi-valued image is obtained from 0. At step S212, the performing subroutine "search in local region" for the image data I _R of the right half. The details are shown in FIG.
When the coordinates (x, y) of the mallet position in the image coordinate system are found in step S212, the process proceeds from step S214 to step S220, and the mallet position coordinates (x, y) in the image coordinate system are calculated according to the following equation. It is converted to the coordinate position (x ′, y ′) of the system (see FIG. 13).

[0040]

(Equation 2)

[0041] Here, M _T is a transformation matrix of 3 × 3 for calibrating the image coordinate system and table coordinate system is known. The coordinate position (x ′, y ′) obtained in step S220 is sent to the game state management unit 5030. If no mallet is found in the local area, "search in global area" is performed in step S216. If a mallet is found in the "search in the global area", the coordinate position is converted to a table coordinate system in step S220. The coordinate position searched in the local or global area is used for searching for a mallet in the local area in the next frame.

FIG. 9 shows a process of searching for a mallet in the local area (details of step S212). However, this process shows a search process in the right field for convenience, but the same applies to a mallet search process in the left field. In step S220, a rectangular area of size (2A + 1) × (2B + 1) pixels defined by the following equation is extracted.

[0043]

(Equation 3)

Here, I _x and I _y are arbitrary coordinate values in the search area I _R , A and B are constants for determining the size of the search area, and the search areas are as shown in FIG. become.
Step S230 is a step of extracting, for all the pixels (x, y) in the rectangular area defined in step S220, those whose characteristic evaluation values I _s (x, y) satisfy a certain condition. For the purpose of searching for a mallet, the feature amount is preferably a similarity between pixel values (infrared light intensity values).
In the present embodiment, since an infrared light emitter is used for the mallet, the one having the characteristic of the intensity of the infrared light is determined to be a mallet.

That is, in step S232, the similarity I _S
Find pixels closer to the mallet than a predetermined threshold. When such a pixel is found, the cumulative value of the frequency of occurrence is stored in the counter N. Further, the x coordinate value and the y coordinate value of such a pixel are cumulatively stored in the registers SUMx and SUMy. That is,

[0046]

(Equation 4)

It is assumed that When step S230 ends,
The number N of all pixels similar to the pattern of infrared light from the mallet in the region of FIG. 12, and the cumulative value of coordinate values
SUMx and SUMy are obtained. If N = 0, step S23
In step 6, the result “Not Found” is output. If N> 0, a mallet-like one has been found. In step S238, the mallet position (I _x , I _y ) is

[0048]

(Equation 5)

The calculation is performed according to Then, a coordinate value obtained by converting the mallet position (I _x , I _y ) into a table coordinate system is passed. FIG. 10 shows a detailed procedure of the global area search in step S216. In step S240 of FIG.
In the image I _R in the right field,

[0050]

(Equation 6)

Among the pixels satisfying the condition, the characteristic evaluation value Is
Is stored in the register Max. Here, C and D are constants that determine the coarseness of the search, and Width and Height are defined in FIG. That is, step S242
Then, it is determined whether or not the characteristic amount I _S exceeds the threshold value stored in the threshold value storage register Max. If such a pixel is found, in step S244, in order to set the feature value as a new threshold value, in step S244,

[0052]

(Equation 7)

Assume that In step S246, the most mallet-like pixels (I _x , I _x
The coordinate value of _y ) is passed to step S220. In this manner, the mallet is found in the image, and its coordinate value is converted to a table coordinate system and passed to the game state management unit 5030. <Game State Management> FIG. 13 shows a game field of the air hockey game of the present embodiment. This field is defined on a two-dimensional plane above the table 1000,
It has x and y axes. Further, it has two left and right virtual goal lines 1200L and 1200R and virtual walls 1300a and 1300b provided in the vertical direction in FIG. The coordinate values of the virtual goal lines 1200L and 1200R and the virtual walls 1300a and 1300b are known, and do not move. In this field, the virtual image of the pack 1500 moves in accordance with the movement of the mallets 260R and 260L.

The pack 1500 contains the coordinate information P of the current position.
and a _p and velocity information v _p, left mallet 260L has coordinate information P _SL and velocity information v _SL of the current position, and the right mallet 260R has coordinate information P _SR and the speed information v _SR of the current position Have. FIG. 14 is a flowchart illustrating a processing procedure in the game state management unit 5030.

In step S10, pack 1500
The initial position P _p0 and the initial speed v _p0 are set. In addition, the pack performs a uniform motion at a speed v _p. Also, the pack makes a full elastic collision when hitting a wall or stick, i.e.
The speed direction is reversed. The game state management unit 5030 obtains speed information v _S from the position information P _{S of} each mallet measured by the mallet position measurement unit 5010.

Step S12 is executed at intervals of .DELTA.t until a win or loss in the game is determined (one of the points precedes three points in step S50). Then, step S12
Then, the position of the pack is

[0057]

(Equation 8)

Is updated to The position of the pack after setting the initial position and the initial speed is generally

[0059]

(Equation 9)

Is represented by In step S14, the updated pack position _Pp is set to the # 1 side of the player (left player).
To see if it is in the field. The case where the puck 1500 is on the left player side will be described. Step S
At 16, it is checked whether or not the current pack position is at a position where it interferes with the left player's stick 1100L. When the puck 1500 is located at the position where the puck 1500 interferes with the stick 1100L, it means that the left player 2000 has performed a mallet operation such that the mallet 260L collides with the puck.
In step S18, by reversing the sign of the x-direction velocity component of the velocity v _p of the pack 1500, the process proceeds to step S20.

Incidentally, instead of simply inverting the sign of the velocity component in the x direction of the velocity v _p ,

[0062]

(Equation 10)

Alternatively, the puck may be moved in the opposite direction with the stick operation speed superimposed. on the other hand,
The current pack position is the left player's stick 1100L
If it is not at a position where it interferes with (NO in step S16), the process proceeds to step S20. In step S20, the position P _{i + 1} of the pack is set to the virtual wall 1300a or 1303.
It is checked whether it is in a position where it collides with 0b. Step S
If the determination at 20 is YES, the y component of the speed of the pack is inverted in step S22.

Next, in step S24, it is checked whether or not the current pack position is within the goal line of the left player. If YES, the score of the opponent player, that is, the score of the right (# 2) player is added in step S26. In step S50, it is checked whether any score has scored three or more points in advance. If the score is three or more, the game ends. If it is determined in step S14 that the puck position _Pp is on the right player side (# 2 player side), step S30 is executed.
Do the following: Steps S30 to S40 are:
The operation is substantially the same as steps S16 to S26.

Thus, the progress of the game is managed. The progress of the game is the position of the puck and the position of the stick. As described above, the image generation unit 5050 (5
050L, 5050R). <Correction of Head Position> FIG.
(5040L, 5040R) shows the entire control procedure of the process. The correction in the correction processing unit 5040 means that the viewpoint position and head posture data obtained by the magnetic sensor considered to have an error are converted into the position (head position) of the camera 240 by the marker position in the image obtained from the CCD camera 240. (Which is also closely related to the position of the part), and finally corrects by changing the viewing transformation matrix of the viewpoint using the correction value.

That is, in step S400, a camera viewing transformation matrix (4 × 4) is calculated based on the output of the magnetic sensor 220. In step S410, the observation coordinates of each marker are predicted based on the ideal perspective transformation matrix (known) of the camera 240 and the three-dimensional position (known) of each marker. The marker position detecting unit 5060 (5060L, 5
060R) tracks the marker in the image obtained from the camera 240 (240L, 240R) attached to the head of the player. Therefore, the marker position detection unit 5060 passes the detected marker position to the correction processing unit 5040 (in step S420). The correction processing unit 5040 (50
40L, 5040R) in step S420
Based on the passed marker position information, a marker being observed, that is, a marker serving as a reference for correction is determined. In step S430, a correction value of the position of the camera 240 is calculated, and based on the correction value, the viewing transformation of the viewpoint is corrected in step S440, and the corrected transformation matrix is output to the image generation unit 5050 (5050L, 5050R). ).

FIG. 17 shows a processing procedure in the marker position detecting section 5060. In step S500, the camera 2
40 takes in the acquired color image. Thereafter, in step S502, “local area search” is performed, and in step S506, “global area search” is performed to detect a marker position (x, y) represented by the camera coordinate system. The “local area search” in step S502 and the “global area search” in step S506 are performed as “local area search” (9th step) in mallet search.
FIG. 10) and “global area search” (FIG. 10) are substantially the same, and are not shown.

It should be noted that the feature value I _S for the marker search (in step _S 232) is the value of the pixel value of the pixel of interest for player # 1 (left).

[0069]

[Equation 11]

Is used. For player # 1, red is used for markers (1600-1604),
This feature quantity represents the degree of redness. Player #
Green marker (1650-165) for 2 (right)
Since 4) is used,

[0071]

(Equation 12)

Is used. The above two quantities are also used for the feature quantity I _s (x, y) in the global search. In step S510, the coordinate values of the markers obtained in steps S502 and S506 are converted into an ideal image coordinate system without distortion using a matrix M (for example, having a size of 3 × 3) for correcting distortion. Convert to The conversion formula at this time is

[0073]

(Equation 13)

Is as follows. Next, step S4 in FIG.
The details of the process 10 will be described with reference to FIG. As described above, in step S400, a transformation matrix M _C (4 × 4 viewing transformation matrix) from the world coordinate system to the camera coordinate system is obtained. On the other hand, a transformation matrix P _C (4 × 4) from the camera coordinate system to the image coordinate system is also given as a known value. The three-dimensional coordinate position (X, Y, Z) of the marker of interest is also given as known.

As is well known, the angle r is a roll in the Z-axis direction at the position of the camera 240, and the angle p is
If the rotation in the X-axis direction (pitch) at the position of 0 and the angle φ is the rotation (yaw) in the Z-axis direction at the position of the camera 240,

[0076]

[Equation 14]

Where d is the focal length of the camera 240, w
The width of the imaging surface of the camera, if the same height h, P _C
Is

[0078]

(Equation 15)

Is represented by In step S520, the coordinate position (X, Y, Z) of the marker of interest is converted into a position (x _h , y _h , z _h ) in the image coordinate system according to the following equation.

[0080]

(Equation 16)

In step S 522, the observed observation positions x and y of the markers are set as

[0082]

[Equation 17]

Is obtained. Next, the process of “marker determination” in step S420 will be described. FIG. 19 shows that the camera 24 of one player
0 indicates the case where the image 600 is acquired. Table 100
The marker is provided on the 0, for example, the M ₁ ~M _7,
It is represented by a symbol. Three-dimensional position M _i of this marker is known. The image 600 includes markers M ₂ , M ₃ , M ₆ , and M ₇ . On the other hand, observation prediction position of each marker M _i are those calculated in step S520, that it and P _i. Further, Q is detected by the marker position detection unit 5060,
The marker position passed from the detection unit 5060 is shown.

The “judgment of marker” in step S420 is as follows.
The marker position Q detected by the marker position detection unit 5060 is
It is to determine which P _i (that is, which M _i ) corresponds. In FIG. 19, the vector e _i represents the length of the vector from the detected marker position Q to the predicted position P _i of each marker, that is, the distance. FIG. 20 shows the details of step S420. That is, the process of FIG. 20, of the distance e _i marker i entering image 6000 (i = 0 to n), and searches for a marker indicating the minimum value, and outputs the identifier i of the marker . That is,

[0085]

(Equation 18)

Is as follows. In the example of Fig. 19, since the shortest distance e ₂ of between P _2, and data using the marker M ₂ to the correction of the magnetic sensor output. Thus, no matter how the player moves, within their field of activity,
Since the camera 240 captures any one or more markers in the image, it is not necessary to restrict the size of the field to a narrow size as in the related art.

Steps S430 and S440
Is the same as the processing described in FIG. <Modification 1> The present invention is not applied only to the above embodiment. In the above embodiment, in the process of detecting a marker in an image, as shown in FIG. 17, the marker found first is set as the marker to be tracked. Therefore, for example, as shown in FIG. 21, the markers M ₁ in a frame
Is obtained, the marker M _i is corrected in the image area 810 of the subsequent frame if the marker is located at the end of the area 810 but is included in the area 810. There is no inconvenience in determining the reference marker. However, in a subsequent frame, for example, an image 820 is obtained, and the marker M _i falls within the area, and instead includes the marker M ₂ , the reference marker for correction is changed to the marker M _2. I have no choice. Such a change of the marker is necessary even when tracking fails, and a newly tracked marker is used for correcting the positional deviation.

As a problem of switching the marker used for correction as described above, a virtual object may move unnaturally at the time of the switching due to a sudden change in the correction value. . Therefore, in order to maintain the temporal consistency of the correction values, it is proposed as a modification that the correction values up to the previous frame are reflected in the setting of the next correction value.

That is, assuming that a correction value (a three-dimensional vector representing a parallel movement in the world coordinate system) in a certain frame is v ^t , and a correction value in a previous frame is ^v′t−1 , the following equation is obtained. Let ^v′t be a new correction value.

[0090]

[Equation 19]

Here, α is a constant of 0 ≦ α <1 which defines the degree of influence of past information. The above equation means that the contribution based on the correction value v′t ^-1 in the previous frame is α
And the correction value v ^t obtained in the current frame is (1−
α). By doing so, a sudden change in the correction value is reduced, and a sudden change (unnatural movement) of the three-dimensional virtual image is eliminated. By setting the new correction value α to an appropriate value, it is possible to prevent unnatural movement of the object due to switching of the marker.

<Modification 2> In the above embodiment, the process of detecting a marker in an image is performed, as shown in FIG. 17, when the marker cannot be found by the local search, the position of the marker in the previous frame. Regardless of this, the point with the highest similarity in the entire screen is used as the marker to be tracked. Here, a modified example in which the marker search is performed centering on the position of the marker found in the previous frame is proposed. This is because, even if the image frame moves along with the movement of the player, there is a high possibility that the marker exists at a position that is not largely shifted from the position existing in the previous frame.

FIG. 22 explains the principle of searching for a marker found in the previous frame in the current frame. When a search is performed along such a search path and a point having a similarity higher than a certain threshold is found, this point is used as a marker to be tracked. <Modification 3> Although the above embodiment uses the optical HMD, the present invention is not limited to the application of the optical HMD, but can be applied to a video see-through HMD.

<Modification 4> Although the above embodiment is applied to an air hockey game, the present invention is not limited to an air hockey game. According to the present invention, since a plurality of works (for example, mallet operation) are imaged and captured by one camera, it is possible to reproduce the works of the plurality of persons in one virtual space. Therefore, the present invention is also suitable for an embodiment of a cooperative work (for example, an MR presentation of a design work by a plurality of persons or a competitive game of a plurality of persons) on the premise of two or more workers.

The processing for correcting the head posture position based on a plurality of markers according to the present invention is not suitable only for cooperative work by a plurality of persons. One worker (or player)
It is also applicable to a system that presents mixed reality.

[0096]

As described above, according to the optical apparatus of the present invention, a camera for acquiring an image in front of the worker is mounted at the position of the worker's head. Even when the three-dimensional position of the worker fluctuates, the posture of the worker can be measured with high accuracy.

[Brief description of the drawings]

FIG. 1 is a view for explaining the principle of camera position correction applied to a conventional technique and to an embodiment of the present invention.

FIG. 2 is a side view showing the configuration of the game device used in the embodiment of the present invention.

FIG. 3 is a view for explaining a scene seen in the field of view of a left player on the game device of FIG. 2;

FIG. 4 is an HMD used in the game device of FIG. 2;
FIG.

FIG. 5 is a view for explaining an arrangement of markers provided on a table of the game device in FIG. 2;

FIG. 6 is a view for explaining changes in markers included in an image captured by a camera mounted on the player's head as the player moves on the table in FIG. 5;

FIG. 7 is an exemplary view for explaining the configuration of a three-dimensional image generation device for the game device according to the embodiment;

FIG. 8 is a flowchart illustrating a processing procedure performed by a mallet position measurement unit according to the embodiment.

FIG. 9 is an exemplary flowchart illustrating a subroutine (local search) of a processing procedure performed by the mallet position measurement unit according to the embodiment;

FIG. 10 is an exemplary flowchart illustrating a subroutine (global search) of a processing procedure performed by the mallet position measurement unit according to the embodiment;

FIG. 11 is a view for explaining division of a processing target area used in the processing of the flowchart in FIG. 8;

FIG. 12 is a diagram showing a setting method of a target area used in the processing of the flowchart in FIG. 8;

FIG. 13 is an exemplary view for explaining the configuration of a virtual game field in the game according to the embodiment;

FIG. 14 is a flowchart illustrating a control procedure of game management in a game state management unit according to the embodiment.

FIG. 15 is a diagram illustrating a technique for detecting a mallet.

FIG. 16 is an exemplary flowchart for entirely explaining the processing procedure of a correction processing unit according to the embodiment;

FIG. 17 is a flowchart for explaining in detail a part of the flowchart of FIG. 16 (tracking of a marker);

FIG. 18 is a flowchart for explaining in detail a part of the flowchart of FIG. 16 (prediction of a marker position);

FIG. 19 is a view for explaining the principle of detection of a marker serving as a reference used for correction.

FIG. 20 is a flowchart illustrating the principle of detection of a reference marker.

FIG. 21 is an exemplary view for explaining transition of reference markers applied to a modification of the embodiment;

FIG. 22 is an exemplary view for explaining the principle of marker search applied to a modification of the embodiment;

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Hiroyuki Yamamoto 6-145 Hanasaki-cho, Nishi-ku, Yokohama-shi Hanasaki Building, Yokohama, Japan

Claims (4)

[Claims] 1. An optical device mounted on a head of an optical see-through, comprising: a main body; a first optical system incorporated in the main body, for guiding an image to an eyeball; A head-mounted optical device comprising a camera having a field of view in front of the head. 2. The apparatus according to claim 1, further comprising a three-dimensional position and orientation sensor for detecting a three-dimensional position and orientation of the head, and a liquid crystal display panel for displaying the image. Head-mounted optical device. 3. The device according to claim 2, wherein the three-dimensional position sensor is a magnetic sensor that detects a change in a magnetic field, and the magnetic sensor is fixed to the main body while being spatially separated from the liquid crystal display unit. 2. The head-mounted optical device according to 1. 4. An optical see-through type optical device, comprising: a second optical system that is incorporated in the main body and guides light from an external environment to an eyeball; and a display unit that displays the image. The head-mounted type according to claim 1, wherein the eyeball receives light of the image from the first optical system and light of an environment from the second optical system. Optical device.