JP3486536B2 - Mixed reality presentation apparatus and method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mixed reality presentation device for presenting a cooperative work by a plurality of workers as mixed reality, such as a battle game.

[0002]

2. Description of the Related Art In recent years, mixed reality (hereinafter referred to as "MR" (Mixed
(Reality)) has become popular.
MR can be experienced only in a situation where it has been separated from the physical space in the past.
For the purpose of coexistence of the world and the real space, it is attracting attention as a technology for enhancing VR.

The applications of MR include the use of medical assistance in which doctors are presented to see through the inside of the patient's body,
It is expected that new fields will be qualitatively completely different from the VRs used up to now, such as the use of work assistance in which the assembly procedure of products is displayed on the actual product in a factory. The applications of MR include medical assistance that presents the doctor with a perspective of the patient's internal state, and work assistance that displays the assembly procedure of the product on the actual product at the factory. A new field that is completely qualitatively different from VR is expected.

What is commonly required for these applications is a technique for removing the "deviation" between the real space and the virtual space. "Misalignment" can be classified into misalignment, time misalignment, and qualitative misalignment. Among them, many efforts have been made to solve misalignment, which is the most basic requirement. Video See-Through (Vid
In the case of eo-See-Through) MR, the problem of alignment results in the problem of accurately obtaining the three-dimensional position of the video camera.

MR of an optical see-through method using a semi-transmissive HMD (Head Mount Display)
The problem of alignment in the case of
Although it is a problem to obtain the three-dimensional position, the use of a three-dimensional position and orientation sensor such as a magnetic sensor, an ultrasonic sensor, or a gyro is generally used as the measuring method, but the accuracy of these is not always sufficient and the error May cause misalignment.

On the other hand, in the case of the video see-through system,
A method of directly performing alignment on an image based on image information without using such a sensor is also conceivable. With this method, the misregistration can be handled directly, so that the positioning can be performed accurately, but there are problems such as lack of real-time performance and reliability. In recent years, it has been reported that the position and orientation sensor and image information are used together to compensate for the defects of the both and realize accurate positioning.

As one attempt, "Dynamic Registrati
on Correction in Video-Based-Augmented Reality Sys
tems '' (Bajura Michael and Ulrich Neuman, IEEE Compute
r Graphics and Applications 15, 5, pp. 52-60, 199
5) (hereinafter referred to as the first document) proposed a method of correcting a position shift caused by an error of a magnetic sensor by image information in a video see-through MR.

In addition, "Superior Augmented Reality Reg
istration by Integrating Landmark Tracking and Mag
netic Tracking '' (State Andrei et al., Proc. of SIGGRAPH
96, pp. 429-438, 1996) (hereinafter referred to as the second document),
We further developed this method and proposed a method to compensate the ambiguity of position estimation by image information by sensor information. In the above-mentioned conventional example, when a video see-through MR is constructed using only the position and orientation sensor, the positional deviation on the image occurs due to the error of the sensor, and the positional deviation is detected as image information. In order to detect from the above, a landmark whose three-dimensional position is known is set in the real space as a clue.

Assuming that the sensor output does not include an error, the coordinates Q _{I of the} landmark actually observed on the image
And the observed observation coordinates P _{I of the} landmark derived from the camera position obtained based on the sensor output and the three-dimensional position of the landmark should be the same. But,
In reality, the camera position obtained based on the sensor output is not accurate, so Q _I and P _I do not match. This P _I and Q _I
The deviation represents the positional deviation between the virtual space and the real space at the landmark position. Therefore, by extracting the landmark position from the image, the direction and size of the deviation can be calculated.

As described above, by quantitatively measuring the positional deviation on the image, it becomes possible to correct the camera position so as to eliminate the positional deviation. The simplest alignment method that uses both the orientation sensor and the image is considered to be the correction of the sensor error using one landmark, and the camera position is translated or rotated according to the displacement of the landmark on the image. The method is proposed by the first document.

FIG. 1 shows the position using one landmark.
The basic idea of misregistration correction is shown below. Below, the camera
Excludes effects such as distortion by making the internal parameters of
The image is taken by the ideal imaging system
Suppose. The viewpoint position of the camera is C, and the run on the image
Q is the observation coordinate of the mark_I, The landmark of real space
Set Q_IThen point Q_IIs point C and point Q_IStraight line l_QUp
Exists in. On the other hand, given by the position and orientation sensor
From the camera position, the landmark in the camera coordinate system
Position P_CAnd the observation coordinates P on the image_ICan be guessed.
Below, point C to point Q_I, P_IA three-dimensional vector to
Each v_l, V₂It is written as. In this method, the corrected
Landmark observation prediction coordinate P^' _IIs Q_IWill match
Sea urchin (that is, the corrected land in the camera coordinate system)
Mark predicted position P ^' _CBut the straight line l_QTo get on),
By correcting the relative position information of the camera and the object,
The positional deviation is corrected.

It is considered that the displacement of the landmark is corrected by rotating the camera position. This can be realized by correcting the position information of the camera so that the camera rotates by the angle θ formed by the two vectors v _l and v ₂ . In the actual calculation, using the vectors v _ln and v _2n obtained by normalizing the above vectors v _l and v ₂ , their outer product v _ln × v _2n is the rotation axis, and the inner product v _1n · v _2n is the rotation angle, and the points Rotate the camera around C.

It is considered that the displacement of the landmark is corrected by the relative translation of the camera position. This can be realized by translating the object position in the virtual world by v = n (v ₁ −v ₂ ). Here, n is a scale factor defined by the following equation.

[0014]

[Equation 1]

Here, | AB | is a symbol indicating the distance between points A and B. The same correction can be performed by correcting the position information of the camera so that the camera moves in parallel by -v. This is because the virtual object is relatively moved by v by this operation. The above two methods are methods of matching the positional deviation on the landmark two-dimensionally, and the camera position cannot be corrected to a three-dimensionally correct position. However, when the sensor error is small, a sufficient effect can be expected. In addition, the calculation cost for correction is very small, and the method is excellent in real time.

[0016]

However, the above-mentioned document does not take into consideration the case where the workers perform a plurality of collaborative works, and therefore, provides only a mixed reality presentation system by a single worker. could not.

[0017]

SUMMARY OF THE INVENTION The present invention has been made in view of such a situation, and an object thereof is to allow a cooperative work by a plurality of workers to be presented by mixed reality. Further, it is an object of the present invention to enable the collaborative work to be easily managed and to generate a virtual image from the viewpoint of each worker with high accuracy.
In order to achieve the above object, the mixed reality presentation device of the present invention has the following configuration. That is, in order for a plurality of workers to perform a predetermined work while sharing a virtual object in a predetermined mixed reality environment, a three-dimensional virtual image of the virtual object is generated and the plurality of workers are A mixed reality presentation device for displaying on a see-through type display device attached to each of the
Detecting means for detecting individual positions of the working parts of the plurality of workers used in a predetermined work in the same coordinate system by image processing from the same photographed image; Using the sensor for detecting the viewpoint position of each worker and the viewpoint position detected by the sensor, a viewing conversion coefficient for converting from the world coordinate system to the camera coordinate system is obtained, and the head of each worker is calculated. Correction processing means for correcting the viewing conversion coefficient for each of the plurality of workers by correcting the viewing conversion coefficient using a captured image of a camera fixed to the position; and detecting in the same coordinate system a progress management means for managing the progress of the predetermined work by using a plurality of workers work part individual positions of which are being managed by the progress management unit Serial 3D showing a virtual object to be shared
A virtual image is generated for each of the plurality of workers by using the corrected viewing coefficient obtained for each of the plurality of workers, and is output to the see-through display device of each of the plurality of workers. And generating means.

[0018]

[0019]

[0020]

[0021]

[0022]

[0023]

BEST MODE FOR CARRYING OUT THE INVENTION A system according to an embodiment in which a mixed reality presentation method and an HMD of the present invention are applied to an air hockey game device will be described below. An air hockey game is a battle-type game in which an opponent exists, and normally, compressed air is supplied from the bottom to float the puck, and the puck is played against each other. In this game, the winner is the one with the most points.
In the air hockey game to which the MR of the present embodiment is applied, the puck is superimposed on a table in the real environment as a virtual three-dimensional image, presented to the player, and the real mallet is used for a fight.

<Structure of Game Device> FIG. 2 is a side view of the game device portion of the system of this embodiment. A mixed reality air hockey game at Table 1
Two opponents 2000 and 3000 face each other with a mallet (260L, 260R) in their hands with 000 in between. Two opponents 2000 and 3000 wear head mount displays 210L and 210R (hereinafter abbreviated as HMD) on their heads. The mallet of this embodiment has an infrared light emitter at its tip. In the present embodiment, the mallet position is detected by image processing, but if the shape and color of the mallet have characteristics, detection by pattern recognition using those characteristics is also possible.

The HMD 210 of the embodiment is a see-through type as shown in FIG. Both opponents 2000,300
0 can observe the surface of the table 1000 even when the HMDs 210L and 210R are mounted. HMD
A three-dimensional virtual image is input to 210 from an image processing system described later. Therefore, the opponent 2000,3000
The three-dimensional image displayed on the display screen of the HMD 210 is viewed by superimposing it on the image of the real space that has passed through the optical system (not shown in FIG. 2) of the HMD 210.

FIG. 3 shows an image viewed from the HMD 210L of the left player 2000 by the left player 2000. Two players strike a virtual puck 1500. The actual mallet 260L held by the player 2000 is used to hit the puck 1500. The player 2000 holds the mallet 260L in his hand. The goal 1200R is visible just before the opponent player 3000. Image processing system (3rd
3D CG is generated and displayed on the HMD 240L so that the goal 1200R can be seen near the opponent.

The player 3000 for the HMD 210
Goal 1200L near player 3000 via R
Will be seen. The pack 1500 is also generated by an image processing system (not shown) and displayed on each HMD. <HMD with Magnetic Sensor> FIG. 4 shows the configuration of the HMD 210. The HMD 210 is disclosed in, for example, Japanese Patent Laid-Open No. 7-33.
The magnetic sensor 220 is attached to the main body of the HMD No. 3551 via a column 221. In the figure, reference numeral 211 is an LCD display panel. Light from the LCD display panel enters the optical member 212, is reflected by the total reflection surface 214, is reflected by the total reflection surface of the concave mirror 213, is transmitted through the total reflection surface 214, and is visible to the observer's eyes. Reach.

The magnetic sensor 220 is, in this embodiment, Po
A magnetic sensor Fastrack manufactured by lhemus was used. Since the magnetic sensor is weak against magnetic noise, it is separated from the display panel 211 and the camera 240, which are noise sources, by the support column 221. The configuration of attaching the magnetic sensor and / or the camera to the HMD shown in FIG. 4 is not limited to the optical see-through HMD, and the magnetic sensor and / or the camera may be attached to the video see-through HMD. , Can be attached to the HMD for the purpose of accurately detecting the head position and posture.

In FIG. 2, each HMD 210 is fixed to the player's head by a band (not shown).
Each player's head has a magnetic sensor 220 as shown in FIG. 4 and a CCD camera 240 as shown in FIG.
(240L, 240R) are fixed.
The field of view of the camera 240 is set in front of the player. In the case of the air hockey game, since the upper surface of the table 1000 is to be viewed, the camera 240 also captures an image of the surface of the table 1000. The magnetic sensor 220 (220L, 220R) is an AC magnetic field generation source 25.
Senses changes in the AC magnetic field generated by 0.

When the player looks diagonally downward to see the surface of the table 1000, the view through the HMD 210 shows the surface of the table 1000 and the virtual puck 1 described above.
500, real mallet 260 (260L, 260
R), virtual goal 1200 (1200L, 1200
R) is visible. When the player moves his head horizontally within a horizontal two-dimensional plane, or makes a tilting motion, a yaw motion, or a rolling motion, the change is first detected by the magnetic sensor 220.
This is observed as a change in the image captured by the CCD camera 240 as the posture of the head changes.

<Plurality of Markers> Each mallet 260 has an infrared ray emitter at its tip, and the two-dimensional plane position of the mallet can be known by the CCD camera 230 which detects this infrared ray. CCD camera 24
0 outputs an image called a marker image.

FIG. 5 shows an example of markers arranged on the table 1000. In FIG. 5, 5 indicated by a circle
One landmark or marker (1600 to 1604) is a marker used for auxiliary detection of the head position of the player 2000, and five landmarks or markers (1650 to 1654) shown by □ are the player 30.
The marker used for auxiliary detection of the head position of 00 is shown. When a plurality of markers are arranged in this way, which marker is visible is determined by the position of the head, particularly the posture. In other words, the CCD mounted on each player.
If the marker in the image captured by the camera 240 can be specified and the position in the image can be detected, the marker can be used to correct the output signal of the magnetic sensor that detects the head posture of the player.

Marker groups (1600 to 1) assigned to two players (2000, 3000), respectively.
608) and the marker groups 1650 to 1658) are colored in different colors. In the present embodiment, the marker for the left player (# 1 player) is colored red and the marker for the right player (# 2 player) is colored green. This is because it is easy to distinguish markers in image processing.

A major feature of this embodiment is that a plurality of markers are arranged. The plurality of arrangements ensures that at least one marker is within the field of view of the CCD camera 240 as long as the player acts on the table 1000 within the movement range of the air hockey game. FIG. 6 illustrates a state in which the image processing range for detecting a marker moves as the head moves when the player moves the head variously. As shown in the figure, one image contains at least one marker. In other words, the number of markers, the interval between markers, etc. should be set according to the size of the table 1000, the viewing angle of the camera 240, and the size of the moving range of the player based on the nature of the game. In this case, the farther from the player, the wider the field of view, so the interval between the markers may be increased. This is because the distance in the image between the markers in the vicinity is equal to the distance in the image between the markers in the distance. This is to prevent unnecessary imaging of a plurality of markers in the same frame.

<MR Image Generating System> FIG.
3 shows the configuration of a three-dimensional image generation / presentation system in the game device shown in the figure. This image generation / presentation system displays three-dimensional virtual images (pack 1500, goal 120 in FIG. 3) on the display devices of the HMD 240L of the left player 2000 and the HMD 240R of the right player 3000, respectively.
0) is output. Generation of a three-dimensional virtual image is performed by the image generation units 5050L and 5050R. In this embodiment, each of the image generation units 5050 is
A computer system ONYX2 manufactured by Silicon Graphics was used.

The image generation unit 5050 inputs the pack position information and the like generated by the game state management unit 5030 and the corrected viewpoint position / head direction information generated by the two correction processing units 5040L and 5040R. Game state management unit 5030 and correction processing units 5040L and 5040R
Each of the is configured by the computer system ONYX2.

The CCD camera 230 fixed above the center of the table 1000 covers the entire surface of the table 1000 in the field of view. The mallet information acquired by the camera 230 is input to the mallet position measuring unit 5010. This measuring unit 5010 is also O2 manufactured by Silicon Graphics
Comprised of computer systems. Measuring unit 501
0 detects the mallet positions of the two players, that is, the positions of the hands. Information regarding the position of the hand is input to the game state management unit 5030, where the game state is managed. That is, the game state and the progress of the game are basically all determined by the position of the mallet.

A position / orientation detecting unit 5000, which is composed of a computer system O2 manufactured by Silicon Graphics, inputs the outputs of the two magnetic sensors 220L and 220R, detects the viewpoint position and head and orientation of each player, and a correction processing unit. Output to 5040L and 5040R. On the other hand, the CCD cameras 240L and 240R fixed to the heads of the respective players acquire marker images, and the marker images are processed by the marker position detecting units 5060L and 5060R, respectively.
The position of the marker within the field of view of each camera 240 is detected. The correction processing unit 5 provides information about the marker position.
040 (5040L, 5040R).

Here, two marker position detectors 5060
(5060L, 5060R) consisted of O2 computer system. <Mallet Position Measurement> FIGS. 8 to 10 are flowcharts showing the control procedure for measuring the mallet position.

In an air hockey game, a player does not advance his mallet to the area of another player. Therefore, in the process of searching the mallet 260L (260R) of the left player 2000 (right player 3000), if the process is concentrated on the image data I _L (image data I _R ) of the left field as shown in FIG. Good.
It is easy to divide the image acquired by the CCD camera 230 at the fixed position into two as shown in FIG.

Therefore, in the flow chart of FIG. 8, the mallet 26 of player # 1 (player 2000) is
For 0L search, in step S100, player #
The search for the mallet 260R of 2 (player 3000) is performed in step S200. Therefore, for convenience, a search for the mallet of the right player (step S20
0) will be described as an example.

First, in step S210, the TV camera 23
Obtain a multi-valued image from 0. In step S212, a subroutine "search in local area" is performed on the right half image data I _R. The details are shown in FIG.
When the coordinates (x, y) of the mallet position in the image coordinate system are found in step S212, the process proceeds from step S214 to step S220, and the mallet position coordinates (x, y) in the image coordinate system are converted to the coordinates of the table 1000 according to the following formula. Convert to the coordinate position (x ', y') of the system (see FIG. 13).

[0043]

[Equation 2]

Here, M _T is a 3 × 3 conversion matrix for calibrating the image coordinate system and the table coordinate system, and is known. The coordinate position (x ', y') obtained in step S220 is sent to the game state management unit 5030. If no mallet is found in the local area, a "search in the global area" is performed in step S216. If the mallet is found in the "search in the global area", its coordinate position is converted into the table coordinate system in step S220. The coordinate position searched in the local or global area is used for searching the mallet in the local area in the next frame.

FIG. 9 shows the processing for searching the mallet in the local area (details of step S212). However, although this processing shows the search processing in the right field for convenience, the same applies to the mallet search processing in the left field. In step S220, a rectangular area of size (2A + 1) × (2B + 1) pixels defined by the following equation is extracted.

[0046]

[Equation 3]

Here, I _x and I _y are arbitrary coordinate values in the search area I _R , A and B are constants for determining the size of the search area, and the search area is as shown in FIG. become.
Step S230 is a step of extracting, from all the pixels (x, y) in the rectangular area defined in step S220, the evaluation value I _S (x, y) of the characteristic satisfying a certain condition. For the purpose of searching the mallet, the feature amount is preferably the similarity of pixel values (intensity value of infrared light).
In the present embodiment, since the infrared ray emitter is used for the mallet, the one having the characteristic of the intensity of the infrared light is temporarily determined to be the mallet.

That is, in step S232, the similarity I _S
Finds pixels that are closer to the mallet than a predetermined threshold. When such a pixel is found, the cumulative value of the occurrence frequency is stored in the counter N. Further, the x coordinate value and the y coordinate value of such a pixel are cumulatively stored in the registers SUMx and SUMy. That is,

[0049]

[Equation 4]

It is assumed that When step S230 ends,
In the region of FIG. 12, the number N of all pixels similar to the pattern of infrared light from the mallet and the cumulative value of coordinate values
SUMx and SUMy are obtained. If N = 0, step S23
At 6, the result “Not Found” is output. If N> 0, a mallet-like thing is found, and in step S238, the position (I _x , I _y ) of the mallet is

[0051]

[Equation 5]

Calculate according to Then, the coordinate values obtained by converting the mallet position (I _x , I _y ) into the table coordinate system are passed. FIG. 10 shows the detailed procedure of the global area search in step S216. In step S240 of FIG.
In the image I _R in the right field,

[0053]

[Equation 6]

Among the pixels that satisfy the condition, the feature evaluation value Is
The maximum value of is stored in the register Max. Here, C and D are constants that determine the roughness of the search, and Width and Height are defined in FIG. That is, step S242
Then, it is determined whether the feature value I _S exceeds the threshold value stored in the threshold value storage register Max. If such a pixel is found, in step S244, in order to set the feature amount as a new threshold value, in step S244,

[0055]

[Equation 7]

It is assumed that In step S246, the most mallet-like pixel (I _x , I found in the global search).
The coordinate value of _y ) is passed to step S220. In this way, the mallet is found in the image, and its coordinate values converted into the table coordinate system are passed to the game state management unit 5030. <Game State Management> FIG. 13 shows the game field of the air hockey game of this embodiment. This field is defined on the two-dimensional plane above table 1000,
It has x and y axes. Further, it has two left and right virtual goal lines 1200L and 1200R, and virtual walls 1300a and 1300b provided in the vertical direction in FIG. The coordinate values of the virtual goal lines 1200L and 1200R and the virtual walls 1300a and 1300b are known and do not move. In this field, the virtual image of the pack 1500 moves in accordance with the movement of the mallets 260R and 260L.

The pack 1500 has coordinate information P of the current position.
and a _p and velocity information v _p, left mallet 260L has coordinate information P _SL and velocity information v _SL of the current position, and the right mallet 260R has coordinate information P _SR and the speed information v _SR of the current position Have. FIG. 14 is a flowchart illustrating a processing procedure in the game state management unit 5030.

In step S10, the pack 1500
The initial position P _p0 and the initial velocity v _p0 are set. Incidentally, the puck performs constant velocity motion at a velocity v _p . The puck also has a full elastic collision when hit against a wall or stick, i.e.
The speed direction is reversed. The game state management unit 5030 obtains the speed information v _S from the position information P _{S of} each mallet measured by the mallet position measurement unit 5010.

Step S12 is executed every Δt time until the win or loss in the game is determined (one wins three points in step S50). Then, step S12
So the position of the pack is

[0060]

[Equation 8]

Is updated to The position of the puck after setting the initial position and initial speed is generally

[0062]

[Equation 9]

It is represented by In step S14, the updated pack position P _p is the player's # 1 side (left player).
Check if it is in the field. A case where the pack 1500 is on the left player side will be described. Step S
At 16, it is checked whether or not the current puck position is in a position where it interferes with the stick 1100L of the left player. The fact that the puck 1500 is in a position where it interferes with the stick 1100L means that the left player 2000 has performed a mallet operation that causes the mallet 260L to collide with the puck. Therefore, in order to reverse the movement of the puck 1500,
In step S18, the sign of the x-direction velocity component of the velocity v _p of the pack 1500 is inverted, and the process proceeds to step S20.

Incidentally, instead of simply inverting the sign of the velocity component in the x direction of velocity v _p ,

[0065]

[Equation 10]

As an alternative, the puck may move in the opposite direction with the stick operating speed superimposed. on the other hand,
Current puck position is left player's stick 1100L
When it is not in a position where it interferes with (NO in step S16), the process directly proceeds to step S20. In step S20, the position P _{i + 1} of the pack is set to the virtual wall 1300a or 130.
It is checked whether or not it is in a position where it collides with 0b. Step S
If the determination in 20 is YES, the y component of the pack speed is inverted in step S22.

Next, in step S24, it is checked whether or not the current puck position is within the goal line of the left player. If YES, the score of the opponent player, that is, the right (# 2) player is added in step S26. In step S50, it is checked whether any one of the scores is three points or more. If the score is 3 or more, the game ends. If it is determined in step S14 that the puck position P _p is on the right player side (# 2 player side), step S30
Do the following: Steps S30 to S40 are
The operation is substantially the same as steps S16 to S26.

Thus, the progress of the game is managed. The progress of the game is the position of the pack and the position of the stick, and as described above, the image generation unit 5050 (5
050L, 5050R). <Correction of Head Position> FIG. 16 shows a correction processing unit 5040.
The whole control procedure of the process in (5040L, 5040R) is shown. The correction in the correction processing unit 5040 means the viewpoint position and the head posture data obtained by the magnetic sensor, which is considered to have an error, from the position of the camera 240 (head) by the marker position in the image obtained from the CCD camera 240. The value is also closely related to the position of the part), and the correction value is used to finally correct the viewing transformation matrix of the viewpoint.

That is, in step S400, the viewing transformation matrix (4 × 4) of the camera is calculated based on the output of the magnetic sensor 220. In step S410, the observation coordinates of each marker are predicted based on the ideal perspective transformation matrix (known) of the camera 240 and the three-dimensional position (known) of each marker. Marker position detection unit 5060 (5060L, 5
060R) is tracking the marker in the image obtained from the camera 240 (240L, 240R) attached to the player's head. Therefore, the marker position detection unit 5060 passes the detected marker position to the correction processing unit 5040 (in step S420). Correction processing unit 5040 (50
40L, 5040R), in step S420,
Based on the passed marker position information, the marker being observed, that is, the marker serving as a reference for correction is determined. In step S430, a correction value for the position of the camera 240 is calculated, and based on this correction value, in step S440, the viewing conversion of the viewpoint is corrected, and the corrected conversion matrix is used as the image generation unit 5050 (5050L, 5050R). ) To.

FIG. 17 shows a processing procedure in the marker position detecting unit 5060. In step S500, the camera 2
The color image acquired by 40 is captured. After that, in step S502, "local area search" is performed, and in step S506, "global area search" is performed to detect the marker position (x, y) represented by the camera coordinate system. The “local area search” in step S502 and the “global area search” in step S506 are, as a procedure, “local area search” in mallet search (9th area).
(Fig.) And "Global area search" (Fig. 10) are substantially the same, so illustration is omitted.

However, as the feature quantity I _S (in step S232) for the marker search, of the pixel value of the pixel of interest for player # 1 (left),

[0072]

[Equation 11]

Is used. Since red is used for the markers (1600 to 1604) for player # 1,
This feature amount represents the degree of redness. Also, player #
For 2 (right), green markers (1650 to 165)
Since 4) is used,

[0074]

[Equation 12]

Is used. The above two quantities are also used for the feature quantity I _S (x, y) in the global search. The marker coordinate values obtained in steps S502 and S506 are processed in step S510 using a matrix M (for example, having a size of 3 × 3) for correcting distortion to generate an ideal image coordinate system without distortion. Convert to. The conversion formula at this time is

[0076]

[Equation 13]

It is Next, step S4 in FIG.
Details of the process of 10 will be described with reference to FIG. As described above, in step S400, the transformation matrix M _C (4 × 4 viewing transformation matrix) from the world coordinate system to the camera coordinate system is obtained. On the other hand, the transformation matrix P _C (4 × 4) from the camera coordinate system to the image coordinate system is also given as a known value. The three-dimensional coordinate position (X, Y, Z) of the marker of interest is also given as a known value.

As is well known, the angle r is the Z-axis rotation at the position of the camera 240, and the angle p is the camera 24.
When the rotation in the X-axis direction at the position of 0 (pitch) and the angle φ as the rotation in the Z-axis direction at the position of the camera 240 (yaw),

[0079]

[Equation 14]

Where d is the focal length of the camera 240 and w
Is the width of the imaging surface of the camera, and h is the same height, P _C
Is

[0081]

[Equation 15]

It is represented by In step S520, the coordinate position (X, Y, Z) of the target marker is converted into the position (x _h , y _h , z _h ) in the image coordinate system according to the following equation.

[0083]

[Equation 16]

At step S522, as the predicted observation position x, y of the marker,

[0085]

[Equation 17]

To obtain Next, the “marker determination” process in step S420 will be described. FIG. 19 shows the camera 24 of one player on the table 1000.
0 indicates a case where the image 600 is acquired. Table 100
The markers provided on 0 are, for example, M _{1 to} M ₇ ,
Represented by a triangle. The three-dimensional position M _i of this marker is known. The image 600 includes markers M ₂ , M ₃ , M ₆ , and M ₇ . On the other hand, the observation predicted position of each marker M _i is obtained in step S520, and is set as P _i . Further, Q is detected by the marker position detection unit 5060,
The marker position passed from the detection unit 5060 is shown.

The "marker determination" in step S420 is
The marker position Q detected by the marker position detection unit 5060 is
It is to determine which P _i (that is, which M _i ) corresponds to. In FIG. 19, the vector e _i represents the length of the vector from the detected marker position Q to the predicted position P _i of each marker, that is, the distance. The details of step S420 are shown in FIG. That is, the process of FIG. 20 searches for the marker having the minimum value among the distances e _i of the markers i (i = 0 to n) included in the image 6000 and outputs the identifier i of the marker. . That is,

[0088]

[Equation 18]

It is In the example of Fig. 19, since the shortest distance e ₂ of between P _2, and data using the marker M ₂ to the correction of the magnetic sensor output. Thus, no matter how the player moves, within their activity range (field)
Since the camera 240 captures any one or more markers in the image, it is not necessary to limit the size of the field narrowly as in the conventional case.

Incidentally, steps S430 and S440
The process in (1) is the same as the process described in FIG. <Modification 1> The present invention is not applied only to the above-described embodiments. In the above-described embodiment, in the process of detecting the marker in the image, the first one found is used as the tracking target marker, as shown in FIG. Therefore, for example, as shown in FIG. 21, the markers M ₁ in a frame
When the image 800 including the image is obtained, if the marker is included in the image area 810 of the subsequent frame, although the marker is included in the area 810 at the end portion of the area 810, the marker M _i is corrected. There is no inconvenience in determining the marker as the reference marker. However, in the subsequent frame, for example, when the image 820 is obtained, and the marker M _i is out of the area and includes the marker M ₂ instead, the reference marker for correction should be changed to the marker M _2. I have no choice. Such a change of the marker is necessary even when the tracking fails, and the newly tracked marker is used to correct the positional deviation.

As a problem of switching the marker used for the correction as described above, there is a case where the virtual object moves unnaturally due to the abrupt change of the correction value at the time of the switching. . Therefore, in order to maintain the temporal consistency of the correction value, it is proposed as a modification that the correction value up to the previous frame is reflected in the setting of the next correction value.

That is, when the correction value in a certain frame (three-dimensional vector representing the parallel movement in the world coordinate system) is v ^t and the correction value in the previous frame is ^v't-1 , it is calculated by the following equation. Let ^v't be a new correction value.

[0093]

[Formula 19]

Here, α is a constant of 0 ≦ α <1 which defines the degree of influence of past information. The meaning of the above equation is that the contribution degree due to the correction value v′t ⁻¹ in the previous frame is α
And the correction value v ^t obtained in this frame is (1-
It is used with the contribution of α). By doing so, the abrupt change in the correction value is alleviated, and the abrupt change (unnatural movement) of the three-dimensional virtual image is eliminated. By setting the new correction value α to an appropriate value, it is possible to prevent the unnatural movement of the object due to the switching of the marker.

<Modification 2> In the above embodiment, the process of detecting a marker in an image is performed by the marker position in the previous frame when the marker cannot be found by the local search as shown in FIG. Regardless of this, the point with the highest degree of similarity on all screens was used as the tracking target marker. Here, a modification is proposed in which the marker search is centered on the position of the marker found in the previous frame. This is because there is a high possibility that the marker will be present at a position that does not largely deviate from the position that was present in the previous frame, even if the image frame has moved as the player moves.

FIG. 22 explains the principle of searching the current frame for the marker found in the previous frame. When a point having a degree of similarity equal to or higher than a certain threshold value is found by performing a search on such a search route, this point is set as a tracking target marker. <Modification 3> Although the above embodiment uses the optical HMD, the present invention is not limited to the application of the optical HMD, and can be applied to a video see-through HMD.

<Variation 4> Although the above embodiment is applied to the air hockey game, the present invention is not limited to the air hockey game. In the present invention, the work of a plurality of people (for example, the mallet operation) is captured and captured by one camera means, so that the work of a plurality of people can be reproduced in one virtual space. Therefore, the present invention is also suitable for an embodiment of a collaborative work (for example, an MR presentation of a design work by a plurality of people or a battle game of a plurality of people) on the premise of two or more workers.

The process of correcting the head posture position based on a plurality of markers according to the present invention is not suitable only for the cooperative work of a plurality of people. One worker (or player)
It is also applicable to a system that presents mixed reality to.

[0099]

According to the present invention, cooperation by a plurality of workers
Work can be presented in mixed reality. Furthermore
The work of the plurality of workers used in the cooperative work.
The individual positions of the parts can be processed from the same captured image by image processing.
Since it is detected in the same coordinate system, management of collaborative work is simplified.
You can simply do it. In addition, detected by the sensor
From the world coordinate system to the camera coordinate system using the selected viewpoint position
Calculating the viewing transform coefficient that transforms the
The images captured by the cameras fixed to each head are used to
To multiple workers by correcting the ingress conversion coefficient.
Find corrected viewing transform coefficients for each
Therefore, virtual images of multiple workers can be accurately generated.
Can be generated.

[Brief description of drawings]

FIG. 1 is a diagram illustrating a principle of camera position correction, which is applied in a conventional technique and in an embodiment of the present invention.

FIG. 2 is a side view showing the configuration of the game device used in the embodiment of the present invention.

FIG. 3 is a diagram for explaining a scene seen in the visual field of the left player on the game device shown in FIG. 2;

FIG. 4 is an HMD used in the game device of FIG.
FIG.

5 is a view for explaining the arrangement of markers provided on the table of the game device shown in FIG.

FIG. 6 is a view for explaining the transition of markers included in an image captured by a camera mounted on the head of the player as the player moves on the table of FIG. 5;

FIG. 7 is a diagram illustrating a configuration of a three-dimensional image generation device for the game device according to the embodiment.

FIG. 8 is a flowchart illustrating a processing procedure by the mallet position measuring unit according to the embodiment.

FIG. 9 is a flowchart illustrating a partial subroutine (local search) of a processing procedure by the mallet position measuring unit according to the embodiment.

FIG. 10 is a flowchart illustrating a partial subroutine (global search) of a processing procedure by the mallet position measuring unit according to the embodiment.

FIG. 11 is a diagram illustrating division of a processing target area used in the processing of the flowchart of FIG.

12 is a view showing a method of setting a target area used in the processing of the flowchart of FIG.

FIG. 13 is a diagram illustrating a configuration of a virtual game field in the game of this embodiment.

FIG. 14 is a flowchart illustrating a game management control procedure in the game state management unit according to the embodiment.

FIG. 15 is a diagram illustrating a method for mallet detection.

FIG. 16 is a flowchart for generally explaining the processing procedure of the correction processing unit in the embodiment.

FIG. 17 is a flowchart for explaining in detail a part (tracking of markers) of the flowchart of FIG.

FIG. 18 is a flowchart for explaining in detail a part (prediction of marker position) of the flowchart of FIG.

FIG. 19 is a diagram illustrating the principle of detection of a reference marker used for correction.

FIG. 20 is a flowchart illustrating the principle of detection of a reference marker.

FIG. 21 is a diagram for explaining the transition of the reference marker applied to the modified example of the embodiment.

FIG. 22 is a diagram illustrating the principle of marker search applied to a modification of the embodiment.

─────────────────────────────────────────────────── ─── Continuation of the front page (72) Hiroyuki Yamamoto, inventor Hiroyuki Yamamoto 6-145, Hanasaki-cho, Nishi-ku, Yokohama Yokohama Hanasaki Building M.R. System Research Co., Ltd. (72) Mawa Fujiki 6 Hanasaki-cho, Nishi-ku, Yokohama 145-chome, Yokohama Hanasaki Building, M.R. System Research Institute Co., Ltd. (56) Reference Japanese Unexamined Patent Publication No. 5-328408 (JP, A) "Successfully mixing CG and" mixed reality "results," Nihon Keizai Shimbun , Nihon Keizai Shimbun, July 7, 1997, p.19 Andrei State, Super Priority Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking. n g, Proceedings of t he23rd annual confe rence on Computer graphics and inter activetechniques, the United States, ACM, 1996 years, p.429-438 Michael Bajura, Dy namic Registration Correction in Vid eo-Based Augmented Reality Systems, C omputer Graphics a nd Applications, USA, IEEE, 1995, Vol. 15, No. 5, p. 52-60 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 1/00 G06T 11/60- 17/50 H04N 13/00-17/06 G06F 3/00

Claims (4)

(57) [Claims] 1. A plurality of workers within a predetermined mixed reality environment
In order to perform a predetermined work while sharing a virtual object ,
It is a mixed reality presentation apparatus which produces | generates the three-dimensional virtual image of a virtual object, and displays it on the see-through type display apparatus mounted | worn by each of the said some worker , Comprising: The said several plurality used for the said predetermined work . Detecting means for detecting individual positions of the working part of the worker in the same coordinate system by image processing from the same captured image, and a sensor attached to each of the plurality of workers and detecting a viewpoint position of the individual worker. Using the viewpoint position detected by the sensor, a viewing conversion coefficient for converting from the world coordinate system to the camera coordinate system is obtained, and a captured image of the camera fixed to each head of the worker is obtained. Correction processing means for correcting the viewing conversion coefficient using the correction processing means for obtaining the corrected viewing conversion coefficient for each of the plurality of workers; A progress management unit that manages the progress of the predetermined work by using the detected individual positions of the working units of the plurality of workers, and the shared management unit that is managed by the progress management unit.
A three-dimensional virtual image showing the virtual object is generated for each of the plurality of workers by using the corrected viewing coefficient obtained for each of the plurality of workers, and each of the plurality of workers is generated. A mixed reality presentation device, comprising: a generation unit for outputting to a see-through display device. 2. The composite according to claim 1, wherein the sensor detects a position and a posture of a head of an individual worker, and calculates the viewpoint position from the detected position and posture. Reality presentation device. 3. The detecting unit detects the position of the working unit by the image processing with respect to a local area in the photographed image, which is limited according to an operator who operates the working unit for detecting the position. The position of the working unit is detected by the image processing for a global region that is the entire captured image when the position of the working unit is not detected from within the local region. 1. The mixed reality presentation device according to 1. 4. A plurality of workers within a predetermined mixed reality environment
In order to perform a predetermined work while sharing a virtual object ,
A mixed reality presentation method for generating a three-dimensional virtual image of a virtual object and displaying the virtual image on a see-through type display device attached to each of the plurality of workers, wherein the plurality of the plurality of workers are used in the predetermined work . A detection step of detecting individual positions of the working part of the worker in the same coordinate system by image processing from the same captured image; and a sensor attached to each of the plurality of workers and detecting a viewpoint position of the individual worker. Using the viewpoint position detected by the above, the viewing transformation coefficient for transforming from the world coordinate system to the camera coordinate system is obtained, and the view is obtained by using the image captured by the camera fixed to each head of the worker. A correction step of correcting the viewing conversion coefficient for each of the plurality of workers by correcting the viewing conversion coefficient; A management step of managing the progress of the predetermined work by using the individual position of the working portion of the number of workers, temporary the shared managed by the management step
A three-dimensional virtual image showing a virtual object is generated for each of the plurality of workers using the corrected viewing coefficient obtained for each of the plurality of workers, and the see-through of each of the plurality of workers is generated. Mixed reality presentation method, which comprises: a generation step of outputting to a type display device.