JPH11136706A - Position attitude sensor, position attitude detection method, composite reality providing device, game machine, composite reality providing method and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a system presenting composite reality that accurately catches a player moving over a wide range. SOLUTION: The game machine is provided with a work stand 1000 having pluralities of markers placed at known positions, attitude sensors 220 (220L, 220R) that are mounted on a player to sense a head attitude of the player, and cameras 240 (240L, 240R) that are set to receive at least one of pluralities of the markers within the visual field. Then the position of the head is detected based on an output from the attitude sensors 220, and the detected head position signal is corrected by applying image processing to an image signal from the cameras 240 to detect positions of the cameras 240 and a virtual image is generated to provide composite reality at a visual point depending on the corrected head position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method of presenting a mixed reality in which a virtual image formed by computer graphics is combined with a real space to a worker, for example, and presenting the worker with a three-dimensional position. Also, the present invention relates to a position / posture detection device and a position / posture detection method for accurately detecting a posture, and more particularly, to an improvement in the accuracy of detecting, for example, a head position of an operator who is presented with mixed reality.

[0002]

2. Description of the Related Art Recently, mixed reality (hereinafter referred to as "M") for the purpose of seamlessly connecting a real space and a virtual space.
R "(called Mixed Reality). MR is a virtual reality (hereinafter, referred to as V) that can be experienced only in a situation separated from the real space.
R) and VR for the coexistence of the real world
Has been attracting attention as a technology to enhance the quality of the product.

[0003] Applications of MR include medical assistance for presenting to a physician a transparent view of the inside of a patient's body,
New fields that are completely different in quality from conventional VRs are expected, such as the use of work assistance to display the assembly procedure of a product on a real product in a factory. A common requirement for these applications is how to remove the "shift" between real space and virtual space.
The “displacement” can be classified into a displacement, a time displacement, and a qualitative displacement, and among these, a great deal of efforts have been made to eliminate the displacement (ie, positioning), which is the most basic requirement. Have been.

In the case of a video-see-through MR in which a virtual object is superimposed on an image shot by a video camera, the problem of alignment is that the three-dimensional position of the video camera is accurately determined. It is a consequence. Optical see-through using transflective HMD (Head Mount Display)
The problem of positioning in the case of (Optic-See-Through) type MR can be said to be the problem of finding the three-dimensional position of the user's viewpoint, and their measurement methods are three-dimensional such as magnetic sensors, ultrasonic sensors, and gyros. Although use of a position and orientation sensor is common, the accuracy of these is not always sufficient, and an error thereof causes a position shift.

On the other hand, in the case of the video see-through system,
A method of directly performing positioning on an image based on image information without using such a sensor is also conceivable. In this method, since the displacement can be directly handled, the positioning can be performed with high accuracy, but there are problems such as lack of real-time performance and reliability. In recent years, attempts have been reported to achieve accurate alignment by using both a position and orientation sensor and image information to compensate for the disadvantages of both.

As one attempt, “Dynamic Registrati
on Correction in Video-Based-Augmented Reality Sys
tems '' (Bajura Michael and Ulrich Neuman, IEEE Compute
r Graphics and Applications 15, 5, pp. 52-60, 199
5) (hereinafter referred to as the first document) proposed a method of correcting a position shift caused by an error of a magnetic sensor by image information in a video see-through type MR.

[0007] Also, "Superior Augmented Reality Reg"
istration by Integrating Landmark Tracking and Mag
netic Tracking '' (State Andrei et al., Proc. of SIGGRAPH
96, pp. 429-438, 1996) (hereinafter referred to as the second document)
We further developed this method and proposed a method to compensate for the ambiguity of position estimation by image information using sensor information. The above-mentioned second document describes a position-azimuth sensor.
When a video see-through type MR presentation system is constructed using only the three-dimensional position, a landmark whose three-dimensional position is known is actually removed in order to eliminate a position shift occurring on an image due to an error of the sensor. Set to space. The landmark serves as a clue for detecting the displacement from the image information.

[0008] When the output of the position azimuth sensor does not contain errors, the landmarks are actually observed on the image coordinates (the Q _I), a camera position obtained based on the sensor output lands observation predicted coordinate landmarks to Dasa led from the three-dimensional position of the mark (and P _I)
Should be the same. However, since actually the camera position obtained based on the sensor output is not exact, the coordinates Q _I and observation predicted coordinate P _I landmarks do not match. The shift between P _I and Q _I represents the shift between the virtual space and the real space at the landmark position. For this reason, the direction and magnitude of the shift are calculated by extracting the landmark position from the image. it can.

As described above, by quantitatively measuring the displacement on the image, the camera position can be corrected so as to eliminate the displacement. The simplest alignment method that uses an azimuth sensor and an image is considered to be correction of a sensor error using a single landmark, and the camera position is translated or rotated according to the displacement of the landmark on the image. An approach has been proposed by the first document.

FIG. 1 shows the basic concept of positional deviation correction using one landmark. In the following, it is assumed that the internal parameters of the camera are known, and the image is captured by an ideal imaging system excluding the influence of distortion and the like. Present the point of view of the camera C, and the observation coordinate of the landmark in the image Q _I, the landmark position in real space and Q _I, on the straight line l _Q is the point Q _I connecting the point C and the point Q _I I do. On the other hand, from the camera position given by the position orientation sensor, and the landmark position P _C in the camera coordinate system, and the observation coordinate P _I on the image can be estimated. Hereinafter, the three-dimensional vectors from the point C to the point Q _I and the point P _I are denoted as v _l and v ₂ , respectively. In this method, the observed predicted coordinate mark P ′ _I of the corrected landmark coincides with Q _I (that is, the corrected landmark predicted position P ′ _C in the camera coordinate system is on the straight line l _Q ). By correcting the relative positional information between the camera and the object, the positional deviation is corrected.

It is considered that the displacement of the landmark is corrected by rotating the camera position. This can be realized by modifying the position information of the camera so that the camera rotates by the angle θ formed by the two vectors v _l and v ₂ . In the actual calculation, the vector v _l, v ₂ a normalized vector v _ln, with v _2n, the cross product v _ln × v _2n
Is rotated around the point C with the rotation axis as the rotation axis and the inner product v _{1n .v 2n} as the rotation angle.

It is considered that the displacement of the landmark is corrected by the relative translation of the camera position. This can be realized by translating the object positions in the virtual world by v = n (v ₁ −v ₂ ). Here, n is a scale factor defined by the following equation.

[0013]

(Equation 1)

Here, | AB | is a symbol indicating the distance between points A and B. The same correction can be made by modifying the position information of the camera so that the camera moves in parallel by −v. This is because this operation is equivalent to relatively moving the virtual object by v. The above two methods are methods for two-dimensionally matching positional displacements on landmarks, and do not correct camera positions to three-dimensionally correct positions. However, when the sensor error is small, a sufficient effect can be expected, and the calculation cost for correction is very small, so that the method is excellent in real time.

[0015]

However, in the technique disclosed in the above-mentioned document, it is necessary to capture the position of the only marker in the captured image, so that the marker is not always photographed by the camera. Due to the restriction that it must not be seen, only a very limited range of space could be seen.

Furthermore, when a plurality of workers share a common mixed reality space, the above restriction is fatal with only one marker.

[0017]

SUMMARY OF THE INVENTION The present invention has been made in view of such a situation, and an object of the present invention is to make it possible to accurately capture a specific work site of a worker who moves in a wide range. It is an object of the present invention to propose a position and orientation detection apparatus and method, and a mixed reality presentation apparatus based on the orientation detection.

In order to achieve the above object, according to claim 1 of the present invention, in order to generate a three-dimensional virtual image representing a work performed by a worker in a predetermined mixed reality environment, the work position of the worker is generated. A position and orientation sensor that measures the three-dimensional position and orientation of the worker and outputs a worker position and orientation signal, and a first plurality of sensors provided at known positions in the environment. A camera that captures an image of the marker, a detection unit that processes an image signal from the camera to track an arbitrary marker among the first plurality of markers, and detects a coordinate value of the tracking marker; Calculating means for calculating a part position and orientation representing the position and orientation of the work part based on the coordinate value of the tracking marker obtained by the detecting means and an operator position and orientation signal from the position and orientation sensor.

In order to achieve the above object, according to claim 40 of the present invention, in order to generate a three-dimensional virtual image related to a work performed by the worker in a predetermined mixed reality environment, the worker is required to generate a three-dimensional virtual image. A position and orientation detection method for detecting a work position includes a step of measuring a three-dimensional position and orientation of the worker and outputting a position and orientation signal, and a camera for imaging a plurality of markers provided at known positions in the environment. Processing the at least one marker to detect the coordinates of the tracking marker, based on the coordinates of the tracking marker and the position / posture signal. And outputting a signal representing the position and orientation of (i).

A position and orientation detecting apparatus for detecting a work position of an operator according to claim 46 of the present invention for achieving the object, measures a three-dimensional position and orientation of the worker and outputs a position and orientation signal. A position and orientation sensor that outputs, a camera that images a plurality of markers provided at known positions in the environment,
A detecting means for processing an image signal from the camera to track an arbitrary marker among the plurality of markers and detecting a coordinate value of the tracking marker; and a coordinate of the tracking marker obtained by the detecting means. Correction means for correcting the output signal of the position and orientation sensor based on the value.

According to a second aspect of the present invention, there is provided the above-mentioned object.
5 is a workbench having a first plurality of markers arranged at a known position, and is attached to the worker to detect the posture of the head of the worker. An attitude sensor, a camera set so that at least one of the first plurality of markers is in a field of view, and processing an image signal from the camera so as to process any one of the first plurality of markers. Detecting a marker of the tracking marker, detecting a coordinate value of the tracking marker, based on a coordinate value of the tracking marker obtained by the detecting unit and a head position / posture signal from the position / posture sensor, Calculating means for calculating a position and orientation signal of the viewpoint representing the position and orientation of the viewpoint of the worker; and generating means for generating a virtual image for presenting mixed reality at a viewpoint position corresponding to the position and orientation signal of the viewpoint. It is characterized by having To.

According to the arrangement of the first, second, third and fourth aspects, at least one marker is captured in the image, so that even if the worker moves over a wide range, the position of the worker can be accurately determined. Correction is assured. According to claim 2, which is a preferred aspect of the present invention, a distance between one marker and another marker among the first plurality of markers in a direction crossing the front of the worker is: The marker is set to be longer as the marker is farther in front of the worker. By equalizing the distance between the markers in the image, the marker detection accuracy can be equalized.
That is, as a result, it is possible to prevent a decrease in the accuracy of marker specification.

According to a preferred embodiment of the present invention, the distribution density of the first plurality of markers in the environment is such that the distribution density of the markers located farther in front of the worker is smaller than that of the first marker. , Is set lower than the distribution density of the markers in the vicinity. This also makes it possible to prevent a decrease in marker identification accuracy. According to claim 3, which is a preferred aspect of the present invention, when a plurality of workers perform cooperative work, a plurality of markers for the same worker are unified in the same expression form. It is easy to distinguish from markers for other workers.

According to a preferred aspect of the present invention, the site is a worker's head. According to claim 5, which is a preferred aspect of the present invention, the detecting means uses a marker first searched in an image acquired by the camera. In the present invention, a specific marker does not need to be kept tracked, but only one marker needs to be found. By using the marker searched first as in this embodiment, the marker search becomes easy.

According to a preferred aspect of the present invention, the detecting means includes means for searching a marker searched in a previous scene image in a current scene image. Features. Continuity is ensured. The position and orientation sensor can be attached to any part of the worker, but according to claim 7, which is a preferred aspect of the present invention, when the position and orientation sensor is provided on the head of the worker, The sensor comes close to the worker's viewpoint position, which facilitates application to the HMD.

According to a preferred aspect of the present invention, the first plurality of markers are arranged in the environment such that at least one of the first plurality of markers is captured in the field of view of the camera. It is characterized by. The placement of the markers ensures that at least one is captured within the camera's field of view. Marker detection is possible on various coordinate systems. According to a tenth aspect of the present invention, the detecting means detects image coordinates of the tracking marker in an image coordinate system. Similarly, according to claim 11, which is a preferred aspect of the present invention, the detecting means detects coordinates of the tracking marker in a camera coordinate system.

According to a twelfth aspect of the present invention, the first plurality of markers are marks drawn on a flat table arranged in the environment. It is suitable when joint work is performed on a table. According to a preferred embodiment of the present invention, the first plurality of markers are arranged at three-dimensional positions in the environment. It is possible to cope with a case where the markers must be arranged three-dimensionally.

According to a fourteenth aspect of the present invention, the detecting means has an identifying means for identifying the tracking marker from the first plurality of markers. Similarly, according to claim 15, when the second plurality of markers are detected in the image captured by the camera, the detecting means selects one marker from the second plurality of markers. And means for tracking.

According to the sixteenth aspect, the identification means identifies the selected tracking marker in an image coordinate system. Further, according to claim 18, the identification means obtains a position and orientation signal of the camera based on the worker position and orientation signal, and the first position is determined by the position and orientation signal of the camera.
By converting the three-dimensional coordinates of the plurality of markers in the world coordinate system into image coordinate values, and comparing the coordinates of the first plurality of markers in the image coordinate system with the image coordinate values of the tracking marker, According to a seventeenth aspect of the present invention, the identification means identifies the selected tracking marker in a world coordinate system. According to claim 19, the identification means obtains a position and orientation signal of the camera based on the worker position and orientation signal, and calculates the coordinates of the tracking marker in the camera coordinate system based on the position and orientation signal of the camera. Converted into a coordinate system value, the coordinates of the tracking marker in this world coordinate system,
The tracking marker is identified by comparing the coordinate values of the first plurality of markers with the world coordinate system.

In the case where an image coordinate system is used, according to a preferred aspect of the present invention, the part is a viewpoint position of a worker, and the calculating means includes Based on the position and orientation signal, the image coordinate value of the tracking marker identified by the identification means, and the error distance between the value obtained by converting the three-dimensional position of the tracking marker into an image coordinate system, the position of the worker's viewpoint is determined. Is obtained. This is because the error of the position and orientation sensor appears in the above error distance.

In a case where the world coordinate system is used, according to a preferred aspect of the present invention, the part is a viewpoint position of an operator, and the calculating means includes The position and orientation signal, a coordinate value obtained by converting the coordinates of the tracking marker identified by the identification means in the camera coordinate system into the world coordinate system, and an error distance between the three-dimensional position of the tracking marker and the A position and orientation signal at the viewpoint position is obtained.

The camera need not be one. Thus,
According to claim 23, which is a preferred aspect of the present invention, the camera has a plurality of units mounted on an operator.
Since the coordinates of the tracking marker can be detected by the camera coordinate value, the error of the position and orientation sensor can be corrected three-dimensionally. Further, since the tracking marker can be identified in the world coordinate system, it is possible to cope particularly with the case where the markers are three-dimensionally arranged. The accuracy can be expected to be improved as compared with the identification of the tracking marker in the image coordinate system.

[0033]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a system according to an embodiment in which a mixed reality presentation method and an HMD of the present invention are applied to an air hockey game device will be described. The air hockey game is a competitive game in which an opponent exists. Usually, a puck is floated by supplying compressed air from the lower part, the puck is beaten, and the puck is scored when the puck is put into the opponent's goal. In this game, the player with the highest score wins.
In the air hockey game to which the MR according to the present embodiment is applied, a virtual puck is displayed as a virtual three-dimensional image on a table in a real environment by superimposing the virtual puck on a table in a real environment, and the virtual puck is provided to a player by a real mallet. It is something to discuss.

The features of this game device are as follows: a common camera is used to photograph a real world common to a plurality of workers, and a working actuator operated by the plurality of workers in the common image (this embodiment) Then, by specifying the mallet), one mixed reality world appears and it can be shared by multiple people. : In order to accurately detect the viewpoint position of a worker who moves greatly in a wide real space, a camera is mounted on the worker's head in addition to a magnetic sensor that detects the position and posture of the head, The camera captures an image of at least one marker among the plurality of markers provided on the game play table. Based on a difference between the image coordinates of the captured marker and the known position of the marker, It corrects the position / posture of the head detected by the magnetic sensor (that is, the position and posture of the viewpoint of the worker).

<Structure of Game Apparatus> FIG. 2 is a side view of a game apparatus portion of the system according to the present embodiment. Mixed reality air hockey game is played on table 1
The two opponents 2000, 3000 face each other with a mallet (260L, 260R) in between. Two opponents 2000, 3000 have a head mounted display (hereinafter abbreviated as HMD) 210L, 2
Attach 10R. The mallet of the present embodiment has an infrared light emitter at its tip. As described later, in the present embodiment, the mallet position is detected by image processing.
If there is a characteristic in the shape and color of the mallet, it is also possible to detect the mallet position by pattern recognition using those characteristics.

The HMD 210 of the embodiment is of a see-through type as shown in FIG. Two opponents 2000, 300
0 indicates that the surface of the table 1000 can be observed even when the HMDs 210L and 210R are mounted. HMD
A three-dimensional virtual image is input to 210 from an image processing system described later. Therefore, the opponents 2000, 3000
The three-dimensional image displayed on the display screen of the HMD 210 is superimposed on the image of the real space passed through the optical system of the HMD 210 (not shown in FIG. 2).

FIG. 3 shows an image viewed from the HMD 210L by the left player 2000. Two players 2
000,3000 hit virtual pack 1500.
Hitting the pack 1500 is a real mallet 260L held by the player 2000 (player 3000).
(260R) is used. Player 2000 holds mallet 260L in his hand. The goal 1200R can be seen just before the opponent player 3000. An image processing system described later (not shown in FIG. 3) generates a three-dimensional CG and generates an HMD 21 so that the goal 1200R can be seen near the opponent.
Display at 0L.

On the other hand, the player 3000 also has the HMD 210
Goal 1200L near the player 3000 via R
Will be seen. The pack 1500 is also generated by an image processing system described later and displayed on each HMD. <HMD with Magnetic Sensor> FIG. 4 shows the configuration of the HMD 210. The HMD 210 is disclosed in, for example,
The magnetic sensor 220 is attached to the main body of the HMD of No. 3551 via a column 221. In the figure, reference numeral 211 denotes an LCD display panel. The light from the LCD display panel enters the optical member 212, is reflected by the total reflection surface 214, is reflected by the total reflection surface of the concave mirror 213, passes through the total reflection surface 214, and passes through the eyes of the observer. Reaches

In the present embodiment, the magnetic sensor 220 is Po
An lhemus magnetic sensor, Fastrack, was used. Since the magnetic sensor is vulnerable to magnetic noise, a plastic support 221 is used.
As a result, the display panel 211 and the camera 240, which are noise sources, were separated. The configuration in which the magnetic sensor and / or camera is attached to the HMD shown in FIG. 4 is not limited to the optical see-through (perspective) HMD, but is a video see-through (shielded) HMD. Also,
A magnetic sensor and / or camera can be attached to the HMD for the purpose of accurately detecting the head position and posture.

In FIG. 2, each HMD 210 is fixed to the player's head by a band (not shown).
A magnetic sensor 220 as shown in FIG. 4 and a CCD camera 240 as shown in FIG.
(240L, 240R) are respectively fixed.
The field of view of the camera 240 is set in the forward direction of the player. When an HMD having such a magnetic sensor 220 and a camera 240 is used for an air hockey game,
Since each player views the upper surface of the table 1000, the camera 240 also captures an image of the surface of the table 1000. The magnetic sensor 220 (220L, 220
R) senses a change in the AC magnetic field generated by the AC magnetic field generation source 250.

As will be described later, the image captured by the camera 240 is used to correct the position / posture of the head detected by the magnetic sensor 220. Player is at table 10
HMD21 when facing diagonally downward to see the surface of 00
0, the surface of the table 1000, the virtual pack 1500 described above, and the real mallet 260 (26
0L, 260R), virtual goal 1200 (1200
L, 1200R). When the player moves the head horizontally in a horizontal two-dimensional plane, or performs a tilting motion, a yaw motion, and a rolling motion, the change is first detected by the magnetic sensor 220, and CCD camera 2 according to the posture change
40 is observed as a change in the image taken. That is, a signal representing the head position from the magnetic sensor 220 is transmitted to the camera 24.
The image is corrected by performing image processing on the 0 image.

<Plurality of Markers> Each mallet 260 gripped by each player has an infrared emitter at its tip, and the position (two-dimensional plane position) of each mallet on the table 1000 is determined from each mallet. Is known by a CCD camera 230 that detects infrared rays. That is, the camera 230 is for detecting the position of the hand of each player (the position of the mallet). By detecting the mallet position, the progress of the hockey game can be determined.

On the other hand, the CCD camera 240 outputs an image called a marker image. FIG. 5 shows an example of a marker arranged on the table 1000. In FIG.
Five landmarks or markers (1600 to 1600) indicated by marks
Reference numeral 1604) denotes a marker used for detecting the head position of the player 2000 in an auxiliary manner.
Two landmarks or markers (1650 to 1654) indicate markers used for auxiliary detection of the head position of the player 3000. When a plurality of markers are arranged as shown in FIG. 5, the position of the head, particularly the posture, determines which marker is visible. In other words, by specifying a marker in an image captured by the CCD camera 240 attached to each player and detecting a position in the image,
The output signal of the magnetic sensor for detecting the player's head posture can be corrected.

The circles and squares in FIG. 5 are used for illustration purposes, and their shapes are not unique, and may be any shape. Two players (20
00,3000) and a marker group (1650-1608) and a marker group (1650-1
658) are respectively colored in different colors.
In the present embodiment, the marker for the left player (# 1 player) is colored red, and the marker for the right player (# 2 player) is green. This is to make it easy to distinguish markers in image processing. Note that markers can be distinguished not by color but by shape or texture.

A major feature of the present embodiment is that a plurality of markers are arranged. The multiple placement ensures that at least one marker is within the field of view of the CCD camera 240 as long as the player acts on the table 1000 within the operating range of the present air hockey game. FIG. 6 illustrates a situation in which the image processing range for detecting a marker moves with the movement of the head when the player moves the head in various ways. As shown in the figure, one image contains at least one marker. In other words, the number of markers, the interval between markers, and the like should be set according to the size of the table 1000, the viewing angle of the camera 240, and the size of the player's movement range based on the nature of the game. In the case of FIG. 5, the farther from the player, the wider the field of view is, so the interval between the markers must be widened. This reduces the number of markers captured in an image of a distant field of view by making the distance between markers in the vicinity of the image in the image equal to the distance in the image between markers in the distance. This is to prevent a decrease in marker detection accuracy. By doing so, the density of the markers captured in the image can be made substantially the same between the distant marker and the nearby marker, thereby preventing unnecessary imaging of multiple markers in the same frame. Can be.

As described later, in the present system, at least one marker exists in the image obtained by the camera 240L (240R), and it is sufficient if the marker can be specified. Therefore, it is not necessary to keep track of a specific marker while the player moves his / her head (while moving the camera 240). <MR Image Generation System> FIG. 7 shows a configuration of a system for generating and presenting a three-dimensional image in the game apparatus shown in FIG. This image generation / presentation system includes a left player 2
000 HMD210L and right player 3000 H
A three-dimensional virtual image (pack 1500, goal 1200 in FIG. 3) is output to each display device of the MD 210R. The generation of the left and right parallax images for the three-dimensional virtual image is performed by the image generation units 5050L and 5050R. In the present embodiment, each of the image generating units 5050 is provided with a computer system “O
NYX2 "was used.

The image generation unit 5050 inputs the pack position information and the like generated by the game state management unit 5030 and the information on the corrected viewpoint position and head direction generated by the two correction processing units 5040L and 5040R. Game state management unit 5030 and correction processing units 5040L, 5040R
Each consisted of a computer system ONYX2.

The CCD camera 230 fixed above the center of the table 1000 covers the entire surface of the table 1000 in the field of view. Mallet information acquired by the camera 230 is input to the mallet position measurement unit 5010. The measuring unit 5010 is also provided with “O” manufactured by Silicon Graphics.
2 "Comprised of a computer system. Measurement unit 5
010 detects the mallet positions of the two players, that is, the positions of the hands. Information on the position of the hand is input to the game state management unit 5030, where the game state is managed. That is, the game state / progress of the game is basically determined by the position of the mallet.

A position / posture detection unit 5000 constituted by a computer system O2 manufactured by Silicon Graphics Co., Ltd. receives the outputs of the two magnetic sensors 220L and 220R (the position and posture of the sensor 220 itself) and is mounted on each player. The position (X, Y, Z) and posture (p, r, φ) of the camera (240L, 240R) are detected and output to the correction processing units 5040L, 5040R.

On the other hand, the CC fixed to the head of each player
The D cameras 240L and 240R acquire marker images, and the marker images are used as marker position detecting units 5060L and 5060L, respectively.
Processed at 5060R, the position of the tracking marker for each player within the field of view of each camera 240 is detected. Information on the tracking marker position is input to the correction processing unit 5040 (5040L, 5040R).

The marker position detector 5 for tracking the marker
060 (5060L, 5060R) was configured with an O2 computer system. <Mallet Position Measurement> FIGS. 8 to 10 are flowcharts showing a control procedure for measuring the mallet position.
By tracking the mallet with one common camera, it is possible to present a common mixed reality by a plurality of workers. According to the flowcharts of FIGS. 8 to 10,
The measurement of the mallet position according to the present embodiment will be described.

In an air hockey game, a player does not advance his own mallet to the area of another player. For this purpose, the process of searching for the mallet 260L (260R) of the left player 2000 (right player 3000) may be performed by focusing on the image data IL (image data IR) of the left field as shown in FIG.
It is easy to divide the image acquired by the CCD camera 230 at the fixed position into two regions as shown in FIG.

Accordingly, in the flowchart of FIG. 8, the mallet 26 of the player # 1 (player 2000)
Regarding the search for 0L, in step S100, the player #
Regarding the search for mallet 260R of player 2 (player 3000), respective processes are performed in step S200. Therefore, for convenience, the search for the mallet of the right player (step S200) will be described as an example.

First, in step S210, the TV camera 2
30 obtains a multi-valued image of the surface of the table 1000 captured. In step S212, a subroutine "search in local area" is performed on the right half image data IR of the multi-valued image. The details of the "search in local area" process are shown in FIG. When the coordinates (x, y) of the mallet position in the image coordinate system are found in step S212, the process proceeds from step S214 to step S220, and the mallet position coordinates (x, y) in the image coordinate system are calculated according to the following equation. It is converted to the coordinate position (x ′, y ′) of the system (see FIG. 13).

[0055]

(Equation 2)

[0056] Here, the matrix M _T is 3 × 3 for calibrating the image coordinate system and table coordinate system
Which is known. The coordinate position (x ′, y ′) obtained in step S220 (in FIG. 3, (x ′, y ′)
y ′) is indicated as “hand position”) is sent to the game state management unit 5030. If no mallet is found in the local area, "search in global area" is performed in step S216. If a mallet is found in the "search in the global area", step S22
At 0, the coordinate position is converted to a table coordinate system. The coordinate position searched in the local or global area is used for searching for a mallet in the local area in the next frame.

FIG. 9 shows processing for searching for a mallet in the local area (details of step S212). However, this process shows a search process in the right field for convenience, but the same applies to a mallet search process in the left field. In step S222, a rectangular area of size (2A + 1) × (2B + 1) pixels defined by the following equation is extracted.

[0058]

Equation 3] _{x = [I x -A, I} x + A] y = [I y -B, I y + B] where, in the formula, I _'x, I' _y was detected in the previous frame The coordinates of the mallet, A and B are constants for determining the size of the search area, and the search area is as shown in FIG.

In step S230, with respect to all the pixels (x, y) in the rectangular area defined in step S222, a pixel whose characteristic evaluation value I _S (x, y) satisfies a certain condition is extracted. It is a process. For the purpose of searching for a mallet, the feature amount is preferably a similarity between pixel values (infrared light intensity values). In the present embodiment, since an infrared light emitter is used for the mallet, the one having the characteristic of the intensity of the infrared light is determined to be a mallet.

That is, in step S232, the similarity I _S
Find pixels closer to the mallet than a predetermined threshold. When such a pixel is found, the cumulative value of the frequency of occurrence is stored in the counter N. Further, the x coordinate value and the y coordinate value of such a pixel are cumulatively stored in the registers SUMx and SUMy. That is,

[0061]

N = N + 1 SUMx = SUMx + x SUMy = SUMy + y When step S230 is completed, the number N of all the pixels similar to the pattern of the infrared light from the mallet in the region of FIG. 12 and the cumulative values SUMx, SU of the coordinate values
My is obtained. If N = 0, the result "Not Found" is output in step S236. If N> 0, a mallet-like thing has been found, and step S238
And the mallet position (I _x , I _y )

[0062]

(Equation 5)

The calculation is performed according to Then, the calculated mallet position (I _x , I _y ) is stored in step S220 (eighth
FIG.), The data is converted into a table coordinate system, and the converted value is passed to the management unit 5030 as a signal representing the “hand position”. FIG. 10 shows a detailed procedure of the global area search in step S216. In step S240 of FIG. 10, in the image IR of the right field,

[0064]

(Equation 6)

｛(X, y) | x> 0, x <Width, x = nC, y>
0, y <Height, y = mD (where n and m are integers)}, the maximum value of the characteristic evaluation value I _S is stored in the register Max. Here, C and D are constants that determine the coarseness of the search, and Width and Height are defined in FIG. That is, in step S242, it is determined whether more than a threshold value characteristic amount I _S is stored in the threshold storage register Max. If such a pixel is found, in step S244, in order to set the feature value as a new threshold value, in step S244,

[0066]

[Mathematical formula-see original document] Let Max = _Is (x, y) _Ix = _xIy = y. In step S246, the coordinates of the most mallet-like pixel (Ix, Iy) found in the global search are passed to step S220.

As described above, the mallet is found in the image, and its coordinate value is converted to a table coordinate system and passed to the game state management unit 5030. <Game State Management> FIG. 13 shows a game field of the air hockey game of the present embodiment. This field is defined on a two-dimensional plane above the table 1000,
It has x and y axes. Further, it has two left and right virtual goal lines 1200L and 1200R, and virtual walls 1300a and 1300b provided in the vertical direction in FIG. The coordinate values of the virtual goal lines 1200L and 1200R and the virtual walls 1300a and 1300b are known, and do not move. In this field, according to the movement of the mallets 260R and 260L, the pack 15
The 00 virtual image moves.

The pack 1500 stores the coordinate information P of the current position.
and a _p and velocity information v _p, left mallet 260L has coordinate information P _SL and velocity information v _SL of the current position, and the right mallet 260R has coordinate information P _SR and the speed information v _SR of the current position Have. FIG. 14 is a flowchart illustrating a processing procedure in the game state management unit 5030.

In step S10, pack 1500
The initial position P _p0 and the initial speed v _p0 are set. In addition, the pack performs a uniform motion at a speed v _p. It is assumed that the pack makes a completely elastic collision when it hits a wall or a mallet, that is, the speed direction is reversed. Game state management unit 503
In the case of 0, the speed information v _S is obtained from the position information P _{S of} each mallet measured by the mallet position measuring unit 5010.

Step S12 is executed at intervals of Δt until a win or loss in the game is determined (one of the points wins three points in step S50). Then, step S12
Then, the position of the pack is

[0071]

## _EQU8 ## P _p = P _p0 + v _p0 ＋ Δt is updated. The position of the pack after setting the initial position and the initial speed is generally

[0072]

## EQU9 ## It is expressed by P _p = P _p + v _p · Δt. In step S14, it is checked whether or not the updated pack position _Pp is in the field on the # 1 side (left player) of the player. The case where the puck 1500 is on the left player side will be described.

In step S16, it is checked whether or not the current pack position is at a position where it interferes with the left player's mallet 1100L. Pack 1500 is Mallet 1100L
Means that the left player 2000 has performed a mallet operation to cause the mallet 260L to collide with the puck. Therefore, in order to reverse the movement of the puck 1500, the puck 1500 is
By inverting the sign of the x-direction velocity component v _px of the velocity v _p
Proceed to step S20.

Incidentally, instead of simply inverting the sign of the velocity component v _{px in} the x direction of the velocity v _p ,

[0075]

[ _Equation 10] _{Assuming that} v _px = −v _px + v _SLx , the pack operation may be performed in the opposite direction by superimposing the x-direction operation speed v _SLx of the mallet on the pack x-direction speed v _px. good. On the other hand, if the current pack position is not at the position where it interferes with the mallet 1100L of the left player (NO in step S16), the process directly proceeds to step S20.

In step S20, it is checked whether or not the pack position P _p is at a position where it collides with the virtual wall 1300a or 1300b. If the determination in step S20 is YES, the y component of the speed of the pack is inverted in step S22. Next, in step S24, it is checked whether or not the current pack position is within the goal line of the left player. YE
In the case of S, the score of the opponent player, that is, the score of the right (# 2) player is added in step S26. Step S
At 50, it is checked whether any score has scored 3 or more points in advance. If the score is three or more, the game ends.

If it is determined in step S14 that the puck position _Pp is on the right player side (# 2 player side), step S30 and subsequent steps are executed. Step S30-
Step S40 has substantially the same operation as steps S16 to S26. Thus, the progress of the game is managed. The progress of the game depends on the position of the pack,
The position of the mallet, as described above, the image generation unit 5
050 (5050L, 5050R).

<Correction of Head Position> FIG. 16 shows the entire control procedure of the processing in the correction processing unit 5040 (5040L, 5040R). The correction in the correction processing unit 5040 means that the output of the magnetic sensor 220 involves an error, and the viewpoint position data and head posture data calculated by the measuring unit 5000 based on such output are obtained from the CCD camera 240. Refers to a process of correcting based on the marker position in the obtained image. That is, this correction processing is performed by the camera 240
Calculates the correction value of the position of the camera 240 (also closely related to the position of the head) from the marker position in the acquired image, and changes the viewpoint viewing transformation matrix of the viewpoint using the correction value. . The changed viewing transformation matrix represents the corrected viewpoint position and orientation data, in other words, the corrected viewing transformation matrix provides a virtual image at the corrected viewpoint position.

FIG. 26 illustrates the principle of correction of the observer's viewpoint position / posture in the first embodiment. Here, the correction of the viewpoint position / posture of the observer in the embodiment is equivalent to obtaining a corrected viewing transformation matrix. In FIG. 26, it is assumed that the camera 240 of the player captures the marker 1603 in the image 300. Marker 1
The position 603 in the image 300 is represented by, for example, (x ₀ , y ₀ ) in the image coordinate system. On the other hand, if it is known that the marker captured by the image 300 is 1603, the coordinates (X ₀ , Y ₀ , Z ₀ ) of the marker 1603 in the world coordinate system are known. (X ₀ , y ₀ ) is the image coordinate value (X ₀ , y ₀ )
Since Y ₀ , Z ₀ ) are world coordinates, these coordinates cannot be compared. In the first embodiment, the viewing transformation matrix M _C of the camera 240 is obtained from the output of the magnetic sensor 220, and the coordinates (X ₀ , Y ₀ , Z ₀ ) in the world coordinate system are calculated using the viewing transformation matrix M _C. To the coordinates (x ^′ ₀ , y ^′ ₀ ) of the image coordinate system. Since the error between (x ₀ , y ₀ ) and (x ^′ ₀ , y ^′ ₀ ) represents an error in the output of the magnetic sensor 22, a correction matrix Δ for correcting this error
Seek M _C.

As apparent from FIG. 26, the image 3
It is necessary to specify that the marker caught in 00 is the marker 1603. However, in the first embodiment, as described later, the three-dimensional positions of all markers in the world coordinate system are determined by the above-mentioned viewing transformation matrix. is converted into the image coordinate system by M _C, the camera coordinate values after conversion is specified as markers captured in the (x _0, y ₀₎ closest marker image 300 within the. This process will be explained with reference to FIGS. 19 and 20.

The processing procedure of the correction processing unit 5040 will be described in detail with reference to FIG. That is, in step S400, based on the output of the magnetic sensor 220, the camera 240
Is calculated (4 × 4). In step S410, based on the viewing transformation matrix obtained in step S400, the ideal perspective transformation matrix (known) of the camera 240, and the three-dimensional position (known) of each marker,
Position coordinates (in image coordinate system) where each marker should be observed
Predict.

On the other hand, the marker position detector 5060 (506
0L, 5060R) track markers in images obtained from cameras 240 (240L, 240R) attached to the player's head. Marker position detector 5060
Passes the detected marker position to the correction processing unit 5040 (in step S420). Correction processing unit 5040
(5040L, 5040R) determines the marker currently being observed, that is, the marker to be a reference for correction, based on the passed marker position information in step S420. In step S430, the position and orientation of the camera 240 detected by the magnetic sensor 220 is corrected based on the difference between the predicted coordinate value of the marker calculated in step S410 and the observed coordinate value of the marker detected by the marker position detection unit 5060. Correction matrix ΔMc is obtained. The position and orientation of the camera 240 can be corrected by the marker position detection unit 5060.
(Marker 1603 in the example of FIG. 26)
And the marker coordinates based on the head position detected by the magnetic sensor should match if the sensor output is accurate. Therefore, the above difference calculated in step S430 will Because it reflects. The relative relationship between the position and orientation of the camera and the position and orientation of the viewpoint is known, and the relationship is represented by three-dimensional coordinate conversion. Accordingly, in step S440, the viewing transformation matrix of the viewpoint calculated in step S400 is corrected based on the correction matrix ΔMc of the position and orientation of the camera, and the corrected conversion matrix is used as the image generation unit 5050 (5050L, 5050R). Pass to.

FIG. 17 shows a processing procedure for marker position detection in the marker position detection unit 5060. In step S500, the color image acquired by the camera 240 is captured. Thereafter, in step S502, "local area search" is performed, and in step S506, "global area search" is performed to detect the marker position (x, y) represented by the image coordinate system. The procedure of “local area search” in step S502 and “global area search” in step S506 include “local area search” (FIG. 9) and “global area search” in mallet search.
Since this is substantially the same as (FIG. 10), the “local area search” (FIG. 9) and the “global area search” (FIG. 10) are referred to, and are not shown. However, in the incorporated control procedure (step S232), the feature amount I _S for marker discovery, player #
For 1 (left), the pixel value of the pixel of interest is

[0084]

[Equation 11]

Is used. For player # 1, red is used for markers (1600-1604),
This feature quantity represents the degree of redness. Player #
Green marker (1650-165) for 2 (right)
Since 4) is used,

[0086]

(Equation 12)

Is used. The above two quantities are also used for the feature quantity I _s (x, y) in the global search. In step S510, the coordinate values of the markers obtained in steps S502 and S506 are converted into an ideal image coordinate system without distortion using a matrix M (for example, having a size of 3 × 3) for correcting distortion. Convert to The conversion formula at this time is

[0088]

(Equation 13)

Is as follows. Next, step S4 in FIG.
The details of the process 10 will be described with reference to FIG.
As described above, in step S400, a transformation matrix M _C (4 × 4 viewing transformation matrix) from the world coordinate system to the camera coordinate system is obtained. On the other hand, a transformation matrix P _C (4 × 4) from the camera coordinate system to the image coordinate system is also given as a known value. The three-dimensional coordinate position (X, Y, Z) of the marker of interest is also given as known.

That is, the angle r is set to Z at the position of the camera 240.
The rotation in the axial direction (roll), the angle p in the position of the camera 240 in the X-axis direction (pitch), and the angle φ in the camera 24
The rotation (yaw) in the Z-axis direction at the position of 0
Is (X ₀ , Y ₀ , Z ₀ ), the viewing transformation matrix Mc of the camera 240 (ie, the transformation matrix from the world coordinate system to the camera coordinate system) Mc is

[0091]

[Equation 14]

Where d is the focal length of the camera 240, w
The camera of the imaging surface of the width, when the same height h, the transformation matrix P _C from the camera coordinate system to the image coordinate system,

[0093]

(Equation 15)

Is represented by Therefore, in step S520 of FIG. 18 (that is, step S410 of FIG. 16), the coordinate position (X, Y, Z) of the target marker is calculated according to the following equation.
It is converted to a position (x _h , y _h , z _h ) on the image plane.

[0095]

(Equation 16)

In step S522, the observed observation coordinate values x and y of the marker in the image coordinate system are set as

[0097]

[Equation 17]

Is obtained. Thus, at step S410, it is possible to obtain the predicted coordinate values (x _i , y _i ) of each marker i in the image coordinate system. Next, step S
The process of “marker determination” in 420 will be described. First
FIG. 9 shows a case where the camera 240 of one player has acquired an image 600 on the table 1000.

The markers provided on the table 1000 are, for example, M _{1 to} M ₇ and are represented by a triangle. Three-dimensional position M _i of this marker is known. The image 600 includes markers M ₂ , M ₃ , M ₆ , and M ₇ . On the other hand, observation prediction position of each marker M _i are those calculated in step S520, that it and P _i. Q indicates the marker position detected by the marker position detection unit 5060 and passed from the detection unit 5060.

The “judgment of marker” in step S420 is as follows.
The marker position Q detected by the marker position detection unit 5060 is
It is to determine which P _i (that is, which M _i ) corresponds. In FIG. 19, the vector e _i represents the length of the vector from the detected marker position Q to the predicted position P _i of each marker, that is, the distance. FIG. 20 shows the details of step S420. That is, the process of FIG. 20, of the distance e _i marker i entering image 6000 (i = 0 to n), and searches for a marker indicating the minimum value, and outputs the identifier i of the marker . That is,

[0101]

(Equation 18)

Is as follows. In the example of Fig. 19, since the shortest distance e ₂ of between P _2, and data using the marker M ₂ to the correction of the magnetic sensor output. Thus, no matter how the player moves, within their field of activity,
Since the camera 240 captures at least one marker in the image, it is not necessary to limit the size of the field to a narrow size as in the related art.

Next, in step S430, a conversion matrix ΔMc representing the correction of the position and orientation of the camera based on the error distance e _min obtained on the basis of the equation 18 is the same as the processing described in FIG. Ask for. On the other hand, in step S432, a viewing transformation matrix M _V at the viewpoint position of the player is obtained based on the magnetic sensor output. Also, _assuming that M _vc is a transformation matrix (known) from the camera coordinate system to the viewpoint coordinate system, in step S440, using this M _vc , the viewing transformation matrix M _v ′ of the corrected viewpoint is calculated by the following equation. Is derived.

[0104]

[Equation 19]

Incidentally, as is clear from FIG. 26 and also from a second embodiment described later, the first embodiment
In the embodiment (the processing in FIG. 16), the error distance e is obtained by converting the image into the image coordinate system. Can be obtained. <Improvement of Head Position Detection Accuracy> Second Embodiment In the first embodiment, the HMD 210L (210R) is provided with one camera 240L (240R) for a front monitor. The processing unit 5060 processes the image of the marker on the table 1000 acquired by the camera 240, and specifies the marker in the image (step S420).
Then, the attitude of the head of the player, that is, the attitude of the camera mounted on the head, in other words, the matrix representing the viewing transformation by the camera having this attitude is determined. However, in the first embodiment, since only the error in the image coordinate system is used, a three-dimensional shift remains in the positional relationship between the camera and the marker.

Further, depending on the application for presenting the mixed reality, the marker may be placed at an arbitrary position in the three-dimensional space. In such a case, as shown in FIG. 16 in the first embodiment. The marker identification method has low reliability. The second proposed embodiment solves the problem of the three-dimensional displacement. That is, the above-mentioned problem is solved by mounting two cameras to one player and performing marker detection in the world coordinate system. The second embodiment also alleviates the constraint that the marker must be arranged on a plane.

More specifically, two HMDs in which two cameras are arranged and mounted on the left and right are used for two players. That is, as shown in FIG.
The (3000) HMD 210L (210R) has two cameras 240LR, 240LL (240RR, 240RR).
RL) and the cameras 240LR, 240LL
From the stereo images obtained from (240RR, 240RL), cameras 240LR, 240LL (240RR, 24RL)
0RL).

The system of the second embodiment can cope with the case where the markers are arranged three-dimensionally. However, in order to clarify the difference from the processing procedure of the first embodiment, the system of the second embodiment will be described. As in the first embodiment, the present invention is applied to an air hockey game using a plurality of markers arranged on a plane. 22nd
The figure shows a part of the image processing system according to the second embodiment. That is, FIG. 22 shows a changed part of the image processing system (FIG. 7) of the first embodiment. That is, comparing FIG. 7 with FIG. 22, the image processing system according to the second embodiment has a marker position detecting unit 5060L ′ (5060L) in addition to the point that each player has two cameras.
R ′) and a correction processing unit 5040L ′ (5040R ′), but the marker position detection unit 5060L ′ (5060R ′) of the second embodiment is different from the first embodiment.
And the correction processing unit 5040L '(5040R') is the marker position detection unit 5060L (5060R) of the first embodiment.
The only difference is that the correction processing unit 5040L (5040R) is different from the software processing.

FIG. 23 shows a control procedure for the left player 2000 in the processing procedure of the second embodiment, and particularly corresponds to the control procedure of FIG. 16 of the first embodiment. , The marker position detector 5060 'and the position
The link operation between the attitude detection unit 5000 and the correction processing unit 5040L 'will be described. In FIG. 23, the position / posture detection unit 5000, which is the same as that in the first embodiment, executes step S3.
At 98, a viewing transformation matrix of the viewpoint is calculated based on the output of the magnetic sensor 220L. In step S400 ', based on the output of the magnetic sensor 220L, the camera 24
The inverse matrix of the 0LR viewing transformation matrix is calculated. This transformation matrix is sent to the correction processing unit 5040 '.

The images from the two cameras 240LL (240LR) are sent to a marker position detecting unit 5060L '.
That is, in step S402, the detection unit 5060 'extracts a marker image m _R in the image R for the right camera 240LR. The coordinates of the extracted marker (that is, the observation coordinates) are represented by _ImR . In step S404, the detection unit 5060 '
Extracts the corresponding marker image m _L from the image L from the right camera 240LL. The coordinates of the extracted marker are represented by I _mL . Since the marker image m _R and the marker image m _L are originally of the same marker m _X , in step S406, the set of observed marker coordinates (I _mR , I _mL ) is used based on the principle of triangulation. , The three-dimensional position C _m of the extracted observation marker in the coordinate system of the camera 240LR is derived.

In step S404, a corresponding point search of the marker image m _L is performed using a general stereoscopic method. However, in order to perform processing at high speed, a search range using a well-known epipolar constraint epipolar bind is used. May be limited. Steps S410 'and S42 in FIG. 23
0 ', step S422, and step S430' indicate processing in the correction processing unit 5040L '.

[0112] First, 'in the three-dimensional position C _m in the camera coordinate system of the observed marker, step S400' Step S410 by using the perspective transformation matrix derived in, the three-dimensional position W _m of the world coordinate system Convert. In step S420 ', taken out three-dimensional position W _mi in the world coordinate system of all the markers m _i a (known) from a predetermined memory, the Euclidean distance between the individual markers m _i and the observed marker m _X | W _mi -W W _mi that minimizes _m | is determined. That is, to identify the known marker in the observed marker m _X closest position.

Although W _mi and W _m are originally at the same position, an error vector D (corresponding to e in the first embodiment) is generated due to a sensor error. Therefore,
In step S420 ′, a marker having a coordinate value W _mi closest to the three-dimensional coordinates (world coordinates) of the observed (tracked) marker is determined. In step S430 ′, the marker between the observed marker and the determined marker is determined. The distance difference vector D is

[0114]

Equation 20] was calculated by D = W _mi -W _m, obtains a transformation matrix ΔMc for moving the camera position by this vector quantity, step S44
At 0 ', the viewing transformation matrix of the viewpoint is corrected by the same method as in the first embodiment.

As described above, according to the present invention, the position of the observation marker can be detected three-dimensionally by using the HMD equipped with two cameras, so that a more accurate viewpoint position and orientation can be detected. This makes it possible to smoothly connect the MR virtual image and the real image. <Modification 1> The present invention is not applied only to the above-described first and second embodiments.

In the first embodiment, in the process of detecting a marker in an image, as shown in FIG. 17, the marker that is found first is used as the marker to be tracked. for that reason,
For example, as shown in FIG. 24, when an image 800 including the marker M1 is obtained in a certain frame, the image area 810 of the subsequent frame includes the marker at the end of the area 810 but within the area 810. If included in
There is no problem in determining the marker Mi as a reference marker for the correction process. However, in a subsequent frame, for example, an image 820 is obtained, and the marker M _i falls within the area, and instead includes the marker M ₂ , the reference marker for correction is changed to the marker M _2. I have no choice.
Such a change of the marker is necessary even when tracking fails, and a newly tracked marker is used for correcting the positional deviation.

As a problem of switching the marker used for correction as described above, the virtual object may move unnaturally at the time of the switching due to a sudden change in the correction value. . Therefore, in order to maintain the temporal consistency of the correction values, it is proposed as a modification that the correction values up to the previous frame are reflected in the setting of the next correction value.

That is, assuming that the correction value (a three-dimensional vector representing a parallel movement in the world coordinate system) in a certain frame is v _t and the correction value in the previous frame is v ′ _t−1 , the following expression is obtained. Let _v't be a new correction value.

[0119]

(Equation 21)

Here, α is a constant of 0 ≦ α <1 which defines the degree of influence of past information. The above equation means that the contribution based on the correction value v ′ _t−1 in the previous frame is α
And the correction value v _t obtained in the current frame is (1−
α). By doing so, a sudden change in the correction value is reduced, and a sudden change (unnatural movement) of the three-dimensional virtual image is eliminated. By setting the new correction value α to an appropriate value, it is possible to prevent unnatural movement of the object due to switching of the marker.

<Modification 2> In the above embodiment, the process of detecting a marker in an image is performed, as shown in FIG. 17, when the marker cannot be found by the local search, the position of the marker in the previous frame. Regardless of this, the point with the highest similarity in the entire screen is used as the marker to be tracked. Here, a modified example in which the marker search is performed centering on the position of the marker found in the previous frame is proposed. This is because, even if the image frame moves along with the movement of the player, there is a high possibility that the marker exists at a position that is not largely shifted from the position existing in the previous frame.

FIG. 25 explains the principle of searching for a marker found in the previous frame in the current frame. When a search is performed along such a search path and a point having a similarity higher than a certain threshold is found, this point is used as a marker to be tracked. <Modification 3> Although the above embodiment uses the optical HMD, the present invention is not limited to the application of the optical HMD, but can be applied to a video see-through HMD.

<Modification 4> Although the above embodiment is applied to the air hockey game, the present invention is not limited to the air hockey game. According to the present invention, since a plurality of works (for example, mallet operation) are imaged and captured by one camera, it is possible to reproduce the works of the plurality of persons in one virtual space. Therefore, the present invention is also suitable for an embodiment of a cooperative work (for example, an MR presentation of a design work by a plurality of persons or a competitive game of a plurality of persons) on the premise of two or more workers.

The process of correcting the head posture position based on a plurality of markers according to the present invention is not suitable only for cooperative work by a plurality of persons. One worker (or player)
It is also applicable to a system that presents mixed reality. <Other Modifications> In the second embodiment, two cameras are used, but three or more cameras may be used.

As described above, it is sufficient for the marker to be captured by at least one of the player's cameras 240. If the number of markers is too large, the number of markers captured in the image increases, and S430 in FIG. 16 and S430 ′ in FIG.
In the tracking marker identification process, there is a high possibility that the marker is erroneously identified. Therefore, if the operation can regulate the movement of the camera 240 to some extent, it is possible to reduce the number of markers so that the camera always captures only one marker.

Although the position and orientation detection apparatus according to the above-described embodiment outputs a corrected viewing transformation matrix at the viewpoint position of the player, the present invention is not limited to this. The present invention can also be applied to a device that outputs a viewpoint position in the form of a corrected value (X, Y, Z, r, p, φ). The marker may have any shape as long as the marker can be recognized as a marker or a mark by the above-described system, and may be a mark instead of a mark.

[0127]

As described above, according to the position and orientation detection apparatus and the position and orientation detection method of the present invention, since a plurality of markers are imaged by a camera, at least one marker is acquired in the image. Becomes Therefore, even if the worker moves in a wide working range or a wide moving range, the position of the head of the worker can be tracked. In particular, a special effect of enabling presentation of mixed virtual reality over a wide range is obtained.

[Brief description of the drawings]

FIG. 1 is a view for explaining the principle of camera position correction applied to a conventional technique and to an embodiment of the present invention.

FIG. 2 is a side view showing the configuration of the game device used in the embodiment of the present invention.

FIG. 3 is a view for explaining a scene seen in the field of view of a left player on the game device of FIG. 2;

FIG. 4 is an HMD used in the game device of FIG. 2;
FIG.

FIG. 5 is a view for explaining an arrangement of markers provided on a table of the game device in FIG. 2;

FIG. 6 is a view for explaining changes in markers included in an image captured by a camera mounted on the player's head as the player moves on the table in FIG. 5;

FIG. 7 is an exemplary view for explaining the configuration of a three-dimensional image generation device for the game device according to the embodiment;

FIG. 8 is a flowchart illustrating a processing procedure performed by a mallet position measurement unit according to the embodiment.

FIG. 9 is an exemplary flowchart illustrating a subroutine (local search) of a processing procedure performed by the mallet position measurement unit according to the embodiment;

FIG. 10 is an exemplary flowchart illustrating a subroutine (global search) of a processing procedure performed by the mallet position measurement unit according to the embodiment;

FIG. 11 is a view for explaining division of a processing target area used in the processing of the flowchart in FIG. 8;

FIG. 12 is a diagram showing a setting method of a target area used in the processing of the flowchart in FIG. 8;

FIG. 13 is an exemplary view for explaining the configuration of a virtual game field in the game according to the embodiment;

FIG. 14 is a flowchart illustrating a control procedure of game management in a game state management unit according to the embodiment.

FIG. 15 is a diagram illustrating a technique for detecting a mallet.

FIG. 16 is an exemplary flowchart for entirely explaining the processing procedure of a correction processing unit according to the embodiment;

FIG. 17 is a flowchart for explaining in detail a part of the flowchart of FIG. 16 (tracking of a marker);

FIG. 18 is a flowchart for explaining in detail a part of the flowchart of FIG. 16 (prediction of a marker position);

FIG. 19 is a view for explaining the principle of detection of a marker serving as a reference used for correction.

FIG. 20 is a flowchart illustrating the principle of detection of a reference marker.

FIG. 21 is a diagram showing a configuration of an HMD used in the second embodiment.

FIG. 22 is a block diagram illustrating a main configuration of an image processing system according to a second embodiment.

FIG. 23 is a flowchart illustrating a part of control of the image processing system according to the second embodiment;

FIG. 24 is a view for explaining changes in reference markers applied to a modification of the embodiment.

FIG. 25 is an exemplary view for explaining the principle of marker search applied to the modification of the embodiment;

FIG. 26 is a view for explaining the principle of the correction processing according to the first embodiment.

──────────────────────────────────────────────────の Continued on the front page (51) Int.Cl. ⁶ Identification symbol FI G06T 15/00 G06F 15/62 360 7/00 415 (72) Inventor Toshiichi Oshima 6-145 Hanasaki-cho, Nishi-ku, Yokohama-shi Hanasaki Yokohama Building Inside M.R.Systems Co., Ltd. (72) Inventor Takasato Taniguchi 6-145 Hanasakicho, Nishi-ku, Yokohama-shi Hanasaki Yokohama Building Inside M.R.Systems Co., Ltd. (72) Akihiro Katayama Nishi-ku, Yokohama 6-145 Hanasakicho Yokohama Hanasaki Building M.R. System Co., Ltd.

Claims (46)

[Claims] 1. A position and orientation detection apparatus for detecting a work position of a worker in order to generate a three-dimensional virtual image representing a work performed by the worker in a predetermined mixed reality environment, the apparatus comprising: A position and orientation sensor that measures a three-dimensional position and orientation and outputs a worker position and orientation signal, a camera that captures a first plurality of markers provided at known positions in the environment, and an image from the camera A signal processing section for tracking an arbitrary marker among the first plurality of markers, detecting means for detecting a coordinate value of the tracking marker, and a coordinate value of the tracking marker obtained by the detecting means; A position and orientation detection device comprising: a computing unit that computes a part position and orientation representing the position and orientation of the work part based on an operator position and orientation signal from the position and orientation sensor. 2. A distance between one marker and another of the first plurality of markers in a direction crossing the front of the worker is greater than a distance of a marker farther ahead of the worker. The position and orientation detection device according to claim 1, wherein the position and orientation are set to be longer. 3. The position according to claim 1, wherein when a plurality of workers perform cooperative work, the first plurality of markers for the same worker are unified into the same expression form. Attitude detection device. 4. The position and orientation detection device according to claim 1, wherein the work site is a viewpoint position of a worker. 5. The position and orientation detection apparatus according to claim 1, wherein the detection means uses a marker first searched in an image acquired by the camera. 6. The position and orientation detection apparatus according to claim 1, wherein said detection means includes means for searching for a marker searched for in a previous scene image in a current scene image. 7. The position and orientation detection device according to claim 1, wherein the position and orientation sensor is provided on a head of an operator. 8. The distribution density of the first plurality of markers in the environment is set such that a distribution density of a marker located farther in front of the worker is lower than a distribution density of a marker located closer thereto. The position and orientation detection device according to claim 1, wherein: 9. The position and orientation detection apparatus according to claim 1, wherein the first plurality of markers are arranged in the environment such that at least one of the markers is captured in a field of view of the camera. . 10. The position and orientation detection apparatus according to claim 1, wherein the detection means detects image coordinates of the tracking marker in an image coordinate system. 11. The position and orientation detection apparatus according to claim 1, wherein the detection unit detects coordinates of the tracking marker in a camera coordinate system. 12. The position and orientation detection apparatus according to claim 1, wherein the first plurality of markers are marks drawn on a planar table arranged in the environment. 13. The apparatus according to claim 1, wherein the first plurality of markers are arranged at three-dimensional positions in the environment.
2. The position and orientation detection device according to claim 1. 14. The position and orientation detection apparatus according to claim 1, wherein said detection means has an identification means for identifying the tracking marker from among the first plurality of markers. 15. The detecting means, when a second plurality of markers is detected in an image captured by the camera, selects one tracking marker from the second plurality of markers. The position and orientation detection device according to claim 1, further comprising: 16. The position and orientation detection apparatus according to claim 1, wherein the identification unit identifies the selected tracking marker in an image coordinate system. 17. The position and orientation detection apparatus according to claim 1, wherein the identification unit identifies the selected tracking marker in a world coordinate system. 18. The identification means obtains a position and orientation signal of the camera based on the worker position and orientation signal, and the three-dimensional marker in the world coordinate system of the first plurality of markers is determined based on the position and orientation signal of the camera. Converting the coordinates into image coordinate values, and identifying the tracking marker by comparing the coordinates of the first plurality of markers in the image coordinate system with the image coordinate values of the tracking marker. The position and orientation detection device according to claim 14. 19. The identification means obtains a position and orientation signal of the camera based on the worker position and orientation signal, and calculates the coordinates of the tracking marker in the camera coordinate system based on the position and orientation signal of the camera in a world coordinate system. The tracking marker is identified by comparing the coordinates of the tracking marker in the world coordinate system with the coordinate values of the first plurality of markers in the world coordinate system. The position and orientation detection device according to claim 15, 20. The method according to claim 19, wherein the part is a viewpoint position of a worker, and the calculating means is configured to control the worker position and posture signal, a coordinate value of a tracking marker identified by the identifying means by a camera coordinater, and the tracking. 15. The position and orientation detection device according to claim 14, wherein a position and orientation signal at a viewpoint position of the worker is obtained based on an error distance from a value obtained by converting a three-dimensional position of the marker into an image coordinate system. 21. The part is a worker's viewpoint position, and the calculating means converts the worker position and orientation signal and image coordinates of the tracking marker identified by the identifying means into a world coordinate system. 15. The position and orientation detection apparatus according to claim 14, wherein a position and orientation signal at a viewpoint position of the worker is obtained based on the value and an error distance between the tracking marker and the three-dimensional position. 22. The position and orientation detection device according to claim 14, wherein the position and orientation sensor is a magnetic sensor provided on a head of the worker. 23. The position according to claim 17, wherein the camera has a plurality of units mounted on an operator, and wherein the identification means identifies the selected tracking marker in a world coordinate system. Attitude detection device. 24. The apparatus according to claim 23, wherein the number of the camera units is two. 25. A worktable having a first plurality of markers arranged at known positions, a posture sensor mounted on the worker for detecting a posture of the head of the worker, and the first plurality of markers. A camera set so that at least one of the markers is in the field of view; and processing an image signal from the camera to track any marker in the first plurality of markers. Detecting means for detecting the coordinate value of the operator; and representing the position and orientation of the viewpoint of the worker based on the coordinate value of the tracking marker obtained by the detecting means and the head position and orientation signal from the position and orientation sensor. A composite comprising: arithmetic means for calculating a position and orientation signal of a viewpoint; and generation means for generating a virtual image for presenting mixed reality at a viewpoint position corresponding to the position and orientation signal of the viewpoint. Reality presentation Location. 26. A distance between one marker and another marker of the first plurality of markers in a direction crossing in front of the worker is greater than a distance of a marker farther ahead of the worker. 26. The mixed reality presentation apparatus according to claim 25, wherein the apparatus is set on the work table so as to be longer. 27. The mixed reality presentation according to claim 25, wherein when a plurality of workers perform cooperative work, a plurality of markers for the same worker are unified into the same expression form. apparatus. 28. The mixed reality according to claim 25, wherein said detecting means further comprises a tracking means for tracking a marker in an image acquired by said camera and outputting an image coordinate value of said marker. Feeling presentation device. 29. The mixed reality presentation apparatus according to claim 28, wherein the tracking means identifies a marker first searched in the image as a marker being tracked. 30. The mixed reality presentation apparatus according to claim 28, wherein the tracking means includes means for searching a marker searched for in a previous scene image in a current scene image. . 31. A distribution density of the first plurality of markers in the environment is set such that a distribution density of a marker located farther in front of the worker is lower than a distribution density of a marker located closer thereto. 3. The method according to claim 2, wherein
6. The mixed reality presentation device according to 5. 32. The mixed reality presentation of claim 25, wherein the first plurality of markers are located in the environment such that at least one is captured within a field of view of the camera. apparatus. 33. The mixed reality presentation apparatus according to claim 25, wherein the detection means detects image coordinates of the tracking marker in an image coordinate system. 34. The mixed reality presentation apparatus according to claim 25, wherein the detection unit detects coordinates of the tracking marker in a camera coordinate system. 35. The mixed reality presentation apparatus according to claim 25, wherein the first plurality of markers are marks drawn on a planar table arranged in the environment. 36. The system according to claim 3, wherein the first plurality of markers are arranged at three-dimensional positions in the environment.
6. The mixed reality presentation device according to 5. 37. The mixed reality presentation apparatus according to claim 25, wherein the detection unit includes an identification unit that identifies the tracking marker from the first plurality of markers. 38. The mixed reality presentation apparatus according to claim 25, wherein said identification means performs identification in an image coordinate system. 39. The mixed reality presentation apparatus according to claim 26, wherein said identification means performs identification in a world coordinate system. 40. A position and orientation detection method for detecting a work position of a worker in order to generate a three-dimensional virtual image related to a work performed by the worker in a predetermined mixed reality environment, the method comprising: Measuring a three-dimensional position and orientation of the environment and outputting a position and orientation signal; and processing an image signal from a camera that captures a plurality of markers provided at known positions in the environment, and
Tracking the two markers and detecting the coordinates of the tracking marker; and outputting a signal representing the position and orientation of the worker's head based on the coordinates of the tracking marker and the position and orientation signal. A position and orientation detection method comprising: 41. A mixed reality presentation method for presenting virtual reality in accordance with the position of the worker's head detected by the position and orientation detection method according to claim 40. 42. A plurality of cameras are mounted on an operator, and the detecting step is performed by processing signals representing a plurality of images in front of the worker, which are captured by a plurality of camera units, using a triangulation method. 41. The method according to claim 40, wherein at least one marker is tracked. 43. A storage medium storing a computer program describing the method of claim 40. 44. A storage medium storing a computer program describing the method according to claim 41. 45. A storage medium storing a computer program describing the method of claim 42. 46. A position and orientation detection device for detecting a work position of an operator, comprising: a position and orientation sensor that measures a three-dimensional position and orientation of the worker and outputs a position and orientation signal; A camera that captures a plurality of markers provided at a position, a detection unit that processes an image signal from the camera, tracks an arbitrary marker among the plurality of markers, and detects a coordinate value of the tracking marker A position and orientation detection device comprising: a correction unit configured to correct an output signal of the position and orientation sensor based on a coordinate value of the tracking marker obtained by the detection unit.