JP2000102036A - Composite actual feeling presentation system, composite actual feeling presentation method, man-machine interface device and man-machine interface method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a composite actual feeling presentation system with less calibration and less position deviation. SOLUTION: The three-dimensional position of a marker provided on an indicator 30 held by a user is measured corresponding to the coordinate system of a camera by using left and right stereo cameras 21R and 21L provided near the view point position of an observer, that is a head part, and a virtual image is outputted and displayed to an HMD (head mounted display) 22 provided on the head part of the observer so as to view the virtual image at the three- dimensional position.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mixed reality presentation system, for example, for fusing a virtual image by computer graphics into a real space and presenting it to an observer, a method thereof, and a man-machine interface used for mixed reality presentation. The present invention relates to an apparatus, a method thereof, and a computer-readable recording medium.

[0002]

2. Description of the Related Art Recently, mixed reality (hereinafter referred to as "M") for the purpose of seamlessly connecting a real space and a virtual space.
R "(called Mixed Reality). MR is a virtual reality (hereinafter, referred to as V) that can be experienced only in a situation separated from the real space.
R) and VR for the coexistence of the real world
Has been attracting attention as a technology to enhance the quality of the product.

[0003] MR fuses a virtual space assembled by a computer with a real space. Fusion is usually performed via an optical See-Through type HMD (Head Mount Display) or a video See-Through type HMD. That is,
When the optical See-Through HMD is used, the real world and the virtual world are fused by seeing through the external world viewed directly and the CG displayed on the LCD etc. in front of the eyes in the HMD. . On the other hand, when the video See-Through HMD is used, the real world and the virtual world are fused by superimposing a CG image on an image captured by a video camera also received by the HMD and presenting it to an observer.

In some applications of MR, it is necessary to identify a position designated by an observer or the like in a virtual world coordinate system. For example, in order to display, for example, a CG image at a position pointed by an observer using a held indicator, it is necessary to detect the pointed position in a reference coordinate system. If the position actually instructed is different from the position recognized by the system, the display position of the presented CG image does not match the real world, giving a sense of discomfort to the observer.

[0005] For this reason, measuring the position of the indicator with high accuracy is a major problem in MR technology. FIG.
Shows a state in which the user 13 wearing the HMD 10 is holding the indicator 12 in his hand in the mixed reality presentation system. In this example, the user's head position (that is, the viewpoint position) is the magnetic sensor 1 provided on the HMD 10.
1 and generates a CG image for giving virtual reality based on the measured viewpoint position and gives it to the HMD. The user interface between the user and the MR system is performed by the indicator 12.

FIG. 2 illustrates the measurement principle of the magnetic sensor. The transmitter emits an electromagnetic field, and the magnetic sensor 11 mounted on the head detects the electromagnetic field, and the head (or HMD) tertiary. Measure the original position. When measuring the three-dimensional position based on the output of the magnetic sensor 11, the measured position is always based on the world coordinate system. On the other hand, as shown in FIG. 3, the indicator 12 is equipped with a plurality (for example, six) of LEDs, and the six LEDs emit light. The user holds the indicator,
While pointing to be imaged by the camera, point the tip of the indicator to a desired position. The three-dimensional position of the indicator measures the positions of the six LEDs based on the image captured by the camera, and based on the deformation of the figure formed by the measured six LEDs, the three-dimensional position in the world coordinate system at the tip of the indicator is determined. The original coordinate position is measured. The MR system recognizes the three-dimensional position as the three-dimensional position designated by the user.

[0007]

In the mixed reality presentation system, the viewpoint position at which the CG is presented in the user's field of view is, as described above, the position of the user's head measured based on the output signal from the magnetic sensor 11. Is converted to the position of the user's eyes. The obtained viewpoint position is represented in the world coordinate system.

On the other hand, the three-dimensional coordinates of the user's designated position obtained from the three-dimensional posture position of the indicator 12 are also represented in the world coordinate system. Here, the indicator 1 is placed in the field of view of the HMD based on the user's viewpoint position observed by the magnetic sensor 11.
The CG figure (for example,
CG button), the CG
The graphic may be displayed or presented at a position that is significantly different from the reality.

FIG. 4 and FIG. 5 explain the reason why a so-called "position shift" occurs in an MR system in which the head position is recognized by a head magnetic sensor and a pointer is used as a user interface. 4, as described above, the user of the viewpoint position recognized by the magnetic sensor (head position) has an error [delta] ₁ to the world coordinate system. Can not be error [delta] ₁ itself "0", calibration is required to reduce. Incidentally, the viewpoint position of the user is further have an error [delta] ₃ relative to the head position, the error [delta] ₃ itself is capable correction.

Similarly, the three-dimensional indicated position measured by the indicator 12 also has an error δ _{2 with} respect to the world coordinate system.
Should have. For this purpose, the CG is placed in the field of view of the HMD based on the user's viewpoint position observed by the magnetic sensor 11 and the user-specified position obtained by the indicator 12.
When a figure (for example, a CG button) is presented, the position of the presented CG figure is presented as having a cumulative error of δ = δ ₁ + δ ₂ (1) with respect to the user's viewpoint position. Will be. The accumulation of this error appears as a large “position shift” of the CG figure with respect to the real world.

This problem is not caused by using a magnetic sensor for detecting the head position and using image processing for measuring the indicated position. Conversely, image processing is performed for measuring the head position. Even if a magnetic sensor is used to measure the indicated position, there is still an error in the sensor itself, and the same occurs as long as the measurement result based on the output from each sensor is based on a different coordinate system.

[0012]

SUMMARY OF THE INVENTION The present invention has been made in view of such circumstances, and has as its object to propose a mixed reality presentation system with less calibration and less displacement. In order to achieve the above object, a mixed reality presentation system according to claim 1 of the present invention, which presents an observer by fusing a virtual space and a real space,
First sensor means provided in the vicinity of the observer's viewpoint position to detect a three-dimensional coordinate value of a presentation position at which a virtual image is to be presented in accordance with a coordinate system related to the viewpoint position; And a virtual image presenting means for presenting a virtual image to the observer at a presentation position detected by the sensor means with reference to.

According to this system, since the sensor means is provided near the viewpoint position of the observer, the three-dimensional coordinate value of the presentation position at which the virtual image is to be presented is determined by the coordinate system related to this viewpoint position. , The three-dimensional coordinate value substantially matches the coordinate system of the viewpoint position of the observer. For this reason, even if the virtual image presenting means presents the virtual image at such a presentation position, in other words, even if the calibration with the world coordinate system is not performed, the positional deviation from the real space substantially does not occur. It can be suppressed to a level that does not cause a problem.

In consideration of use, the first sensor means is preferably provided at the position of the observer's head.
Therefore, according to claim 2 which is a preferred aspect of the present invention, the first sensor means has an image pickup means provided at a position of a head of the observer, and the virtual image presenting means comprises It has a head mounted display attached to the observer's head.

According to a preferred aspect of the present invention, the head mounted display is of a video see-through type. The video see-through head-mounted display presents the image of the real world taken by the camera to the observer as an image, so there is in principle less displacement than the optical see-through type. ,
Even if calibration with the world coordinate system is not performed, it can be suppressed to a level that does not substantially cause a problem.

According to a preferred embodiment of the present invention, the first sensor means includes a stereo camera mounted on a head and separated to the left and right, and a stereo image data output by the stereo camera. Means for detecting a three-dimensional coordinate value of the presentation position on the image plane of the stereo camera based on a stereoscopic technique.

In order to minimize the displacement, the mixed-reality presentation system according to claim 5, further comprising second sensor means for detecting the viewpoint position of the observer according to a world coordinate system. Is also preferred. The presentation position of the virtual image is specified by the operation tool. Therefore, according to claim 6 which is a preferable aspect of the present invention, the operating tool is located within a sensing range of the sensor means and operated by an observer.

To indicate the presentation position using the operation tool,
It is necessary for the observer to inform the system that the position is the desired position. Therefore, according to claim 7, which is a preferable aspect of the present invention, the operation tool is provided with a switch which is operated by the observer and outputs a predetermined event timing signal by the observer. And

According to a preferred embodiment of the present invention, the operating tool has at least one marker provided at an end thereof. When the position of the marker is detected by the sensor means, the position is defined as the presentation position. According to a ninth preferred embodiment of the present invention, one marker is provided, and the sensor outputs a three-dimensional coordinate position of the presentation position.

According to a preferred embodiment of the present invention, two markers are provided, and the sensor means outputs a three-dimensional coordinate position and a direction of the presentation position. According to claim 11, which is a preferable aspect of the present invention, the three markers are provided, and the sensor outputs a plane coordinate passing through the presentation position. According to claim 12, which is a preferable aspect of the present invention, the marker includes a plurality of markers colored differently. By giving different colors to different markers, the distinguishability between the markers is improved.

According to a preferred embodiment of the present invention, the markers include a plurality of markers colored differently and at least one marker colored the same. In this case, a constraint condition is imposed on the movement of the observer. According to claim 14, which is a preferred aspect of the present invention,
The marker has light emitting means for emitting different colors.

There may be a plurality of operation tools for indicating the presentation position. This is because it is possible to indicate a plurality of presentation positions at the same time, furthermore, in a hierarchical manner or in association with each other. Therefore, according to claim 15 which is a preferable aspect of the present invention,
A plurality of operating tools are provided within a sensing range of the sensor means and operated by an observer. For example, virtual images of a plurality of graphic user interfaces are presented at a presentation position designated by one operating tool, and one virtual graphic user interface among the virtual images is displayed as another one.
It can be instructed by one operation tool.

The marker is not always in a state that can be measured by the sensor means. The observer moves freely. Therefore, a preferable aspect of the present invention is claim 16.
According to the above, the operating tool is further provided with a magnetic sensor as a position sensor. As a result, position information according to the world coordinate system based on the magnetic sensor output is obtained, and it is possible to supplement even when marker information cannot be used.

The above object can also be attained by providing a mixed reality presentation method according to the present invention. Also,
The above object can also be achieved by a computer-readable storage medium that stores a program that presents an observer by fusing a virtual space and a real space. Such a computer-readable storage medium uses a sensor provided in the vicinity of the observer's viewpoint position to calculate the three-dimensional coordinate value of the presentation position at which the virtual image is to be presented. And a program code for presenting a virtual image to the observer at a presentation position detected by the sensor with reference to the viewpoint position of the observer.

Another object of the present invention is to provide a man-machine interface in addition to the presentation of mixed reality. Therefore, a man-machine interface for inputting a presentation position at which a virtual image is to be presented to a mixed reality presentation system via an operation tool operated by a user is provided near an observer's viewpoint position. An imaging unit including the operation tool in an imaging visual field, and a coordinate system related to a viewpoint position of the observer, based on image data obtained by the imaging unit, and a three-dimensional coordinate value of a predetermined portion of the operation tool. Conversion means for converting to a three-dimensional coordinate value according to, and output means for outputting the converted three-dimensional coordinate value of the predetermined part to the mixed reality presentation system as a virtual image presentation position instructed by an observer. It is characterized by having.

The user interface instructs the mixed reality presentation system such that when a predetermined portion of the operation tool is directed to a designated position desired by the observer, the designated position is a presentation position of a virtual image. The user interface of the present invention includes:
It is also useful as a tool for inputting user instruction input to the mixed reality presentation system.

For this purpose, as in claim 21,
A man-machine interface for inputting a user instruction to a mixed reality presentation system via an operation tool operated by a user, provided near a viewpoint position of an observer,
Imaging means including the operating tool in an imaging field of view, and, based on image data obtained by the imaging means, three-dimensional coordinate values of a predetermined portion of the operating tool in a coordinate system related to a viewpoint position of the observer. A conversion unit for converting the three-dimensional coordinate value of the predetermined part into a three-dimensional coordinate value according to the user's instruction input, and an output unit for outputting the converted three-dimensional coordinate value to the mixed reality presentation system. A man-machine interface device is provided.

As the user instruction, an icon or the like selected by the user among icons or the like displayed as virtual images may be input to the mixed reality presentation system. The user interface of the present invention can be achieved not only by a man-machine interface device but also by a man-machine interface method (claims 20 and 22) or by a computer-readable recording medium (claims 23 and 24).

[0029]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of a mixed reality presentation system to which the present invention is applied will be described in detail with reference to the drawings. <Structure> FIG. 6 illustrates the principle structure of this embodiment.

Numeral 20 denotes a magnetic sensor for detecting the position of the head.
Us Fastrak is used. 21R and 21L are a pair of video cameras mounted on the HMD 22.
The camera 21R captures the outside world in front of the observer from the right side of the observer, and the camera 21L captures the outside world in front of the observer from the left side of the observer.

An indicator 30 is held by the user as shown in FIG. The indicator 30 has a rod-like shape so that the user can easily hold it, and LEDs 30a and 30b for functioning as markers are provided at both ends thereof.
The LEDs 30a and 30b are controlled by the position measuring device 40, and are lit during measurement. In addition, LED3
The emission colors 0a and 30b emit different colors of light so that they can be identified by image processing.

The output of the magnetic sensor 20 is supplied to a position measuring device 40.
And the device 40 measures the three-dimensional coordinate position of the observer's head based on the output signal. The observer points to a desired pointing position with the pointing device 30. At this time, if the observer needs to notify the system of the indicated position as an event, the switch 31 provided with the indicator 30 is provided.
Press (Fig. 8). The output of the switch 31 is input to the computer 50 (FIG. 10) as an event signal.

FIG. 10 is a diagram for explaining a data processing system of the MR system according to the embodiment. In FIG. 10, video cameras 21R and 21L mounted on the user's head acquire a pair of left and right images of the outside world in front of the user. Each of the left and right images includes an image of the indicator 30. The left and right images are input to the image processing devices 60R and 60L, where the LEDs 30a and 30b at both ends of the indicator 30 are provided.
Are extracted.

The image data observed by the video cameras 21R and 21L are sent to the image processing devices 60R and 60L, respectively. The image processing devices 60R and 60L detect the positions of the LEDs 30a and 30b on the image using the characteristics of the color components. The position coordinates are input to a computer (for example, 02 system manufactured by Silicon Graphics) 50. The calculator 50 calculates the three-dimensional positions of the LEDs 30a and 30b by a stereoscopic method.

On the other hand, the output of the head position sensor is input to the position and orientation measurement device 40, and the three-dimensional position and orientation of the head are measured as described above. The output is also input to the computer 50. You. The computer 50 includes the position and orientation measurement device 4
The head position measured by 0 is converted into the right eye viewpoint position, and a viewing transformation matrix at this position is calculated. This matrix is sent to the image generation device 70. The image generation device generates a CG image from the right viewpoint position using the viewing transformation matrix. Further, to this viewing transformation matrix, a relative movement transformation of the left viewpoint position with respect to the right viewpoint position is applied, and a viewing transformation matrix corresponding to the left viewpoint is obtained. Generate a CG image. The left and right CG images generated in this way are displayed on the HMD 22 to present a three-dimensional virtual image to the user.

In the example of FIG. 7, the computer 50 generates a three-dimensional image of a cursor or an icon based on the measured three-dimensional coordinates of the tip position of the indicator, and displays the three-dimensional image on the HMD 22. As shown in FIG. 7, the user who views the HMD 22 observes the three-dimensional cursor or icon as indicated by 61, superimposed on the tip position of the indicator. <Principle> FIG. 11 explains the reason why the MR system of the embodiment shown in FIGS. 6 to 10 does not generate a displacement even without performing calibration.

In the figure, δ ₁ represents a measurement error of the three-dimensional position of the user's head measured by the output of the magnetic sensor 20 with respect to the world coordinate system. Δδ represents the deviation of the video camera position from the head position. Δ ₂ is a measurement error of the indicated position measured by the indicator 30. The feature of the present embodiment is that the computer 50 in the system of FIG. 10 uses the CG based on the position measured by the position sensor (magnetic sensor 20).
The feature is that a graphic is generated by an image generating apparatus. Then, in FIG. 11, since the position sensor output is used as a reference, the error δ ₁ between the position sensor output and the world coordinate system does not affect the indicated position generated by the indicator. In other words,
As shown in FIG. 11, the error δ in the designated position is δ = Δδ + δ ₂ (2), and δ ₁ is not involved. Since the presentation of the CG image is based on the image plane, it is not necessary to refer to the world coordinate system as in the related art. Therefore, the head position measured by the position sensor 20 (which has an error δ ₂₎ Even if a CG image is generated based on the three-dimensional coordinates of the designated position based on the reference position, no positional displacement occurs as in the related art.

The deviation Δδ between the camera position and the head position
Since the head position and the head-mounted camera position have a fixed relationship, calibration can be performed in advance, and calibration is not required in real time each time the MR system is operated. However, since the indicated position changes in real time, calibration is performed on δ ₂ by adjusting camera parameters (described later). In other words, according to this embodiment,
Conventionally, two steps of real-time calibration were required, but only one step was required, and the system startup was simplified.

The method of this embodiment is applied to a video See-Th
If the method is applied to a rough type HMD, calibration of δ ₂ is not required. <Measurement Processing of Pointed Position> Embodiment FIG. 12 is a flowchart showing a processing procedure for the pointed position measurement in the computer 50 of FIG.

In step S10, the right and left images are acquired by the video camera 21 via the image processing device 60 (see also step S1 in FIG. 9). In step S12, the left and right LEDs (hereinafter, referred to as a marker) are provided to the image processing device 60 for each of the left and right images.
(Step S in FIG. 9)
2). In step S14, the four markers recognized in the left and right images are associated with each other. As a result of this association, as shown by the solid line in FIG.
Alternatively, an association as shown by a broken line will be obtained.

In step S16, the respective vectors of the four markers are calculated. In step S18, the intersection of the extension line of the vector of the pair of corresponding points of one marker is determined by a stereophotogrammetry technique (known). The intersection is
As shown in FIG. 4, the intersection of the extension line between the respective projection positions of the left and right video cameras and the respective coordinate positions of one marker on the left and right image planes, and the cubic of the marker based on the viewpoint position Indicates the original position.

Since the three-dimensional position of the marker obtained by the stereophotogrammetry in step S18 is based on the viewpoint position of the video camera 21, the three-dimensional position of the marker is, as described with reference to FIG. thereby enclosing the measurement error [delta] _1. Generally, two vectors do not always cross at one point. In this case, as shown in FIG. 15, the midpoint of the line segment connecting two points (100 and 101 in the example of FIG. 15) where the two line segments on the vector are closest (point 102 in the example of FIG. 15). ) Is the "intersection".

The processing in step S18 is performed for the left and right markers, and three-dimensional positions P ₁ and P ₂ are determined based on the camera viewpoint position of each marker. That is, P ₁ = (x ^C ₁ , y ^C ₁ , z ^C ₁ ) P ₂ = (x ^C ₂ , y ^C ₂ , z ^C ₂ ), and the subscript C is based on the camera viewpoint position. Means

In step S20, the position and orientation of the indicator 30 are identified from the three-dimensional positions P ₁ and P _{2 of} the markers. Regarding the method of identifying the position and orientation of the indicator 30 (step S20), the respective principles when three different markers are used will be described. Indicator with Two Markers FIG. 16 is an example of an indicator provided with two markers. For convenience, when the user grips the indicator 30 like a writing instrument, the marker located on the lower side is called a “top marker”, the position is indicated by a vector p ₀ , and the marker located on the upper side is a “rear marker” And the position is indicated by a vector p ₁ . The target position is indicated by the “tip marker”.

If the "front end marker" and the "rear end marker" are distinguished from each other by being given different colors as described above, the target designated position is, as shown in FIG. ₀ direction vector: may be obtained as p ₀ -p _1.

It should be noted that the color of the marker may be taken in a broad sense and light in the infrared region may be used. Further, when the color of both markers are the same, the user is always the position of the rear end marker p _1, as compared with the distal marker p _0, is constrained to put closer to the observer. Under such a constraint condition, since the two markers are the identification tombs, the target designated position can be obtained as the position coordinate: p ₀ direction vector: p ₀ −p _{1 in the} same manner as in the case where it can be identified by color. it can.

A case will be described in which an indicator having three markers is used. As an example of such an indicator, the indicator 20 in FIG.
There is 0. The positions of the three markers are represented as p ₀ , p ₁ , and p ₂ . If the three markers are all different colors,
The position of the designated position designated by the user using the indicator 40 is represented by an orthogonal coordinate system uvw defined by the following equation (see also FIG. 20): w = p ₁ −p ₀ u = (p ₀ −p ₁₎ ) × w (3) The origin position (vector p ₀ in the world coordinate system xyz) of v = w × u
And the attitude is represented by the vector p ₀ -p ₁ .

When all three markers have the same color
Has the following constraints: • Tip marker p₀Is always the marker farthest from the observer
And the upper end marker p_TwoIs the rear end marker p₁More always above
If all three markers have different colors,
Position p above ₀And posture p₀-P₁And the same
Can be obtained.

Also, two of the three markers are
Other markers of the same color (eg, color A)
If the color is different from (for example, color B),
Constraint condition, ・ Tip marker p₀Is B, the top marker p_Two
Is the rear end marker p₁It is always above, or • a trailing marker p₁Is B, the tip marker p₀
Is the top marker p_TwoMore always ahead or below
Or the top marker p_TwoIs B, the tip marker p₀
Is the rear end marker p₁If the three markers are all different colors,
Position p above ₀And posture p₀-P₁And the same
Can be obtained.

The tablet indicator diagram 21 shows a case indicator is a tablet-type indicator 300. The tablet-type indicator 300 is provided with three markers: an origin marker (vector p ₀ ), an upper right marker (vector p ₁ ), and an upper left marker (vector p ₂ ). These marker positions are as shown in FIG. 22 in the world coordinate system.

If the origin marker, the upper right marker, and the upper left marker are all different colors, the user
The position of the designated position designated by using is represented by an orthogonal coordinate system uvw defined by the following equation (see also FIG. 23): w = p ₀ −p ₂ u = w × (p ₁ −p ₀ ). 4) The origin is indicated by u = v × w (the position of the vector p ₀ in the world coordinate system xyz), and the posture is indicated by the vector p ₀ −p ₂ .

When all three markers have the same color
Has the following constraints: • Tip marker p₀Is always the marker closest to the observer
And the upper left marker p_TwoIs the upper right marker p₁More always to the left
If all three markers have different colors,
Position p above ₀And posture p₀-P_TwoAnd the same
Can be obtained.

[0053]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below. [Embodiment 1] The MR system of Embodiment 1 is an optical See-Th
Head CT (Comput) by user wearing rough type HMD
The present invention is applied to an instruction for a cut surface of er Tomography).

FIG. 24 shows a head model to be subjected to CT in the first embodiment. The head model is similar in appearance to a human head, and CT data of the human head is stored in advance in a computer device (not shown). The user uses the three-point marker type indicator 200 shown in FIG. 25 to indicate the CT cutting position and the cutting plane. That is, the cutting position is determined by the position of the tip marker of the indicator 200, and the cut surface is a plane formed by the three markers. Actually, as shown in FIG. 26, the indicator 200 is brought close to the head model so as to be at a desired cutting position and a cutting plane, and when reaching the desired position, the event switch 201 is turned on.

In the MR system of the first embodiment, at the cutting position indicated by the tip marker position at the time when the event switch 201 is pressed, the CT data having the plane parallel to the plane formed by the three markers as the cutting plane is used as the disk. And causes the image generation device 70 of the embodiment to generate a CT display image, for example, and displays it on an optical See-Thorugh type HMD.
Thus, the CT image is superimposed on the optical image of the phantom and displayed to the user.

FIG. 27 shows a control procedure of the computer 50 when the computer 50 of the embodiment is used as the computer of the first embodiment. [Embodiment 2] Embodiment 2 is an example using two indicators. In FIG. 28, the user operates the tablet type indicator 3
Holding the 00 in the left hand, the 3-marker stylus type indicator 20
I have a 0 in my right hand. The tablet type indicator 300 has markers 405, 406 and 407, and the three-marker stylus type indicator 200 has markers 210, 211 and 212. Since the two indicators are used in close proximity, they all have different colors so that the markers between the indicators are not confused.

The application program of the MR presentation system according to the second embodiment provides an optical See-Through that the user wears as if images 401, 402, 403, and 404 are displayed on the surface of the tablet type indicator 300. Send a virtual CG image to the type HMD. By viewing the CG projected in the HMD, the user can view the images 401, 402, 403, 40 on the tablet surface.
4 is perceived as being displayed. This is because the markers 50, 406, and 407 cause the computer 50 to indicate the indicator 3
The three-dimensional position and orientation of the 00 surface can be detected,
The left and right parallax images are output to the HMD so as to have a stereoscopic effect as if a virtual image is displayed at that position.

On the surface of the tablet type indicator 300, the virtual image 401 represents a tomographic image of the head, and the virtual images 402,
403 and 404 are virtual icons, respectively. The stylus type indicator 200 and the application program of the computer 50 provide the user with a user interface for selecting a virtual icon. That is, when the user brings the tip of the stylus type indicator 200 close to the virtual icon 404 and presses the switch 201, it is determined that the print command has been selected.

Next, when the stylus type indicator 200 is brought close to an arbitrary point 408 on the virtual image 401 of the CT tomogram and the switch 201 is pressed, the position 408 is selected by the user as an object to be enlarged / reduced. The application program decides. Next, the user brings the stylus-type indicator 200 close to the virtual icon 402 to switch 2.
When the user presses 01, it is determined that the user has selected the enlarged display of the area 408, the CT data is retrieved and retrieved from a disk or the like, and a left / right parallax image of the CT image is generated to generate an HMD.
Output to When the virtual icon 403 is selected, the reduction function is selected.

A significant effect of the present invention is that the number of calibration steps is reduced by one step. When two optical indicators are used as in the second embodiment, the calibration is reduced by one step by one optical indicator.
2 compared to the conventional method using two magnetic sensors
The number of processes is reduced, and the effect is great. <Modifications> The present invention is not limited to the above embodiments and examples. A modification is proposed below.

This modified example is an MR presentation system in which the method of the above embodiment and the conventional method are used together. In this modification, as shown in FIG. 29, a position sensor 401 such as a magnetic sensor is used in order to use the method of the related art (positioning based on the world coordinate system) in some cases.
The indicator 400 provided with is used. The indicator 400 has markers 40 at both ends in order to apply to the method of the present embodiment.
2,403 are provided.

When the user moves using the indicator 400 to indicate the indicated position, the marker is not always caught by the camera 22. In case of such a case,
The position sensor 401 is provided on the indicator 400, and the position and orientation of the indicator 400 are measured using the output signal of the sensor according to a conventional method. As described above, only when the marker is hidden, the position and orientation of the indicator 400 with respect to the world coordinate system is measured using the output of the position sensor 401, and as shown in FIG. Measurements are made via In other words, according to this modification, the range of action of the user is significantly widened, and the constraint condition is relaxed.

<Other Modifications> Modification 1 The number of markers is not limited to two or three.
If only the degree of freedom to be measured is sufficient, the position of the indicator can be measured in principle even with only one marker. Modified Example 2: The marker is not limited to a light emitting type such as an LED. For example, a fluorescent tape or a tape of a different color may be used. Modification Example 3 In a stylus-type indicator, it is not preferable to provide a marker at the tip of the stylus. This is because, for example, a pen or a scalpel for a doctor may be used for the indicator. In this case, the designated position is made at the tip of the stylus, and the tip position is different from the marker position. Therefore, as shown in FIG.

[0064]

(Equation 1)

The correction is made according to the following.

[0066]

As described above, according to the present invention, the number of times of calibration can be reduced by reducing the generated positional deviation.

[Brief description of the drawings]

FIG. 1 is a diagram illustrating a state in which an observer inputs a designated position via an operation tool (indicator) in the related art.

FIG. 2 is a diagram for explaining the operation principle of a magnetic position sensor used in the prior art and used in some embodiments of the present invention.

FIG. 3 is a diagram illustrating a configuration and an operation principle of an indicator used in the related art and used in some embodiments of the present invention.

FIG. 4 is a view for explaining the principle of generation of a defect (position shift) in the related art.

FIG. 5 is a view for explaining the principle of occurrence of a defect (position shift) in the conventional technique.

FIG. 6 is an exemplary view for explaining the configuration of a mixed reality presentation system according to the embodiment;

FIG. 7 is an exemplary view for explaining that the indicator functions as a man-machine interface device in the mixed reality presentation system of the embodiment;

FIG. 8 is an exemplary view for explaining an operation method of the indicator used in the embodiment.

FIG. 9 is a flowchart illustrating a procedure for detecting a position of a marker provided on the indicator in the system according to the embodiment.

FIG. 10 is a block diagram illustrating a data processing system of the system according to the embodiment.

FIG. 11 is an exemplary view for explaining the reason why no displacement occurs in the system according to the embodiment;

FIG. 12 is a flowchart of a control procedure used in the system of the embodiment.

FIG. 13 is a view for explaining the correspondence of markers between two left and right images.

FIG. 14 is a view for explaining the principle of detecting a three-dimensional position of a marker in the embodiment.

FIG. 15 is a view for explaining a method of determining a marker three-dimensional position when straight lines do not intersect when detecting the three-dimensional position of the marker.

FIG. 16 is an exemplary view for explaining the structure of a two-marker type indicator (stylus type) used in the system according to the embodiment;

FIG. 17 is a view for explaining the principle of determining an indicated position and orientation when the indicator of FIG. 16 is used.

FIG. 18 is an exemplary view for explaining the structure of a three-marker type indicator (stylus type) used in the system according to the embodiment;

FIG. 19 is a view for explaining the principle of determining an indicated position and orientation when the indicator of FIG. 18 is used.

FIG. 20 is a view for explaining the principle of determining an indicated position and orientation when the indicator of FIG. 18 is used.

FIG. 21 is an exemplary view for explaining the structure of a tablet type indicator used in the system according to the embodiment;

FIG. 22 is a view for explaining the principle of determining an indicated position and orientation when the indicator of FIG. 21 is used.

FIG. 23 is a view for explaining the principle of determining the designated position and orientation when the indicator of FIG. 21 is used.

FIG. 24 is a specific application example 1 of the present invention;
The figure explaining the head model used for the.

FIG. 25 is a diagram illustrating the structure of a stylus-type indicator used in the first embodiment.

FIG. 26 is a view showing a state when the indicator of FIG. 25 is used for determining the pointing position of the head model of FIG. 24;

FIG. 27 is a flowchart illustrating a control procedure according to the first embodiment.

FIG. 28 shows two indicators (30) used in the second embodiment.
(0, 200).

FIG. 29 is a diagram illustrating a configuration of a presentation system according to another embodiment of the present invention.

FIG. 30 is a view for explaining the principle of another embodiment.

FIG. 31 is a view for explaining the principle of correcting the position coordinates of the tip end position of the indicator in still another modification.

 ──────────────────────────────────────────────────続 き Continued on the front page F term (reference) 5B050 BA04 BA11 CA07 DA04 EA07 EA19 EA27 EA28 FA02 FA06 FA08 FA09 5B087 AA07 AE00 BC05 BC12 BC13 BC16 BC17 BC32 BC34 DD03 DH04 DJ03 5C061 AA01 AA29 AB11 AB12 AB16 AB18 AB24

Claims (24)

[Claims] 1. A mixed reality presentation system that presents an observer by fusing a virtual space and a real space, wherein the three-dimensional presentation position is provided near an observer's viewpoint position and a presentation position at which a virtual image is to be presented. First sensor means for detecting a coordinate value in accordance with a coordinate system related to the viewpoint position, and a virtual image to the observer at a presentation position detected by the sensor means with reference to the observer's viewpoint position. A mixed reality presentation system comprising: a virtual image presentation means for presenting. 2. The apparatus according to claim 1, wherein the first sensor unit has an image pickup unit provided at a position of a head of the observer, and the virtual image presenting unit has a head mounted display mounted on the head of the observer. The mixed reality presentation system according to claim 1, wherein the system is provided. 3. The mixed reality presentation system according to claim 2, wherein the head mounted display is of a video see-through type. 4. The first sensor means comprises: a stereo camera mounted on a head and separated from side to side; and, based on stereo image data output by the stereo camera, a stereo view method, 4. The mixed reality presentation system according to claim 1, further comprising: means for detecting a three-dimensional coordinate value on an image plane of the stereo camera. 5. The mixed reality presentation system according to claim 1, further comprising second sensor means for detecting a viewpoint position of the observer according to a world coordinate system. . 6. The mixed reality presentation system according to claim 1, further comprising an operation tool operated by an observer within a sensing range of said sensor means. 7. The operating tool is provided with a switch that is operated by the observer and outputs a predetermined event timing signal by the observer.
3. The mixed reality presentation system according to [1]. 8. The mixed reality presentation system according to claim 6, wherein the operation tool has at least one marker provided at an end thereof. 9. The mixed reality presentation system according to claim 8, wherein one marker is provided, and the sensor unit outputs a three-dimensional coordinate position of the presentation position. 10. The mixed reality presentation system according to claim 8, wherein two markers are provided, and the sensor unit outputs a three-dimensional coordinate position and a direction of the presentation position. 11. The mixed reality presentation system according to claim 8, wherein the three markers are provided, and the sensor outputs a plane coordinate passing through the presentation position. 12. The marker according to claim 8, wherein the marker has a plurality of markers colored differently.
The mixed reality presentation system according to any one of the above. 13. The mixed reality according to claim 8, wherein the markers include a plurality of markers colored differently and at least one marker colored the same. Feeling presentation system. 14. The mixed reality presentation system according to claim 1, wherein said marker has a light emitting means for emitting different colors. 15. The mixed reality presentation system according to claim 1, further comprising a plurality of operation tools that are operated within a sensing range of said sensor means and operated by an observer. 16. The mixed reality presentation system according to claim 6, wherein the operation tool further includes a magnetic sensor as a position sensor. 17. A mixed reality presentation method in which a virtual space and a real space are combined and presented to an observer, using a sensor provided near a viewpoint position of the observer,
A three-dimensional coordinate value of a presentation position at which a virtual image is to be presented is detected in accordance with a coordinate system related to this viewpoint position.The virtual image is displayed at the presentation position detected by the sensor based on the viewpoint position of the observer. A mixed reality presentation method, characterized by presenting to the observer. 18. A computer-readable storage medium storing a program for presenting to an observer by fusing a virtual space and a real space, wherein a sensor provided near a viewpoint position of the observer is used.
A program code for detecting a three-dimensional coordinate value of a presentation position at which a virtual image is to be presented according to a coordinate system related to the viewpoint position, and a presentation position detected by the sensor with reference to the viewpoint position of the observer. A computer-readable storage medium storing a program code for presenting a virtual image to the observer. 19. A man-machine interface for inputting a presentation position at which a virtual image is to be presented to a mixed reality presentation system via an operation tool operated by a user, the man-machine interface being provided near an observer's viewpoint position. An imaging unit including the operating tool in an imaging visual field; and a coordinate system related to a viewpoint position of the observer, based on image data obtained by the imaging unit, and a three-dimensional coordinate value of a predetermined portion of the operating tool. Conversion means for converting the three-dimensional coordinate value of the predetermined part into a virtual image presentation position instructed by an observer, to the mixed reality presentation system, A man-machine interface device comprising: 20. A man-machine interface method for inputting a presentation position at which a virtual image is to be presented to a mixed reality presentation system via an operation tool operated by a user, the method comprising: An imaging unit is arranged so as to include the operating tool in an imaging visual field, and a three-dimensional coordinate value of a predetermined portion of the operating tool is related to a viewpoint position of the observer based on image data obtained by the imaging unit. And converting the converted three-dimensional coordinate values of the predetermined part to the mixed reality presentation system as a virtual image presentation position designated by an observer. Man-machine interface method. 21. A man-machine interface for inputting a user instruction to a mixed reality presentation system via an operation tool operated by a user, wherein the man-machine interface is provided near an observer's viewpoint position and captures an image of the operation tool. Imaging means included in a field of view, based on image data obtained by the imaging means, three-dimensional coordinate values of a predetermined portion of the operating tool, three-dimensional coordinates according to a coordinate system related to the viewpoint position of the observer A man-machine interface device, comprising: a conversion unit for converting the three-dimensional coordinate value of the predetermined part into a user instruction input to the mixed reality presentation system; . 22. A man-machine interface method for inputting a user's instruction to a mixed reality presentation system via an operation tool operated by a user, wherein the operation tool is imaged near an observer's viewpoint. The imaging means is arranged so as to be included in the image data, based on the image data obtained by the imaging means, the three-dimensional coordinate value of a predetermined portion of the operating tool, according to the coordinate system related to the viewpoint position of the observer A man-machine interface method, wherein the method converts the three-dimensional coordinate value of the predetermined part into a three-dimensional coordinate value and outputs the converted three-dimensional coordinate value to the mixed reality presentation system as a user instruction input. 23. An image pickup means is arranged near an observer's viewpoint position so as to include the operation tool in an imaging field of view, and a presentation position at which a virtual image is to be presented via an operation tool operated by a user. A computer-readable recording medium that stores a man-machine interface program to be input to the mixed reality presentation system, based on image data obtained by the imaging unit, a three-dimensional coordinate value of a predetermined portion of the operating tool, A program code for converting to a three-dimensional coordinate value according to a coordinate system related to the viewpoint position of the observer; and the converted three-dimensional coordinate value of the predetermined part as a virtual image presentation position designated by the observer. A computer-readable recording medium for storing a program code to be output to a mixed reality presentation system. 24. An image pickup means is arranged near an observer's viewpoint position so that the operation tool is included in an imaging field of view, and a user instruction is input to the mixed reality presentation system via the operation tool operated by a user. A computer-readable recording medium that stores a man-machine interface program to be input, based on image data obtained by the imaging unit, the three-dimensional coordinate value of a predetermined portion of the operating tool, the viewpoint position of the observer A program code for converting to a three-dimensional coordinate value according to a coordinate system related to, and a program code for outputting the converted three-dimensional coordinate value of the predetermined part to the mixed reality presentation system as a user instruction input. A computer-readable recording medium characterized by storing.