JP2004013309A - Information processing method and apparatus for presenting augmented reality - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To superpose intended CG at all times regardless of the kinds of sensors mounted by making it possible to select suitable CG according to the positional relationship of sensors, and avoiding unexpected, unnatural CG superposition due to errors in mounting the sensors. <P>SOLUTION: A plurality of sensors are installed in different positions. After the sensors are installed, the sensors are matched against computer graphics according to the output values of the plurality of sensors. Based on this matching, computer graphics matching the sensors are synthesized in the positions corresponding to the output values obtained from the plurality of sensors. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to a technique for combining computer graphics with a position corresponding to an output value of a sensor with respect to an actual image.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, a real-world image and a three-dimensionally modeled CG image have been superimposed and displayed, and an object (virtual object) drawn by CG (computer graphics) exists in the real world. There is a mixed reality presenting apparatus that can make it look like this. This is achieved by combining a real image photographing means (for example, a video camera) for photographing a real image, and a CG image generating means for producing a CG image as viewed from the position where the real image is being photographed. It is an apparatus comprising video display means (for example, an HMD (head-mounted display) or a monitor) that can be displayed. Here, even if the position of the line of sight of the real image photographing means changes, the positional relationship between the CG and the real image becomes correct, and the line of sight and the line of sight of the real image photographing means (video camera). Is provided with gaze position / posture detection means (for example, a position / posture sensor) for detecting the position. The CG image generation means places the CG modeled in three dimensions in a virtual space of the same scale as the real space, and renders the CG as viewed from the gaze position and gaze direction detected by the gaze position / posture detection means. When the CG image generated in this way and the real image are superimposed, as a result, even if the real image photographing means observes from any gaze position and gaze direction, the image generated by the CG correctly in the real space is obtained. Can be displayed.
[0003]
Further, an additional position and orientation detection device is attached to a place where a CG is desired to appear, and the CG is caused to appear at a position obtained therefrom. For example, a position / posture detection device (position / posture sensor) is attached to a hand, and a CG is generated at the position of the sensor, so that the CG is always on the hand (no matter how the hand is moved, (As if a CG is on top of the camera).
[0004]
The real image photographing means for photographing a picture in the real space is, for example, a video camera, and is a means for photographing a picture in a line of sight of the camera and capturing it in a memory.
[0005]
For example, an HMD (Head Mount Display) is used as an image display device that synthesizes and displays an image of a real space and CG. By using an HMD instead of a normal monitor and further mounting the video camera in the direction of the line of sight of the HMD, an image in the direction in which the observer is facing can be displayed on the HMD, and when facing in that direction CG can also be drawn, so that the immersion feeling of the observer can be enhanced.
[0006]
As the position and orientation detection means, a magnetic position and orientation sensor or the like is used. By attaching this to the video camera (or the HMD to which the video camera is attached), the position and orientation values of the line of sight of the video camera can be obtained. To detect. The magnetic position / posture sensor detects a relative position / posture between a magnetic generator (transmitter) and a magnetic sensor (receiver), and includes a product FASTRAK manufactured by Polhemus, USA. Can be This is a device that detects the three-dimensional position (X, Y, Z) and posture (Roll, Pitch, Yaw) of the sensor in a specific area in real time.
[0007]
With the above configuration, the observer can observe the world in which the real video and the CG video are superimposed through the HMD. When the observer looks around, the real image capturing device (video camera) provided on the HMD captures a real image, and the gaze position / posture detecting means (position / posture sensor) provided on the HMD provides position gaze of the video camera. The direction is detected, and the CG image generation means generates a CG image viewed from the line-of-sight position / posture in response to the direction, and displays the CG image superimposed on the real image.
[0008]
It is also possible to provide a plurality of observers. A video camera, a display device (such as an HMD) for capturing the real space, and a position and orientation sensor are prepared for the number of observers, and the real space is photographed and a CG is generated from each observer's viewpoint, and synthesized, and each is combined. It may be displayed to the observer.
[0009]
[Problems to be solved by the invention]
In the conventional system, when CG is superimposed on the position of a sensor, the CG is associated with the ID of the sensor in advance, and a certain type of CG is superimposed on the position of a certain sensor. For example, when it is desired to superimpose the glove CG on the player's hand and the boot CG on the foot, it is necessary to correctly mount a position sensor corresponding to the hand CG on the hand and a position sensor corresponding to the foot CG on the foot. was there. If they were worn by mistake, the CG of the boot was superimposed on the hand and the CG of the glove was superimposed on the foot.
[0010]
In addition, when different types of CGs are to be superimposed when mounted on the right hand and when mounted on the left hand, it is necessary to prepare two sensors for the right hand and the left hand.
[0011]
The present invention has been made in view of the above-mentioned problems, and the invention of claim 1 of the present application can select a suitable CG according to the positional relationship of the sensor, and can prevent unexpected CG due to incorrect mounting of the sensor. It is an object of the present invention to avoid natural CG superimposition and to always superimpose CG as intended regardless of which sensor is attached.
[0012]
It is an object of the invention of claim 5 of the present application to enable the CG to be superimposed to be dynamically changed according to the positional relationship of the sensors.
[0013]
[Means for Solving the Problems]
In order to achieve the above object, the present invention has the following configuration.
[0014]
The invention according to claim 1 is an information processing method for synthesizing computer graphics at a position corresponding to an output value of a sensor with respect to an actual image, wherein a plurality of sensors are installed in different parts, and the sensors are installed. After that, each sensor and computer graphics are associated with each other from the output values of the plurality of sensors, and based on the association, a position corresponding to each output value obtained from each of the plurality of sensors is associated with each sensor. Computer graphics to be synthesized.
[0015]
According to a fifth aspect of the present invention, there is provided a computer graphics image generating means for generating a computer graphics image, a real video image capturing means for capturing a real world video, and a display device which combines the computer graphics image and the captured image. In a mixed reality presentation device having video display means for displaying, a position sensor for detecting a position where a computer graphics image is superimposed, and a type of computer graphics to be superimposed on a position obtained from the position sensor are moved. And a CG selecting means for making a judgment.
[0016]
BEST MODE FOR CARRYING OUT THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0017]
In the present embodiment, the player wears position and orientation sensors on the head, hands, and feet, and superimposes CG on the position where the sensors on the hands and feet are worn to present mixed reality. Here, as an example, it is assumed that the CG of the glove is superimposed on the hand and the CG of the boot is superimposed on the foot. The head position sensor is used to detect the direction of the line of sight of a video camera attached to the HMD.
[0018]
FIG. 1 is a configuration diagram illustrating the present embodiment. CPU 101, memories 103 and 104, position and orientation sensors 105 (a), 105 (b), 105 (c), 105 (d), 105 (e), HMD 106 (a), video camera 107 (a) 102. The memories 103 and 104 are the same. A plurality of sensors 105 (b), 105 (c), 105 (d), 105 (e) are attached to a sensor body 105 (a), and 105 (a) is connected to the computer bus 102, (B), 105 (c), 105 (d) and 105 (e) are attached to the sensor body 105 (a). The sensor 105 (b) is attached to the HMD 106, and is used for detecting the position and orientation of the line of sight of the video camera 107 and also for detecting the position and orientation of the player's head. The sensor 105 (c) is attached to the player's hand and is used for detecting the position and orientation of the hand to superimpose CG on the hand. The position sensor 105 (d) is mounted on the foot of the player, and is used for detecting the position and orientation of the foot to superimpose CG on the foot. Here, an ID is assigned to each position sensor. Here, the ID of 105 (b) is 1, the ID of 105 (c) is 2, and the ID of 105 (d) is 3.
[0019]
Note that the system may include a plurality of players. Also, here, three sensors are provided for one player (attached to the player's head, hands, and feet), but there is no problem as long as necessary.
[0020]
The memory 104 also stores CG model data 125. This is the CG data to be rendered by the CG image generating means 111. In this example, the CG of the glove and the CG of the boot are stored.
[0021]
FIG. 2 is a flowchart for explaining the flow of the entire process.
[0022]
First, in step S201, the CG position / posture detection unit 113 detects the position / posture of each sensor after installing the sensor at an arbitrary part, and stores the position / posture of the CG position / posture 122 in the memory 104. In this embodiment, since the position sensors 105 (c) (ID = 2) and 105 (d) (ID = 3) superimpose CG, the position and orientation of these two sensors are detected and recorded. The CG position / posture detection means 113 detects the position (XYZ) and posture (Roll, Pitch, Yaw) of the line of sight using a conventionally used technology (such as a magnetic sensor).
[0023]
Next, in step S202, a CG suitable for each sensor is selected by the CG selecting means 115. Details of the CG selection means 115 will be described later. By this step, the combination of the sensor ID and the CG corresponding to the sensor ID is stored in the CG correspondence table 124 in the memory 104.
[0024]
Next, in step S203, the gaze position / posture detection unit 112 writes the position and posture of the video camera 107 into the gaze position / posture 121 in the memory 104. The line-of-sight position / posture detecting means 112 detects the position (XYZ) and posture (Roll, Pitch, Yaw) of the line of sight by using a conventionally used technology (such as a magnetic sensor). Here, detection is performed using the position sensor 105 (b) (ID = 1).
[0025]
Next, in step S204, the real image photographing means 110 writes the image of the real space into the image memory 121 in the memory 104. The real image capturing means 110 captures an image captured by the video camera 107 and writes the captured image into the image memory 120 in the memory 104.
[0026]
In step S205, the variable N is initialized to 1. The variable N is for counting the number of position and orientation sensors for CG. In the present embodiment, two sensors for CG are the sensors 105 (c) and 105 (d) in FIG.
[0027]
In step S206, the CG position / posture detecting unit 113 detects the position / posture of the position sensor attached to the place where the N-th CG should be displayed, and stores it in the CG position / posture 122 in the memory 104.
[0028]
In step S207, a CG associated with the ID of the sensor is selected from the CG model data 123 according to the CG correspondence table 124, and the CG is set to an appropriate position according to the position and orientation detected in the previous step S206. And the posture.
[0029]
Next, in step S208, the variable N is counted up by one, and in the next step S209, it is determined whether N has exceeded the number of CGs (2 in this embodiment). In this way, the position and orientation of the CG are operated for all the CG position sensors.
[0030]
When the operation has been completed for all CGs, the CG image generation unit 111 writes the CGs into the image memory 120 in step S210. The CG is rendered as if the object placed in the position and orientation indicated by the CG position and orientation 122 was observed from the line of sight of the line of sight position and orientation 121. The image of the real space has already been written in the image memory 120 by the actual image photographing means 110 in step S203, and the CG image is drawn over this.
[0031]
Next, in step S211, the video written in the video memory 120 is displayed on the HMD 106 by the video display unit 114. The video display means 114 writes the video memory into the frame memory of the HMD, and performs actual display.
[0032]
In step S212, it is determined whether or not to end the entire process. If not, the process returns to step S203 and repeats the process. By repeating the processing, a video is sequentially captured from the video camera 107, a CG video is overwritten on the video, and a cycle of displaying the video is continuously performed.
[0033]
Next, the CG selection unit 115 in step S202 will be described.
[0034]
FIG. 3 is a flowchart illustrating the CG selection unit 115.
[0035]
In this embodiment, the heights of the positions of the two sensors for CG superimposition, 105 (c) (ID = 2) and 105 (d) (ID = 3), are used as selection criteria.
[0036]
In step S301, the heights of the sensor with ID = 2 and the sensor with ID = 3 are compared based on the sensor position data stored in the CG position / posture 122 in the memory 104 in step S201 described above. If the sensor with ID = 2 is at a higher position than the sensor with ID = 3, the process proceeds to step S302, and the CG of the glove is associated with the sensor of ID = 2, and the CG of boots is associated with the sensor with ID = 3. If the sensor with ID = 3 is at a higher position than the sensor with ID = 2, the process proceeds to step S303, where the sensor with ID = 3 is associated with the CG of the glove, and the sensor with ID = 2 is associated with the CG of the glove. This association is stored as a CG correspondence table 124 in the memory 104.
[0037]
At this time, it is possible to use not only the up-down relationship between the two sensors but also the left-right relationship. Further, based on the information on the gaze position obtained from the sensor 105 (b), it is also possible to associate the CG depending on whether the other sensor is mounted on the left or right of the head. For example, when the sensor is attached to the right hand, it can be detected that the sensor is attached to the right hand because the sensor is located to the right of the head.
[0038]
FIG. 4 is a diagram for explaining the effect of the present embodiment. FIG. 4A is a diagram of a player wearing the sensor. FIG. 4B shows a state in which CG is superimposed on the player shown in FIG. FIG. 4C is a view of a player who has worn the sensor by hand and foot. FIG. 4D shows how CG is superimposed when the present embodiment is not used. FIG. 4E shows how the CG is replaced and superimposed in the case of the above (C) when the present embodiment is used. Of course, even when the present embodiment is used, the CG is superimposed as shown in FIG. 4E even when the sensor is mounted as shown in FIG.
[0039]
With the above configuration, it is possible to select a suitable CG according to the positional relationship of the sensors, avoid unexpected and unnatural CG superimposition due to a wrong mounting of the sensor, and always perform the CG as intended regardless of which sensor is mounted. Can be superimposed.
[0040]
That is, it is not necessary to mount the sensor on a predetermined portion, and the load on the user for mounting the sensor can be reduced. Further, it is possible to prevent a mismatch between the CG and the actual image as shown in FIG.
[0041]
(Modification)
If the branch destination from step S212 in FIG. 2 is returned to step S201 instead of step S203, the CG selection is always redone during play. By doing so, it is possible to change the CG to be superimposed during play without replacing the mounted sensor. That is, it is possible to dynamically change the CG to be superimposed according to the positional relationship of the sensors.
[0042]
For example, it is possible to dynamically switch the type of CG between a case where one sensor is mounted on the right hand and a case where the sensor is mounted again on the left hand.
[0043]
(Other embodiments)
A software program for realizing the functions of the above-described embodiment is provided to an apparatus or a computer in the system connected to the various devices so as to operate the various devices so as to realize the functions of the above-described embodiment. The present invention also includes a system in which a computer (CPU or MPU) of the system or apparatus is operated by supplying the code and operating the various devices according to a stored program.
[0044]
In this case, the program code itself of the software realizes the function of the above-described embodiment, and the program code itself and a unit for supplying the program code to the computer, for example, the program code is stored. The storage medium constitutes the present invention.
[0045]
As a storage medium for storing such a program code, for example, a floppy (R) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, nonvolatile memory card, ROM or the like can be used.
[0046]
When the computer executes the supplied program code, not only the functions of the above-described embodiments are realized, but also the OS (Operating System) in which the program code runs on the computer, or other application software. It goes without saying that such a program code is also included in the embodiment of the present invention when the functions of the above-described embodiment are realized in cooperation with the above.
[0047]
Further, after the supplied program code is stored in a memory provided in a function expansion board of a computer or a function expansion unit connected to the computer, a CPU or the like provided in the function expansion board or the function storage unit based on the instruction of the program code. It is needless to say that the present invention includes a case in which the functions of the above-described embodiments are implemented by performing part or all of the actual processing.
[0048]
【The invention's effect】
According to the invention of claim 1 of the present application, it is possible to select a suitable CG according to the positional relationship of the sensors, avoid unexpected and unnatural CG superposition due to mounting errors of the sensors, and use any sensor. CG can be always superimposed as intended.
[0049]
According to the invention of claim 5 of the present application, it is possible to dynamically change the CG to be superimposed according to the positional relationship of the sensors.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of an embodiment.
FIG. 2 is a flowchart of a process according to the embodiment.
FIG. 3 is a flowchart of a CG selection unit.
FIG. 4 is an explanatory diagram for explaining effects of the embodiment.

Claims (7)

An information processing method for synthesizing computer graphics at a position corresponding to an output value of a sensor with respect to an actual image,
Install multiple sensors at different locations,
After installing the sensor, perform the correspondence between each sensor and computer graphics from the output values of the plurality of sensors,
An information processing method, comprising: compositing computer graphics corresponding to each sensor at a position corresponding to an output value obtained from each of the plurality of sensors based on the association. 2. The information processing method according to claim 1, wherein the association is performed based on a distinction between upper and lower or left and right output values of the plurality of sensors. 2. The information processing method according to claim 1, wherein the association is performed at the start of a play. A program for realizing the information processing method according to claim 1. CG image generating means for generating a computer graphics image;
A real image shooting means for shooting images of the real world,
In a mixed reality presentation device having video display means for combining the computer graphics image and the captured image and displaying the combined image on a display device,
A position sensor for detecting a position at which the computer graphics image is superimposed,
CG selecting means for dynamically determining the type of computer graphics to be superimposed on the position obtained from the position sensor. A CG image generating step of generating a computer graphics image;
A real video shooting process for shooting real world video,
A video display step of combining the computer graphics image and the photographed image and displaying the combined image on a display device,
An input step of inputting an output value from a position sensor for detecting a position where a computer graphics image is superimposed,
A CG selecting step of dynamically determining a type of computer graphics to be superimposed on a position corresponding to an output value from the position sensor. An information processing method according to claim 1, wherein said CG selection step is always making a determination during play.

Images