JP2004013310A - Information processing method and apparatus for presenting augmented reality - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To create CG images in sizes matching players' physiques. <P>SOLUTION: An apparatus for presenting augmented reality includes a computer graphics image creating means for creating CG images; a real image taking means for taking images of the real world; an image display means for synthesizing the CG images with the real images and displaying the results in a display device; a physique sensing means for sensing the players' physiques; and a CG size setting means for setting the size of the CG images created by the CG image creating means, according to the players' physiques data sensed by the physique sensing means. <P>COPYRIGHT: (C)2004,JPO

Description

[0001]
TECHNICAL FIELD OF THE INVENTION
The present invention relates to an apparatus for synthesizing a computer graphics image generated on a real image.
[0002]
[Prior art]
2. Description of the Related Art Conventionally, an image of a real world and an image of a three-dimensionally modeled CG (computer graphics image) are superimposed and displayed, and as if an object (virtual object) drawn by the CG exists in the real world. There is a mixed reality presentation device that can make it look as if it were a creature. This is achieved by combining a real image photographing means (for example, a video camera) for photographing a real image, and a CG image generating means for producing a CG image as viewed from the position where the real image is being photographed. It is an apparatus comprising video display means (for example, an HMD (head-mounted display) or a monitor) that can be displayed. Here, even if the position of the line of sight of the real image photographing means changes, the positional relationship between the CG and the real image becomes correct, and the line of sight and the line of sight of the real image photographing means (video camera). Is provided with gaze position / posture detection means (for example, a position / posture sensor) for detecting the position. The CG image generation means places the CG modeled in three dimensions in a virtual space of the same scale as the real space, and renders the CG as viewed from the gaze position and gaze direction detected by the gaze position / posture detection means. When the CG image generated in this way and the real image are superimposed, as a result, even if the real image photographing means observes from any gaze position and gaze direction, the image generated by the CG correctly in the real space is obtained. Can be displayed.
[0003]
Further, an additional position and orientation detection device is attached to a place where a CG is desired to appear, and the CG is caused to appear at a position obtained therefrom. For example, a position / posture detection device (position / posture sensor) is attached to a hand, and a CG is generated at the position of the sensor, so that the CG is always on the hand (no matter how the hand is moved, (As if a CG is on top of the camera).
[0004]
The real image photographing means for photographing a picture in the real space is, for example, a video camera, and is a means for photographing a picture in a line of sight of the camera and capturing it in a memory.
[0005]
For example, an HMD (Head Mount Display) is used as an image display device that synthesizes and displays an image of a real space and CG. By using an HMD instead of a normal monitor and further mounting the video camera in the direction of the line of sight of the HMD, an image in the direction in which the observer is facing can be displayed on the HMD, and when facing in that direction CG can also be drawn, so that the immersion feeling of the observer can be enhanced.
[0006]
As the position and orientation detection means, a magnetic position and orientation sensor or the like is used. By attaching this to the video camera (or the HMD to which the video camera is attached), the position and orientation values of the line of sight of the video camera can be obtained. To detect. The magnetic position / posture sensor detects a relative position / posture between a magnetic generator (transmitter) and a magnetic sensor (receiver), and includes a product FASTRAK manufactured by Polhemus, USA. Can be This is a device that detects the three-dimensional position (X, Y, Z) and posture (Roll, Pitch, Yaw) of the sensor in a specific area in real time.
[0007]
With the above configuration, the observer can observe the world in which the real video and the CG video are superimposed through the HMD. When the observer looks around, the real image capturing device (video camera) provided on the HMD captures a real image, and the gaze position / posture detecting means (position / posture sensor) provided on the HMD provides position gaze of the video camera. The direction is detected, and the CG image generation means generates a CG image viewed from the line-of-sight position / posture in response to the direction, and displays the CG image superimposed on the real image.
[0008]
It is also possible to provide a plurality of observers. A video camera, a display device (such as an HMD) for capturing the real space, and a position and orientation sensor are prepared for the number of observers, and the real space is photographed and a CG is generated from each observer's viewpoint, and synthesized, and each is combined. It may be displayed to the observer.
[0009]
[Problems to be solved by the invention]
In a conventional system, when CG is superimposed on a player's body, CG of the same size is superimposed regardless of the player's body type. For this reason, there is a problem that a CG having a size significantly different from the player's body shape is superimposed, resulting in an unnatural appearance and an unnatural operation.
[0010]
For example, in a mixed reality system in which a helmet-type CG is superimposed on a head to make it appear as if a helmet is worn, when the height of a player is extremely lower than an assumed height (for example, when a child There is a problem that the appearance becomes ugly.
[0011]
Also, for example, in a mixed reality system in which a palm-sized water surface appears to be on a hand, if the player's palm is smaller than an assumed size, the superimposed CG water surface is There is a problem such that it protrudes greatly.
[0012]
An object of the present invention is to solve these problems and to generate a CG image in a size suitable for a player's body shape.
[0013]
[Means for Solving the Problems]
In order to achieve the above object, the invention according to claim 1 of the present application is an information processing method for superimposing a generated computer graphics image on a real image, comprising inputting body type information of a player and responding to the body type information. The size of a computer graphics image is set, the computer graphic image is generated based on the set size, and the generated computer graphic image is superimposed on a real image.
[0014]
According to a sixth aspect of the present invention, there is provided a CG image generating means for generating a computer graphics image, a real image shooting means for shooting a real world image, and a display device for synthesizing the computer graphics image and the shot image. A computer graphic image generated by the CG image generating means according to the player's body data detected by the body detecting means. And a CG size setting unit that performs the setting.
[0015]
BEST MODE FOR CARRYING OUT THE INVENTION
(Embodiment 1)
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0016]
The present embodiment is a system having two players. Each player wears a position and orientation sensor on the head and hand, and presents mixed reality by superimposing CG on the position where each sensor is mounted. System. Here, as an example, it is assumed that the CG of the helmet is superimposed on the head and the CG of the glove is superimposed on the hand.
[0017]
FIG. 1 is a configuration diagram illustrating the present embodiment. CPU 101, memories 103 and 104, position and orientation sensors 105 (a), 105 (b), 105 (c), 105 (d), 105 (e), HMD 106 (a), video camera 107 (a) 102. The memories 103 and 104 are the same. A plurality of sensors 105 (b), 105 (c), 105 (d), 105 (e) are attached to a sensor body 105 (a), and 105 (a) is connected to the computer bus 102, (B), 105 (c), 105 (d) and 105 (e) are attached to the sensor body 105 (a). The sensor 105 (b) is attached to the HMD 106, and is used to detect the position and orientation of the line of sight of the video camera 107 and also to detect the position and orientation of the head of the first player. The sensor 105 (c) is attached to the hand of the first player, and is used for detecting the position and orientation of the hand to superimpose the CG on the hand. The position sensor 105 (d) is mounted on the head of the second player, and 105 (e) is mounted on the hand of the second player. Note that the second player may have the HMD and be able to observe the mixed reality space, but the description is omitted here. Also, here, the number of sensors is four (attached to the heads and hands of the two players), but there is no problem as long as necessary.
[0018]
The memory 104 also stores CG model data 125. This is the CG data rendered by the CG image generating means 111. In this example, the CG of the helmet and the CG of the glove are stored. The size of each CG is made according to the assumed body shape at the time of performing the CG modeling. As the assumed body data, assumed body data 126 is stored in the memory 104. In the following example, the height is used as the body type. Therefore, in this embodiment, it is assumed that the assumed height is stored as the assumed body type data.
[0019]
FIG. 2 is a flowchart for explaining the flow of the entire process.
[0020]
First, in step S 201, the body type of each player is detected by the body type detecting unit 114 and stored in the body type data 124 in the memory 104. In the present embodiment, the approximate height of the player is detected and used as body data. The body type detection means 114 obtains the height of each player using the output values from the position and orientation sensors 105 (b) and 105 (d) mounted on the head of each player.
[0021]
In the present embodiment, in step S201, the player has a predetermined pose, a height component is obtained from the output value of the position and orientation sensor 105 (b) or 105 (d) at that time, and the height component is used as body data. Used.
[0022]
In addition, it is also possible to use a position sensor attached to each player's hand as a standard detection means, detect the player's amplitude data by measuring the position by having the user touch the body with his hand, and use it as body data. is there. Further, a set of height data and amplitude data may be used as body data. A more convenient method is to manually input the necessary numerical values.
[0023]
Next, in step S202, the CG size setting unit 115 calculates the size (magnification) of each CG and stores it in the CG size 123 in the memory 104. It is assumed that the CG size 123 stores the CG size of each CG. In the present embodiment, the CGs to be displayed are the CGs of two players or helmets and gloves of each player, so four values are stored.
[0024]
The CG size setting unit calculates the CG magnification based on the body type data recorded by the body type detection unit 114 described above.
[0025]
For example, the height of each player is divided by the numerical value of the assumed body data 126 at the time of creating the CG model data, thereby obtaining the CG magnification. That is, when the height of the player is half of the assumed height, the magnification is set to 0.5. Strictly speaking, the size of the head and the size of the hand are not simply proportional to the height, so in that case, they are set using an appropriate function.
[0026]
In this embodiment, in order to set the size with higher precision, different functions are used for each of the two types of CGs, that is, the helmet and the glove, and the size for each CG of each player is set from the height and the assumed height of the player. I do.
[0027]
In order to fit the CG more accurately, the body type detection means 114 may detect more detailed body type data. If the CG does not need to be fitted to the body shape, 1.0 may always be stored as the magnification of the CG.
[0028]
By doing so, it is possible to appropriately set the size corresponding to the type of the player and the CG.
[0029]
Next, in step S203, the gaze position / posture detection unit 112 writes the position and posture of the video camera 107 into the gaze position / posture 121 in the memory 104. The line-of-sight position / posture detecting means 112 detects the position (XYZ) and posture (Roll, Pitch, Yaw) of the line of sight by using a conventionally used technology (such as a magnetic sensor).
[0030]
Next, in step S204, the real image photographing means 110 writes the image of the real space into the image memory 121 in the memory 104. The real image capturing means 110 captures an image captured by the video camera 107 and writes the captured image into the image memory 120 in the memory 104.
[0031]
In step S205, the variable N is initialized to 1. The variable N is for counting the number of position and orientation sensors for CG. In the present embodiment, there are four sensors for CG, namely, the sensors 105 (b), 105 (c), 105 (d) and 105 (e) in FIG. Note that the position sensor 105 (b) is also a sensor used to detect the line-of-sight position and orientation of the video camera 107 attached to the HMD. However, since the position sensor 105 (b) superimposes CG on the head of the first player wearing the HMD. This is also used as a CG position detection sensor.
[0032]
In step S206, the CG position / posture detecting means 113 detects the position / posture at which the N-th CG should be displayed, and stores it in the CG position / posture 122 in the memory 104. The CG position / posture detecting means 113 is the same as the above-mentioned gaze position / posture detecting means 112.
[0033]
In step S207, the CG image generation unit 111 selects and draws the CG image corresponding to the Nth sensor from the CG model data 125 in the image memory 121 in the memory 104. That is, since the first (105 (b)) and third (105 (d)) sensors are mounted on the heads of the respective players, the CG of the helmet is changed to the second (105 (c)) and the fourth (105 (c)). Since the sensor of (105 (e)) is attached to each player's hand, the CG of the glove is used. Further, at this time, the CG image generating means draws the CG by enlarging or reducing the CG based on the N-th data of the CG size 123 in the memory 104 (the CG size calculated for the N-th CG). . The CG image generating means 111 writes (overdraws) the CG in the image memory 120 based on the line-of-sight position / posture 122 and the CG position / posture 123. The CG is rendered as if the object placed in the position and orientation indicated by the CG position and orientation 123 was observed from the line of sight position and orientation 121. The image of the real space has already been written in the image memory 120 by the actual image photographing means 110 in step S203, and the CG image is drawn over this.
[0034]
Next, in step S208, the variable N is counted up by one, and in the next step S209, it is determined whether N has exceeded the number of CGs (4 in this embodiment). In this way, the CG is enlarged / reduced for all the CG position sensors based on the CG size (magnification) calculated in advance and drawn.
[0035]
Next, in step S210, the video written in the video memory 120 is displayed on the HMD 106 by the video display means 118. The image display means 118 writes the image memory to the frame memory of the HMD, and performs actual display.
[0036]
In step S211, it is determined whether to end the entire process. If not, the process returns to step S206 and repeats the process. By repeating the processing, a video is sequentially captured from the video camera 107, a CG video is overwritten on the video, and a cycle of displaying the video is continuously performed.
[0037]
This is the end of the description of the overall processing flow.
[0038]
FIG. 3 is a diagram for explaining the effect of the present embodiment. FIG. 3A is a diagram of the player wearing the sensor. FIG. 3 (B) shows how, when CG data is superimposed according to a modeled size without using the present invention, a player having an assumed height and a player having a half height of an assumed height are displayed. Is shown. FIG. 3C shows how the CG is resized and superimposed in the case of the above (B) when the present invention is used.
[0039]
With the above configuration, it is possible to superimpose the CGs, which have been respectively enlarged and reduced so as to fit the body shape of the player, on the real video. Furthermore, since the adjustment can be performed independently for each type of CG, a more accurate mixed reality can be provided.
[0040]
(Other embodiments)
A software program for realizing the functions of the above-described embodiment is provided to an apparatus or a computer in the system connected to the various devices so as to operate the various devices so as to realize the functions of the above-described embodiment. The present invention also includes a system in which a computer (CPU or MPU) of the system or apparatus is operated by supplying the code and operating the various devices according to a stored program.
[0041]
In this case, the program code itself of the software realizes the function of the above-described embodiment, and the program code itself and a unit for supplying the program code to the computer, for example, the program code is stored. The storage medium constitutes the present invention.
[0042]
As a storage medium for storing such a program code, for example, a floppy (R) disk, hard disk, optical disk, magneto-optical disk, CD-ROM, magnetic tape, nonvolatile memory card, ROM or the like can be used.
[0043]
When the computer executes the supplied program code, not only the functions of the above-described embodiments are realized, but also the OS (Operating System) in which the program code runs on the computer, or other application software. It goes without saying that such a program code is also included in the embodiment of the present invention when the functions of the above-described embodiment are realized in cooperation with the above.
[0044]
Further, after the supplied program code is stored in a memory provided in a function expansion board of a computer or a function expansion unit connected to the computer, a CPU or the like provided in the function expansion board or the function storage unit based on the instruction of the program code. It is needless to say that the present invention includes a case in which the functions of the above-described embodiments are implemented by performing part or all of the actual processing.
[0045]
【The invention's effect】
According to the present invention, it is possible to superimpose a computer graphic image having a size corresponding to the player's body type, and to present a natural mixed reality.
[Brief description of the drawings]
FIG. 1 is a system configuration diagram of an embodiment.
FIG. 2 is a flowchart of a process according to the embodiment.
FIG. 3 is an explanatory diagram for explaining effects of the embodiment.

Claims (8)

An information processing method for superimposing a generated computer graphics image on a real image,
Enter the body type information of the player,
Set the size of the computer graphics image according to the body information,
Generating the computer graphic image based on the set size,
An information processing method comprising superimposing the generated computer graphic image on a real image. Inputting an output value of a sensor for detecting viewpoint information of the player,
An information processing method for generating the computer graphic image based on the viewpoint information,
The information processing method according to claim 1, wherein the body type information of the player is obtained using the sensor. An information processing method capable of synthesizing a plurality of types of computer graphics images with the real image,
2. The information processing method according to claim 1, wherein a size is set from the body type information using a condition according to a type of the computer graphics image. Inputting position information detected by a position sensor attached to the player,
An information processing method for synthesizing the generated computer graphics at a position indicated by the position information,
The information processing method according to claim 1, wherein the body type information of the player is input using a position sensor mounted on the player. A program for realizing the information processing method according to claim 1. CG image generating means for generating a computer graphics image;
A real image shooting means for shooting images of the real world,
Video display means for combining the computer graphics image and the captured image and displaying the combined image on a display device,
Body shape detection means for detecting the body shape of the player;
A mixed reality presentation apparatus comprising: a CG size setting unit configured to set a size of a computer graphic image generated by the CG image generation unit in accordance with a player's body data detected by the body type detection unit. The mixed reality presentation apparatus according to claim 6, wherein the body type detection unit uses information obtained from a position sensor mounted on a player's head. 7. The mixed reality presentation device according to claim 6, wherein the body type detecting means uses body width data of the player obtained by using a position sensor worn on a player's hand.

Images