JP2005128877A - Complex sense of reality providing system and method, information processing device and method, and computer program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make an object in a real space visible when the object in the real space is hidden by the existence of a CG object in a virtual space which locates before the object in the real space. <P>SOLUTION: A complex sense of reality providing system is provided with; a head mount display 1 equipped with a video camera 13 which picks up the image of a real world; a PC 2 which generates a computer graphic image, carries out the synthesis of the computer graphic image and image of the real world picked up with the video camera 13, and provides the head mount display 1 with the image thus synthesized. The PC 2 is provided with a CG-transparentizing part 19 which makes the computer graphic image transparent. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a mixed reality presentation system and method, an information processing apparatus and method, and a computer program that synthesize and display a real-world video and a computer graphics image.

  Conventionally, a real-world image and a three-dimensional modeled CG (computer graphics) image are superimposed and displayed, and it is as if an object (virtual object) drawn in CG exists in the real world. Mixed reality technology that can make it look like this is known. This includes a real image photographing means (for example, a video camera) for photographing a real world image, a CG image generating means for creating a CG image as seen from the position where the real world image is photographed, and both. This is an apparatus comprising video display means (for example, an HMD (Head Mount Display) or a monitor) that can be displayed in a synthesized manner.

  Here, even if the position of the line of sight of the real image photographing means changes, the positional relationship between the real world image and the CG image is correct, and the line of sight of the real image photographing means (video camera) is correct. A line-of-sight position / orientation detection unit (for example, a position / orientation sensor) for detecting the position and the line-of-sight direction is provided.

  The CG image generating unit places the CG modeled three-dimensionally in a virtual space having the same scale as the real space, and renders the CG as observed from the line-of-sight position and the line-of-sight direction detected by the line-of-sight position and orientation detection unit. When the CG image generated in this way and the real-world video are superimposed, as a result, even if the real video shooting means observes from any gaze position and gaze direction, it is correctly generated in the real space by CG. An image can be displayed as if an object is placed.

  For example, a video camera is used as the real video shooting means for shooting the real world video, and the video in the line-of-sight direction of the video camera is shot and captured in the memory.

  For example, an HMD (Head Mount Display) is used as a video display device that synthesizes and displays a real-world video and a CG image. By using an HMD instead of a normal monitor and mounting the video camera in the direction of the line of sight of the HMD, it is possible to project an image in the direction that the observer is facing on the HMD and Since drawing of CG can also be performed, an observer's immersion feeling can be improved.

  As the position / orientation detection means, a magnetic position / orientation sensor or the like is used. By attaching this to the video camera (or HMD to which the video camera is attached), the position / orientation value of the line of sight of the video camera is detected. . A magnetic position / orientation sensor detects a relative position / orientation between a magnetism generator (transmitter) and a magnetic sensor (receiver). For example, a product FASTRAK of Polhemus, USA Can be mentioned. This is a device that detects the three-dimensional position (X, Y, Z) and posture (Roll, Pitch, Yaw) of a sensor in real time in a specific region.

  With the above configuration, the observer can observe the world where the real-world video and the CG image are superimposed through the HMD. When the observer looks around, the real image shooting device (video camera) provided in the HMD takes a picture of the real world, and the line-of-sight position and orientation detection means (position and orientation sensor) provided in the HMD is the position of the video camera. The line-of-sight direction is detected, and the CG video generation means generates a CG image viewed from the line-of-sight position / orientation in response to this, and superimposes it on the real-world video for display.

Japanese Patent Laid-Open No. 11-88913

  However, when the CG object in the virtual space exists in front of the object in the real space that the observer wants to operate, the CG object in the virtual space is simply superimposed on the real-world video. As a result, the object in the real space that the observer wants to operate is hidden behind the CG object in the virtual space and cannot be seen.

  The present invention has been made in view of the above points, and when a CG object in a virtual space exists in front of an object in the real space and hides the object in the real space, the real space The purpose is to make the object visible.

The mixed reality presentation system of the present invention is a mixed reality presentation system that synthesizes a real-world video and a computer graphics image and displays them on video display means. In the mixed reality space, an object in the real space is displayed. It is characterized in that it comprises means for making transparent the computer graphic object in the virtual space when the computer graphic object in the virtual space previously exists and the object in the real space is hidden. .
The mixed reality presentation method of the present invention is a mixed reality presentation method that combines a real-world video and a computer graphics image and displays them on a video display means. In the mixed reality space, an object in the real space is displayed. It has a feature in that it has a procedure for making a computer graphics object in the virtual space transparent when an object by computer graphics in the virtual space exists and hides the object in the real space. .
A computer program according to the present invention is a computer program that causes a computer to execute a process of synthesizing a real-world video and a computer graphics image and displaying them on video display means. When an object by computer graphics in the virtual space exists in advance and the object in the real space is hidden, the process of making the object by the computer graphics in the virtual space transparent is executed by a computer Have
The mixed reality presentation system of the present invention generates a head mounted display having a photographing means for photographing a real-world image, a computer graphics image, and the reality obtained by photographing the computer graphics image by the photographing means. And an information processing device provided to the head-mounted display in combination with a video image of the world, wherein the information processing device includes a transparent means for transparentizing at least a part of the computer graphics image. Have. In this case, the head-mounted display includes a position / orientation sensor for detecting the line-of-sight position and direction of the observer wearing the head-mounted display, and the transparency means detects the detection result from the position / orientation sensor. Based on this, the computer graphics image may be made transparent.
The information processing apparatus according to the present invention generates a computer graphics image, synthesizes the computer graphics image with a real-world image captured by an imaging unit included in the head mounted display, and provides the synthesized image to the head mounted display. An information processing apparatus is characterized in that the information processing apparatus includes a transparentizing unit that transparentizes at least a part of the computer graphics image.
The information processing method according to the present invention generates a computer graphics image and combines the computer graphics image with a real-world image captured by an imaging unit included in the head mounted display, and provides the synthesized image to the head mounted display. An information processing method by information processing means, characterized in that the information processing apparatus has a procedure for making at least a part of the computer graphics image transparent.
The computer program of the present invention generates a computer graphics image, and synthesizes the computer graphics image with a real-world image captured by the imaging means included in the head mounted display, and provides it to the head mounted display. Is a computer program for causing a computer to execute transparency processing for making at least a part of the computer graphics image transparent.

  According to the present invention, when a computer graphics object in the virtual space exists in front of an object in the real space and the object in the real space is hidden, the computer graphics object in the virtual space is displayed. Since it can be made transparent, the observer can see the object in the real space existing ahead while confirming the outline of the object by computer graphics in the virtual space.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 shows a schematic configuration of the mixed reality system of the present embodiment. As shown in the figure, an HMD (head mounted display) 1 and a PC (personal computer) 2 are connected via a cable or the like so as to be able to perform data communication.

  The HMD 1 includes a position / orientation sensor 12 for detecting the line-of-sight position and direction of the observer 21 wearing the HMD 1, a video camera 13 for capturing a real-world image, and an image display unit 14.

  The PC 2 includes a position / orientation calculation unit 15, a capture unit 16, an image composition unit 17, a CG creation unit 18, and a CG transparency unit 19. Note that the PC 2 in this embodiment corresponds to the information processing apparatus referred to in the present invention.

  The HMD 1 is a device for the observer 21 to feel mixed reality, and the PC 2 is a device that creates a mixed reality space.

  The position and orientation sensor 12 of the HMD 1 sends the position and orientation data of the observer 21 to the position and orientation calculation unit 15 of the PC 2. The position / orientation calculation unit 15 calculates a position / orientation in the mixed reality space from the position / orientation data received from the HMD 1, and sends the calculated value to the CG creation unit 18.

  Further, the CG transparency unit 19 sends an instruction of a CG object to be made transparent to the CG creation unit 18.

  The CG creation unit 18 creates a CG object in the virtual space from the viewpoint of the observer 21 based on the position and orientation calculated by the position and orientation calculation unit 15. When the CG object is created, the CG object designated by the CG transparency unit 19 is made transparent. The CG in the virtual space created in this way by the CG creation unit 18 is sent to the image composition unit 17.

  The video camera 13 of the HMD 1 captures a real world image viewed from the viewpoint of the observer 21. The video imaged by the video camera 13 is sent to the capture unit 16 of the PC 2. The capture unit 16 captures the real space video and sends it to the image composition unit 17.

  The image composition unit 17 superimposes the CG in the virtual space on the real-world video and sends it to the video display unit 14 of the HMD 1. The video display unit 14 displays a mixed reality space video, that is, a video in which a real-world video and a CG image are combined, and the observer 21 can view the video.

  Hereinafter, a specific usage method will be described with reference to FIGS. As shown in FIGS. 2 and 3, the observer 21 wearing the HMD 1 tries to operate the button 3 placed on the desk 6 in the real space. However, it is assumed that the CG object 4 in the virtual space exists between the observer 21 and the button 3 in the mixed reality space. The CG object 4 in the virtual space is an opaque object, and the observer 21 is disturbed by the CG object 4 in the virtual space and cannot see the button 3 in the real space.

  FIG. 4 shows an image viewed by the observer 21 in the state of FIG. When the CG object 4 in the virtual space is superimposed on the real world video without being opaque, the button 3 in the real space is hidden behind the CG object 4 in the virtual space and cannot be seen as shown in FIG.

  Therefore, as shown in FIG. 5, the CG object 4 in the virtual space is made transparent by the CG transparency unit 19. FIG. 5 shows the same situation except that the CG object 4 in the virtual space is made transparent from the state of FIG. As a result, as shown in FIG. 6, the video seen by the observer 21 can see the button 3 on the real space ahead of the CG object 5 on the transparent virtual space.

(Other embodiments)
The functions of the above-described embodiments can also be realized by a computer executing a computer program. Further, means for supplying a computer program to a computer, for example, a computer-readable recording medium such as a CD-ROM in which such a program is recorded, or a transmission medium such as the Internet for transmitting such a program is also applied as an embodiment of the present invention. be able to. Further, a computer program product such as a computer-readable recording medium in which the above program is recorded can be applied as an embodiment of the present invention. The above computer program, recording medium, transmission medium, and computer program product are included in the scope of the present invention. As the recording medium, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  The above-described embodiments are merely examples of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. That is, the present invention can be implemented in various forms without departing from the technical idea or the main features thereof.

It is a figure which shows schematic structure of the mixed reality system of embodiment. It is a figure which shows the state which the observer who suffered HMD is going to operate the button mounted on the desk in real space. It is a figure which shows the state which the observer who suffered HMD is going to operate the button mounted on the desk in real space. It is a figure for demonstrating the image | video which an observer sees. It is a figure which shows the state which the observer who suffered HMD is going to operate the button currently mounted on the desk in real space. It is a figure for demonstrating the image | video which an observer sees.

Explanation of symbols

1 HMD
2 PC
3 Button in Real Space 4 CG Object in Virtual Space 5 CG Object in Transparent Virtual Space 6 Desk 12 Position / Attitude Sensor 13 Video Camera 14 Video Display Unit 15 Position / Attitude Calculation Unit 16 Capture Unit 17 Image Composition Unit 18 CG creation part 19 CG transparency part 21 Observer

Claims (8)

A mixed reality presentation system that synthesizes a real-world video and a computer graphics image and displays them on a video display means,
In the mixed reality space, when an object by computer graphics in the virtual space exists in front of an object in the real space and hides the object in the real space, the object by the computer graphics in the virtual space is A mixed reality presentation system comprising means for making transparent. A mixed reality presentation method for combining a real-world image and a computer graphics image and displaying the image on a video display means,
In the mixed reality space, when an object by computer graphics in the virtual space exists in front of an object in the real space and hides the object in the real space, the object by the computer graphics in the virtual space is A mixed reality presentation method characterized by having a procedure for transparency. A computer program for causing a computer to execute a process of synthesizing a real-world image and a computer graphics image and displaying the image on a video display means
In the mixed reality space, when an object by computer graphics in the virtual space exists in front of an object in the real space and hides the object in the real space, the object by the computer graphics in the virtual space is A computer program for causing a computer to execute a transparent processing. A head-mounted display with a photographing means for photographing real-world images;
An information processing device that generates a computer graphics image and combines the computer graphics image with a real-world image captured by the imaging unit and provides the synthesized image to the head mounted display;
The mixed information presenting system, wherein the information processing apparatus includes a transparent means for transparentizing at least a part of the computer graphics image.   The head-mounted display includes a position / orientation sensor for detecting the position and direction of an observer who is wearing the head-mounted display, and the transparent means is configured to detect the computer based on a detection result from the position / orientation sensor. The mixed reality presentation system according to claim 4, wherein the graphics image is made transparent. An information processing apparatus that generates a computer graphics image and combines the computer graphics image with a real-world image captured by an imaging unit included in the head-mounted display to provide the head-mounted display.
An information processing apparatus comprising transparency means for making at least a part of the computer graphics image transparent. An information processing method by an information processing unit that generates a computer graphics image and combines the computer graphics image with a real-world image captured by an imaging unit included in the head mounted display and provides the image to the head mounted display. There,
An information processing method, wherein the information processing means includes a procedure for making at least a part of the computer graphics image transparent. A computer program that generates a computer graphics image, and synthesizes the computer graphics image with an image of the real world photographed by photographing means included in the head mounted display, and causes the computer to execute processing to be provided to the head mounted display. Because
A computer program for causing a computer to execute a transparency process for making at least a part of the computer graphics image transparent.

Images