JP2006107437A - Information processing method and apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information processing method and apparatus for preventing a user from feeling anxiety caused by the user's feet being invisible because of computer graphics. <P>SOLUTION: When generating a virtual world image and presenting an image generated by combining a real space image with the generated virtual world image to the user, the position/posture of the user is acquired (step S100). When the user is inside a virtual structure, both an object inside the structure and a transparent object are generated (step S120), and the generated objects are combined with the real space image (step S140). By displaying the composite image (step S160), the real image is displayed for the user's feet. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an information processing method and apparatus, and more particularly, to an information processing apparatus and method for presenting a user with an image obtained by synthesizing a virtual image in the real world.

  A virtual reality (VR) system is a system that makes a virtual space feel as if it is real by presenting a computer with three-dimensional CG (computer graphics) created by a computer. In recent years, a technology for presenting information that does not exist in the real world to the user by synthesizing 3D CG with real-world images has also been developed, such as AR (Augmented Reality) system and patent literature. 1 is called an MR (Mixed Reality) system.

In the MR system, it is possible to see a 3D CG superimposed on a real object, and a system is proposed in which a user can freely manipulate a virtual object by superimposing a virtual object on the real object ( For example, see Patent Document 1).
Japanese Patent Laid-Open No. 11-136706

  Usually, in the MR system, since the CG is displayed in front of the screen rather than the actual image, the user's hand and feet may be hidden by the CG from the user's viewpoint. For example, in an MR system for experiencing the inside of a virtual building, when the user moves into the virtual building, the entire area around the user is covered with a virtual floor or wall. Therefore, in such a system, the user's hand is hidden by the CG, and inconvenience is felt when operating something.

  Moreover, if a user's step is hidden, a user will feel anxiety.

  This invention is made | formed in view of this point, and it aims at suppressing that the user who is an experience person feels anxiety because a user's step is hidden by CG.

  It is another object of the present invention to make it possible to see the real space at the user's feet by utilizing the features of MR.

  In order to achieve the above object, an information processing method of the present invention is an information processing method for generating an image of a virtual world and presenting to the user an image in which the image of the virtual world is synthesized in a real space. An acquisition step of acquiring the position and orientation of the user, and a generation step of generating an image of the virtual world according to the position and orientation of the user from the position and orientation of the user and the CG data of the virtual world, The virtual world image is generated so that a real space image is displayed at the user's foot.

  The information processing apparatus according to the present invention is an information processing apparatus that generates a virtual world image and presents to the user an image obtained by synthesizing the virtual world image in real space, and acquires the position and orientation of the user. Obtaining means; and generating means for generating a virtual world image corresponding to the position and orientation of the user from the user's position and orientation and CG data of the virtual world; and The virtual world image is generated so that a real space image is displayed.

  According to the present invention, since the real world image at the user's foot is displayed, the user's anxiety that the real world image at the user's foot cannot be seen can be eliminated.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

(Embodiment 1)
In the present embodiment, an MR system capable of experiencing the inside of a virtual building will be described.

  First, the entire system configuration will be described.

  FIG. 1 is a diagram showing the system configuration of the present embodiment. In FIG. 1, a system control unit 101 is a unit that controls the entire system, and includes an image input unit 102, an image composition unit 103, an image output unit 104, a camera position and orientation measurement unit 105, and a virtual world generation unit 106. Yes.

  The video see-through head mounted display (HMD) 132 includes a camera 133, an image output unit 134, an image input unit 135, and an image display unit 136. Two cameras 133 are installed so as to correspond to the right eye and the left eye of the user, respectively. The image display unit 126 includes a display unit for the right eye and the left eye of the user.

  Next, the data flow of the system configured as described above will be described.

  The camera 133 of the HMD 132 mounted on the user's head captures each real space that can be seen from the user's right eye and left eye as an image. The image output unit 134 transmits the real space image captured by the camera 133 to the image input unit 102 of the system control unit 101.

  The camera position / orientation measurement unit 105 uses, for example, a magnetic position / orientation sensor (not shown) or estimates a camera position / orientation based on an input image from the camera 133. The posture (that is, the posture of the user (gaze direction)) is measured. The virtual world generation unit 106 generates a three-dimensional CG image viewed from the position and orientation of the camera 133 based on the position and orientation information measured by the camera position and orientation measurement unit 105 and a scene graph held in advance.

  Here, the scene graph indicates the structure of the virtual space, and defines, for example, the positional relationship between CG objects, geometric information, and the like. In the present embodiment, a transparent floor object is described in order to display a real space image at the foot in addition to the object defining the virtual world experienced by the user.

  The image composition unit 103 generates a mixed reality space image by combining the real space image received by the image input unit 102 and the virtual space image (three-dimensional CG image) generated by the virtual world generation unit 106, The image is transmitted to the image output unit 104. The image output unit 104 transmits the mixed reality space image synthesized by the image synthesis unit 103 to the image input unit 135 of the HMD 132. The image input unit 135 receives the mixed reality space image transmitted from the image output unit 104. The image display unit 136 displays the mixed reality space image received by the image input unit 105 on the right-eye display unit and the left-eye display unit, respectively. The mixed reality space image can be observed by the user.

  According to such a system, a mixed reality space image corresponding to the position and orientation of the user wearing the HMD can be displayed. Therefore, the user can freely experience the MR space.

  FIG. 2 shows a tree structure of a scene graph used in this embodiment.

  Since this embodiment is an MR system capable of experiencing a virtual building, it is an object for displaying a real space image through a virtual world scene 202 showing objects related to the virtual building and the CG floor. It has a permeable bed 201.

  The virtual world scene 202 includes objects 203 to 205 related to the interior of the virtual building such as a floor, a wall, and a roof, and other objects 206 indicating objects related to the appearance and the like. Therefore, when the user enters the virtual building indoors, not only the wall and roof, but also the floor CG exists at the user's feet.

  The object of the transparent floor 201 is an object whose attribute is transparent, and exists on a path searched before the scene 202 of the virtual world. The size of the plane of the object is set so that the virtual world image is transmitted and the real world is displayed at the user's feet. The height of the plane of the object is set to the same thickness as the floor of the scene or a thickness that is somewhat thick.

  For example, when the thickness of the object on the floor 203 is 10 mm and a real image is to be displayed for a circular area having a radius of 1 m, a cylinder having a height of 12 mm and a radius of 1 m is set as the object of the transparent floor 201.

  By having such a scene graph, the transparent floor is preferentially drawn over the floor 203. Therefore, the image composition unit 103 composes the real image and the transparent image, and as a result, the real image is displayed in the area of the transparent floor 201.

  The transparent object follows the horizontal movement of the camera 133 (that is, the movement of the user). Based on the position information output from the camera position / orientation measurement unit 105, the horizontal position of the transparent object is set, and the height (vertical position) is set to the same level as the floor of the virtual world. In this way, it is possible to follow the horizontal movement of the camera 133 by the horizontal position on the same plane as the floor of the virtual world, and the transparent object is always arranged in the vertical lower part of the user. The real space can be transmitted at the foot. When the height of the floor of the virtual world changes, the height of the transparent object also changes. Thus, the virtual floor surface is always transmitted even in an application that changes the height of the floor surface.

  Since the transparent object has only the same thickness as the floor surface located on the virtual floor surface, the transparent object in the vertically upward direction of the transparent object is not transmitted and is not invisible.

  Note that some graphic libraries automatically change the order of transparent objects and try to display objects ahead of the transparent objects. In the case of such a graphic library, the mode for combining and displaying the image as it is without changing the order is selected.

  Here, a space for experiencing the MR system of the present embodiment will be described. FIG. 3 is a diagram showing a space where the MR system of the present embodiment can be experienced.

  The space in FIG. 3 is surrounded by a floor, wall, and ceiling in the real world, and a virtual building is displayed in an area 301. When the user is located outside the area 301 (for example, the position indicated by 302), the appearance of the virtual building can be seen. When the user is located inside the area 301 (for example, the position indicated by 303), the virtual building is viewed. You can see the inside of the building.

  Next, the processing of this embodiment will be described according to the flowchart shown in FIG.

  First, in step S100, the camera position and orientation measurement unit 105 measures the position and orientation of the camera 133 (that is, the position and orientation of the user).

  Next, in step S <b> 110, the virtual world generation unit 106 determines whether or not it is in a virtual building based on the measured position and orientation. When it is determined that the object is inside the building, the virtual world generation unit 106 generates a virtual space image based on the transparent object and the object inside the building (step S120). If it is determined that the object is not a building, the virtual world generation unit 106 generates a virtual space image based on an object outside the building (step S130).

  Next, the image composition unit 103 synthesizes the virtual space image generated in step S120 or step S130 and the real space image received by the image input unit 102 (step S140). The image output unit 104 outputs the synthesized image to the HMD 132 (step S150). Then, in the HMD 132, images are displayed on the display units for the right eye and the left eye of the image display unit 136 (step S160).

  Here, the display result and the effect in this Embodiment are demonstrated using FIG. 3, FIG. 5-FIG.

  First, a conventional MR system, that is, a case where a transparent object is not arranged will be described.

  Reference numeral 301 in FIG. 3 represents a floor area where a virtual building in the real world is displayed. FIG. 5 shows a state where a virtual world floor is superimposed on a real world floor area 301 and a user stands there. When the user looks vertically downward through the HMD from this state, conventionally, only the CG on the floor can be seen as shown in FIG. This is because the CG on the floor covers and hides the image in the real space. As described above, if the feet are hidden by the CG, the experience person feels anxiety.

  Next, an MR system according to the present embodiment, that is, a case where a transparent object is arranged will be described. In the present embodiment, the transparent object is arranged on the same plane as the floor of the virtual world. Therefore, a cylindrical transparent object is placed in the vertically lower part of the user, and the real world image appears to be transparent. FIG. 7 shows a state in which the virtual world floor and the transparent object 501 are superimposed on the real world floor, and the user stands there. When the user looks vertically downward through the HMD in this state, the real space including the user's feet is seen through only the shape of the transparent object as shown in FIG. Therefore, it can be suppressed that the experience person feels anxiety because his / her feet are hidden by CG.

  In addition, since the hand can be confirmed in the real-world image area, the operation at hand can be performed while confirming the real-world image. Therefore, compared with the case where the hand is hidden by CG, the operation at hand can be easily performed.

  Here, the foot refers to a predetermined range centered on the user's position as described above. As described below, the foot is a predetermined range in the user's traveling direction or a predetermined distance from the user's position as described below. The predetermined range is shown.

(Other Embodiments—Modification of Transparent Object)
In the above embodiment, a cylinder is used as the transparent object, but other shapes such as a rectangular parallelepiped may be used.

  The transparent object may be deformed according to the moving speed of the user. For example, an elliptical cylinder as shown in FIG. 9 is used as the shape of the transparent object (the elliptical cylinder is seen from above). The direction of the major axis of the elliptical cylinder and the direction of travel of the user are matched (the direction of the arrow shown in the figure matches the direction of travel of the user) so that it becomes an indicator of the direction of travel of the user. The major axis / minor axis is changed in proportion to the moving speed, and is used as an index for transmitting the current moving speed to the user.

  Further, the direction of the major axis of the elliptic cylinder and the direction of the user's line of sight may be matched (the direction of the arrow shown in the figure matches the direction of the user's line of sight).

  Here, the broken-line circle in the figure indicates the position of the user. In this way, it may be shifted from the center position of the elliptical column in the traveling direction or the line-of-sight direction.

  Furthermore, a transparent object having the shape shown in FIGS. 10 and 11 may be used in addition to the cylindrical and elliptical cylinders (both figures show a state viewed from above).

  The shape in FIG. 10 is a figure having a shape in which the traveling direction (or line-of-sight direction) is widened when viewed from the user's position. By using such a shape, it is possible to confirm the foot securely.

  The shape in FIG. 11 is a donut-shaped transparent object, and a virtual floor is drawn in the user's position itself, but the real-world floor is drawn in the surrounding donut-shaped range. By making the transparent object into a donut shape, the user's position image can be confirmed and the user's anxiety can be resolved.

  Further, an MR system for experiencing another virtual world may be used instead of the MR system capable of experiencing the inside of a virtual building. It does not matter as long as CG is superimposed on the feet of the experience person.

  Further, the transmission floor may be arranged on substantially the same plane as the floor of the virtual world. In other words, it may be set dynamically based on the position and orientation information of the camera and the floor arrangement information of the virtual world. For example, the arrangement position of the transparent floor may be set to a position closer to the viewpoint position than the floor of the virtual world.

  Further, by controlling the α blending at the boundary between the transparent objects, a process for blurring the boundary between the transparent object and the floor object may be added.

  Further, the functions of the above-described embodiment (for example, the image composition unit 103 and the computer) are connected to an apparatus or a computer in the system connected to the various devices so as to operate the various devices so as to realize the functions of the above-described embodiments. Implemented by supplying software program code for realizing the functions of the virtual world generation unit 106 and operating the various devices according to the stored program in the system or apparatus computer (CPU or MPU) Are also included in the scope of the present invention.

  In this case, the program code of the software itself realizes the functions of the above-described embodiments, and the program code itself and means for supplying the program code to the computer, for example, a memory storing the program code The medium constitutes the present invention.

  As a storage medium for storing the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

  Further, by executing the program code supplied by the computer, not only the functions of the above-described embodiments are realized, but also the OS (operating system) in which the program code is running on the computer, or other application software, etc. Needless to say, the program code is also included in the embodiment of the present invention even when the functions of the above-described embodiments are realized in cooperation with the embodiment.

  Further, the supplied program code is stored in the memory provided in the function expansion board of the computer or the function expansion unit connected to the computer, and then the CPU provided in the function expansion board or function storage unit based on the instruction of the program code However, it is needless to say that the present invention also includes a case where the function of the above-described embodiment is realized by performing part or all of the actual processing.

It is a figure showing composition of a system of an embodiment. It is a figure showing the scene graph of the virtual world in embodiment. It is a figure showing the space which can experience MR system in an embodiment. It is a flowchart which shows the processing operation of embodiment. It is a figure showing a mode that the user is standing on mixed reality space. It is a figure when the user on mixed reality space looks at the perpendicular lower part. It is a figure showing a mode that the user has stood on the mixed reality space which has arrange | positioned the transparent object. It is the figure on the mixed reality space which has arrange | positioned the transparent object which looked at the vertically downward part. It is a figure which shows the other shape of a transparent object. It is a figure which shows the other shape of a transparent object. It is a figure which shows the other shape of a transparent object.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 System control part 102 Image input part 103 Image composition part 104 Image output part 105 Camera position and orientation measurement part 106 Virtual world generation part 132 HMD
133 Camera 134 Image output unit 135 Image input unit 136 Image display unit

Claims (10)

An information processing method for generating an image of a virtual world and presenting an image obtained by synthesizing the image of the virtual world in a real space to a user,
An acquisition step of acquiring the position and orientation of the user;
Generating a virtual world image corresponding to the user's position and orientation from the user's position and orientation and virtual world CG data;
The information processing method characterized in that the generation step generates an image of the virtual world so that a real space image is displayed at a step of the user. The CG data of the virtual world includes a transparent object for drawing at the user's feet,
A setting step of setting the position of the transparent object according to the position and orientation;
The information processing method according to claim 1, wherein a real space image is displayed when the transparent object set in the setting step makes the image of the virtual world transparent. The virtual world is inside a virtual building;
The virtual world CG data includes a floor object;
The information processing method according to claim 2, wherein the setting step sets the position and orientation of the transparent object so as to be positioned substantially on the same plane as the floor object.   The information processing method according to claim 2, wherein the setting step sets a vertical position of the transparent object in accordance with a vertical position of the floor object.   The information processing method according to claim 2, wherein the transparent object is always arranged in a vertically downward direction of the user by horizontally moving the transparent object in accordance with the movement of the user in the horizontal direction.   The information processing method according to claim 2, wherein the size of the transparent object changes according to movement of a user's position.   The information processing method according to claim 2, wherein the size of the transparent object is different between a front position and a rear position of the user position according to the position of the user. A determination step of determining whether the user's position is within a predetermined area;
The generating step generates an image of the virtual world so that a real space image is displayed on a user's foot when the position of the user is within a predetermined area in the determining step. Item 6. The information processing method according to Item 1.   The program for implement | achieving the information processing method of any one of Claim 1 thru | or 8 with a computer. An information processing apparatus that generates an image of a virtual world and presents an image obtained by combining the virtual world image in a real space to a user,
Acquisition means for acquiring the position and orientation of the user;
Generating means for generating a virtual world image according to the position and orientation of the user from the user's position and orientation and virtual world CG data;
The information processing apparatus is characterized in that the generation means generates the image of the virtual world so that a real space image is displayed at a step of the user.

Images