JP2005165776A - Image processing method and image processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To present an explanation regarding a device which an observer carried so as to be easy to see to the observer. <P>SOLUTION: A video in a virtual space seen from the point of view of the observer 100 who wears an HMD 110 on the head is superimposed on a real space and, in order to present it to a display part which the HMD 110 has, a position posture in the world coordinate system of a controller 120 is obtained. When a virtual object which states the explanation regarding the controller 120 is arranged near the position of the controller 120 and at a position nearer to the position of point of view than the former position, a picture of the virtual object is displayed onto the display part. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a technique for superimposing and presenting an image of a virtual space on a real space.

  The mixed reality device provides a user with a mixed reality space image obtained by combining a real space image and a virtual space image generated according to the user's viewpoint position and line-of-sight direction. The mixed reality device can be presented to the observer as if a virtual object is actually present in the real space. The theme park, amusement facility, various exhibitions, exhibitions, museums, aquariums, botanical gardens, It is expected to be used in zoos.

  For example, in Patent Document 1, the position direction of the arm is detected by a position / direction sensor attached to the observer's arm, and the change in the position direction is used as an interface unit.

On the other hand, in a virtual space video device such as a video game device and a virtual reality device, a dedicated controller such as a joystick, a mouse, a keyboard, and the like have been used as an interface with the virtual space. These devices can also be used in the mixed reality apparatus.
JP 2000-353248 A

  However, in a mixed reality apparatus using a head-mound display, it is difficult to operate since a device such as a jay stick cannot be directly seen. In addition, the display area for displaying menus and the like is limited as compared with a normal virtual space video apparatus. When stereoscopic viewing is performed, if a menu is arranged in a three-dimensionally overlapping place with a virtual object and a real object, the stereoscopic effect becomes inconsistent and is difficult to see.

  As described above, stereoscopic viewing is not difficult, does not cause three-dimensional contradiction, minimizes the field of view, and provides easy-to-understand menu display and easy operations. It needs to be realized.

  The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for presenting an explanation related to an apparatus carried by an observer to the observer in an easy-to-see manner.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, in order to superimpose a virtual space image seen from the viewpoint of the observer wearing the head-mounted display device on his / her head on the real space and present it on the display unit of the head-mounted display device. Image processing method,
A position and orientation calculation step for obtaining a position and orientation in the world coordinate system of the indicator based on data obtained from a device that measures the position and orientation of the indicator held by the observer;
When a virtual object in which the description of the pointing tool is described is arranged near the position of the pointing tool and closer to the viewpoint than the position of the pointing tool, an image of the virtual object is displayed on the display unit. An image processing method comprising: a display control step for displaying the image on the display.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, in order to superimpose a virtual space image seen from the viewpoint of the observer wearing the head-mounted display device on his / her head on the real space and present it on the display unit of the head-mounted display device. Image processing method,
A position and orientation calculation step for obtaining a position and orientation in the world coordinate system of the indicator based on data obtained from a device that measures the position and orientation of the indicator held by the observer;
A display control step of displaying on the display unit an image in which the description of the pointing tool is described in the vicinity of the display position of the pointing tool on the display screen of the display unit after the superimposing process. Image processing method.

  In order to achieve the object of the present invention, for example, an image processing method of the present invention comprises the following arrangement.

That is, an information processing method for superimposing a virtual space image updated in response to an instruction from an operating device operated by an experienced person and a real space image,
A real space image acquisition step of acquiring a real space image obtained by photographing the real space;
A first detection result input step of inputting a detection result of a viewpoint position and / or a line-of-sight direction of the photographed real space image;
A virtual space image generation step of generating a virtual space image according to the viewpoint position and / or the line-of-sight direction;
A second detection result input step for inputting a detection result of the position and / or direction of the operating device operated by the experience person;
A virtual screen generating step for generating a virtual screen for assisting the operation according to the position and / or direction of the operating device;
An instruction input step of inputting an instruction of an experienced person using the operation device;
Means for updating the virtual space image in response to an instruction from the experience person;
And a mixed reality space image generation step of generating a mixed reality space image by superimposing the real space image and the virtual space image.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, in order to superimpose a virtual space image seen from the viewpoint of an observer wearing the head-mounted display device on his / her head on the real space and present it on the display unit of the head-mounted display device. Image processing apparatus,
Measuring means for measuring the position and orientation of the pointing tool held by the observer;
Based on the data indicating the position and orientation measured by the measurement means, the position and orientation calculation means for obtaining the position and orientation in the world coordinate system of the pointing tool;
When a virtual object in which the description of the pointing tool is described is arranged near the position of the pointing tool and closer to the viewpoint than the position of the pointing tool, an image of the virtual object is displayed on the display unit. And a display control means for displaying above.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, in order to superimpose a virtual space image seen from the viewpoint of the observer wearing the head-mounted display device on his / her head on the real space and present it on the display unit of the head-mounted display device. Image processing apparatus,
Measuring means for measuring the position and orientation of the pointing tool held by the observer;
Based on data indicating the position and orientation measured by the measurement means, a position and orientation calculation means for obtaining a position and orientation in the world coordinate system of the pointing tool;
Display control means for displaying, on the display unit, an image in which the description of the pointing tool is described in the vicinity of the display position of the pointing tool on the display screen of the display unit after the superimposing process. To do.

  According to the configuration of the present invention, it is possible to present the explanation related to the device carried by the observer more easily to the observer.

  Hereinafter, the present invention will be described in detail according to preferred embodiments with reference to the accompanying drawings.

[First Embodiment]
The image processing apparatus according to the present embodiment is a mixed reality apparatus capable of performing a pseudo-experience that communicates with a fish as a virtual object. This mixed reality device is intended to allow a plurality of observers to experience a submarine walk in an aquarium or to have a simulated experience of communicating with fish.

  FIG. 1 is an external view of a mixed reality apparatus (exhibit apparatus) according to the present embodiment.

  Each observer (100a, 100b, 100c, 100d) has an HMD (head mounted display) (110a, 110b, 110c, 110d) that displays a mixed reality space image on his / her head in order to perform the pseudo-experience. Wearing and standing in the stage. In this device, the number of observers who can experience at the same time is limited to four, but this is not a limitation.

  The HMD 110 that each observer (100a, 100b, 100c, 100d) wears on his / her head (since it is common to each observer, unless otherwise specified, a, b, c, d are not attached below) 2A and 2B, two video cameras 111a and 111b for left and right eyes, two small display devices 112a and 112b for left and right eyes, and a position and orientation sensor The receiver 113 is common to each HMD, and unless otherwise specified, a, b, c, and d are not attached hereinafter.

  The small display devices 112a and 112b are constituted by a liquid crystal display and an optical prism. The video cameras 111a and 111b image the real space according to the orientation of the head of the observer wearing the HMD 100 from the positions of the left eye and right eye of the observer, respectively. The position and orientation sensor receiver 113 measures its own position and orientation by a combination of a position and orientation sensor transmitter, which will be described later, and the main body.

  In the small display devices 112a and 112b, “virtual spaces that can be seen from the left and right eyes of the observer” generated based on the real space images acquired by the video cameras 111a and 111b and the measurement values by the position and orientation sensor receiver 113, respectively. A mixed reality space image synthesized with “the image of the image” is displayed. The mixed reality space image is preferably provided as a stereoscopic image so that the observer can perform a high-quality simulated experience.

  Returning to the description of FIG. Reference numerals 130a and 130b denote position and orientation sensor transmitters, and 131a and 131b denote position and orientation sensor bodies. The position and orientation sensor receivers 113a and 113b and the position and orientation sensor transmitter 130a are connected to the position and orientation sensor main body 131a, and the position and orientation sensor receivers 113c and 113d and the position and orientation sensor transmitter 130b are connected to the position and orientation sensor main body 131b. Has been.

  Magnetism is transmitted from the position and orientation sensor transmitters 130a and 130b, and the position and orientation sensor receivers 113a and 113b detect the magnetism transmitted from the position and orientation sensor transmitter 130a, and the position and orientation sensor receivers 113c and 113d. Detects the magnetism transmitted from the position and orientation sensor transmitter 130b.

  The position / orientation sensor main body 131a receives signals indicating the magnetic reception intensity input from the position / orientation sensor receivers 113a and 113b, and calculates the position / orientation of each receiver. The calculated position / orientation is a position / orientation in a coordinate system (sensor coordinate system) in which the position of the position / orientation sensor main body 131a is the origin and three axes orthogonal to each other are x, y, and z axes, respectively.

  The position / orientation sensor main body 131b receives signals indicating the magnetic reception intensity input from the position / orientation sensor receivers 113c and 113d, and calculates the position and orientation of each receiver. The calculated position / orientation is a position / orientation in a coordinate system (sensor coordinate system) in which the position of the position / orientation sensor main body 131b is the origin and three axes orthogonal to each other are x, y, and z axes, respectively.

  Data indicating the position and orientation of each receiver obtained by the position and orientation sensor main bodies 131a and 131b is transmitted to a computer 151 described later.

  In this embodiment, the position and orientation of a maximum of two observers are measured with one position and orientation sensor transmitter and position and orientation sensor main body, but this is not a limitation, and one position and orientation is measured. The position and orientation of all four people may be measured with the sensor transmitter and the position and orientation sensor main body. That is, each position and orientation sensor receiver receives magnetism generated from one position and orientation sensor transmitter, and each position and orientation sensor receiver outputs a magnetic reception intensity signal to one position and orientation sensor main body. The main body calculates the position and orientation of each position and orientation sensor receiver.

  In this way, the number of position / orientation sensor main bodies and position / orientation sensor transmitters and the assignment of which position / orientation sensor receivers measure the position / orientation of which position / orientation sensor main body and position / orientation sensor main body are particularly limited. In any case, in any case, the data indicating the position and orientation of each receiver obtained by the position and orientation sensor main body can identify which receiver indicates the position and orientation of each receiver. It is transmitted to the computer 151.

  Further, each observer (100a, 100b, 100c, 100d) has a controller 120 in hand (the following description is common to each controller, and unless otherwise specified, a, b, c, d are not attached below). Using this controller 120, the observer can interact with the mixed reality space. FIG. 9 is a view showing an example of the external appearance of the controller 120.

  As shown in the figure, the controller 120 includes a trigger 121, a cross button 122, and a position / orientation sensor receiver 123. The position / orientation sensor receiver 123 is connected to the position / orientation sensor main body 131. In the configuration shown in FIG. 1, the position and orientation sensor transmitters provided in the controllers 120a and 120b are connected to the position and orientation sensor main body 131a, and the position and orientation sensor transmitters provided in the controllers 120c and 120d are the position and orientation sensor main bodies. 131b. The position / orientation sensor receiver 123 outputs a signal indicating the magnetic reception intensity to the position / orientation sensor main body 131, similarly to the position / orientation sensor transmitter mounted on the HMD 110.

  The position and orientation sensor main body 131 receives a signal indicating the magnetic reception intensity input from the position and orientation sensor receiver 123 and calculates the position and orientation of the position and orientation sensor receiver 123. The calculated position / orientation is a position / orientation in a coordinate system (sensor coordinate system) in which the position of the position / orientation sensor body 131 is the origin and the three axes orthogonal to each other are the x, y, and z axes, respectively.

  Data indicating the position and orientation of the position and orientation sensor receiver 123 obtained by the position and orientation sensor main body 131 is transmitted to a computer 151 described later.

  In this embodiment, the position and orientation sensor can use FASTRAK manufactured by US Polhemus, Flock of Birds manufactured by US Ascension Technology, etc., but is not limited to this, and other commercially available ultrasonic sensors An optical position and orientation sensor can also be used.

  141a, 141b, 141c, and 141d are computers that perform a process of generating a mixed reality space image provided to the viewers 100a, 100b, 100c, and 100d, respectively. FIG. 5 is a block diagram showing the basic configuration of the video apparatus according to this embodiment including this computer.

  The video apparatus according to the present embodiment includes a computer 141 (each computer has the same configuration, and unless otherwise specified, a, b, c, and d are omitted), an HMD 110, and a controller 120. .

  Since the video apparatus is for generating a mixed reality space video to be presented to an observer and presenting it to the observer, one image device is required for each observer. Therefore, this video apparatus may be provided according to the number of observers.

  The trigger-type trigger 121 and the cross button 122 are connected to the controller input port 128 of the computer 141, and various instruction signals input using the trigger-type trigger 121 and the cross button 122 are sent to the controller input port 128. Via the CPU 146.

  The computer 141 includes a right-eye video capture board 142b, a right-eye video capture board 142a, a right-eye graphic board 144b having a CG drawing support function, a left-eye graphic board 144a, a network interface 149, an HDD (hard disk drive) 145, a CPU 146, and a memory. 147.

  The right-eye capture board 142b receives a real-space image corresponding to the right eye of the observer, which is captured by the right-eye video camera 111b, and outputs this as data to the subsequent memory 147. The left-eye capture board 142a receives, as a signal, a real-space image corresponding to the left eye of the observer, which is imaged by the left-eye video camera 111a, and outputs this as data to the subsequent memory 147.

  The right-eye graphic board 144b generates a virtual space image that can be seen from the observer's right eye, and superimposes it on the right-eye real space image input to the memory 147 via the right-eye capture board 142b. To generate a mixed reality space image and output it to the right-eye display device 112b. On the other hand, the left-eye graphic board 144a generates an image of the virtual space that can be seen from the left eye of the observer, and superimposes the image on the left-eye real space image input to the memory 147 via the left-eye capture board 142a. To generate a mixed reality space image and output it to the left-eye display device 112a.

  Accordingly, the right and left eye positions are displayed to the observer who wears the HMD 110 on his / her head so that the right eye display apparatus 112b and the left eye display apparatus 112a are positioned in front of the eyes of the observer's right eye and left eye, respectively. A mixed reality space image corresponding to the posture can be presented.

  The HDD 145 stores an OS (operating system) and programs, data, and the like for causing the CPU 146 to execute processing described later as what the computer 141 performs, and these are read to the memory 147 as necessary. is there.

  The CPU 146 controls the entire computer 141 and executes processing that will be described later as what the computer 141 performs.

  The memory 147 is an area for temporarily storing programs and data loaded from the HDD 145, and an area for temporarily storing real space images input from the right-eye capture board 142b and the left-eye capture board 142a. The CPU 146 includes an area used for performing various processes. That is, an area for temporary storage of data being processed and temporary storage of data for data transmission / reception with the outside is provided.

  The network interface 149 is connected to the network 160, and data with a computer 151 and other computers (computers 141b, 141c, and 141d in the case of itself 141a) to be described later via the network interface 149 and the network 160. Can send and receive. For example, Ethernet (R) is applicable to the network interface 149.

  Here, in order for the computer 141 to generate a mixed reality space image and to provide it to the observer via the HMD 110, a mixed reality image generation program to be executed by the CPU 146 is loaded from the HDD 145 into the memory 147. .

  The mixed reality video generation program generates a video of a virtual space that can be viewed from the right eye and left eye of the observer based on mixed reality space information notified from the computer 151 described later, and captures the right eye capture board from the video cameras 111b and 111a, respectively. 142b, a composite reality space image for the right eye and the left eye is generated by synthesizing it on the image of the real space input via the capture board 142a for the left eye, and the generated mixed reality space image for the right eye and the left eye is generated. Are to be executed by the CPU 146 to output to the right-eye display device 112b and the left-eye display device 112a via the right-eye graphic board 144b and the left-eye graphic board 144a, respectively.

  By causing the CPU 146 to execute the above processing, an observer who wears the HMD 110 on the head has a virtual aquatic life as if it were a real object in the stage device. Appears to be floating.

  FIG. 6 is a block diagram showing the basic configuration of the video apparatus according to this embodiment including the computer 151 and the relationship with peripheral devices. The computer 151 obtains data indicating the position and orientation of each position and orientation sensor receiver output from the position and orientation sensor body, mixed reality space management, notification of mixed reality space information, and generation of mixed reality space video at a fixed viewpoint This is for performing environmental music reproduction, sound effect reproduction, and the like.

  The computer 151 includes a capture board 152, a graphic board 153, a serial interface 154, a sound board 155, an HDD 156, a CPU 157, and a memory 158 network interface 159.

  An objective viewpoint video camera 184 is connected to the capture board 152. The objective viewpoint video camera 184 is fixed at one point in the real space (or its position and orientation can be changed as appropriate), and captures an image of the real space that can be seen according to the position and orientation at the one point. is there. The captured real space image is output to the memory 158 via the capture board 152. Note that data indicating the position and orientation is always stored in the memory 158 regardless of whether the position and orientation of the objective viewpoint video camera 184 is fixed or changed as appropriate.

  A display device 180 is connected to the graphic board 153. The display device 180 positions the objective viewpoint video camera 184 with respect to an observer who is not wearing the HMD on the head (an observer other than the observers 100a, 100b, 100c, and 100d in FIG. 1). It provides images of mixed reality space that can be seen according to posture. Therefore, the graphic board 153 performs the same processing as the graphic boards 144b and 144a in the computer 141, and generates a composite image of the real space video and the virtual space video that can be seen according to the position and orientation of the objective viewpoint video camera 184. This is output to the display device 180.

  Position / orientation sensor main bodies 131a and 131b (a and b are omitted in the figure) are connected to the serial interface 154, and the sensor coordinates of each position / orientation sensor receiver output from the position / orientation sensor main body 131 are connected. Data indicating the position and orientation in the system is received and output to the memory 158.

  A speaker 182 is connected to the sound board 155, generates a sound signal based on sound data to be described later, and D / A converts it to output it to the speaker 182.

  The HDD 156 stores an OS and programs and data for causing the CPU 157 to execute processing described later as what the computer 151 performs, and these are read into the memory 158 as necessary.

  The CPU 157 controls the entire computer 151 and executes processing described later as what the computer 151 performs.

  The memory 158 is an area for temporarily storing programs and data loaded from the HDD 156, an area for temporarily storing real space video input from the capture board 152, and the CPU 157 performs various processes. An area used for this purpose is provided. That is, an area for temporary storage of data being processed and temporary storage of data for data transmission / reception with the outside is provided.

  The network interface 159 is connected to the network 160, and data transmission / reception with the above-described computer 141 (141a, 141b, 141c, 141d) can be performed via the network interface 159 and the network 160. For example, Ethernet (R) is applicable to the network interface 149.

  Here, a program to be executed by the CPU 157 is loaded from the HDD 156 into the memory 158. That is, a mixed reality video generation program for generating a mixed reality space image at a fixed viewpoint, position and orientation acquisition of each position and orientation sensor receiver, mixed reality space management, mixed reality space notification for performing mixed reality space information notification It is a sound program for space management program, environmental music playback, and sound effect playback.

  The mixed reality video generation program generates a video of a virtual space according to mixed reality space information, which will be described later, which is visible according to the position and orientation of the objective viewpoint video camera 184 held by the computer 151, and captures from the objective viewpoint video camera 184. In order to cause the CPU 156 to execute a process of generating a mixed reality space image by synthesizing it on the image of the real space input via 152 and outputting the generated mixed reality space image to the display device 180 via the graphic board 153. belongs to.

  By causing the CPU 157 to execute the above processing, aquatic organisms that are virtual objects appear as if they are real objects in the stage device that can be seen from the position and orientation of the objective viewpoint video camera 184 for an observer who is not wearing the HMD on the head. It appears to be floating in the stage as an object.

  The mixed reality space management program is for causing the CPU 157 to execute the following processes (1) to (4).

  (1) The time set in the virtual space is advanced in accordance with a preset scenario, and the position and orientation and state of an arbitrary object in the virtual space are updated.

  (2) The position / orientation of the position / orientation sensor receivers 113 and 123 is acquired from the position / orientation sensor main body 131, and based on this, the position / orientation of the observer's right eye and left eye in the world coordinate system (first Position and orientation), and the position and orientation (second position and orientation) of the controller 120 in the world coordinate system. If the position of the position / orientation sensor main body 131 in the world coordinate system is known, the position / orientation in the sensor coordinate system can be converted into a position / orientation in the world coordinate system. It is assumed that the position of the position / orientation sensor body 131 in the world coordinate system is loaded in the memory 158 as data.

  (3) Based on the relationship between the first position and orientation determined in (2) and the second position and orientation, processing of the line-of-sight relationship between observers, update processing of the mixed reality space state, and the like are executed.

  (4) Then, a part of the information about the mixed reality space (mixed reality space information) obtained by the above processing is output to each computer 141 via the network 160. Strictly speaking, the mixed reality video generation program running on the computer 141 is notified.

  In this embodiment, it is assumed that the position and orientation data obtained from the position and orientation sensor main body 131 is the HMD 110a, 110b, 110c, and 110d that indicates the position and orientation of the HMD by a known technique. . For example, it is assumed that the position / orientation sensor main body 131 transmits each position / orientation data transmitted to the computer 151 with a unique ID added to each HMD.

  As described above, a technique for processing information transmitted from a plurality of computers and transmitting the processed information to a transmission source computer is well known, and a description thereof will be omitted.

  In addition, a part of the mixed reality space information is notified to the mixed reality video generation program and the sound program being executed by itself. The sound program reproduces sound (music and sound effects) based on sound data (assumed to be read from the HDD 156 to the memory 158 in advance) according to the mixed reality space information notified from the mixed reality space management program. To do. The reproduced music and sound effects are output from the speaker 182 at an appropriate volume level that can be recognized by all observers in the stage apparatus.

  This process is actually executed by the CPU 157 executing a sound program.

  The sound effect is used to inform the observer of an event in the mixed reality space (for example, a collision between virtual objects or a collision between the real object and the virtual object) or a mixed reality space state for a user interface.

  FIG. 3 is a diagram showing an example of the appearance of the stage apparatus when there is no observer. FIG. 4 is a diagram showing an example of the appearance of the stage apparatus when there is an observer.

  On the stage equipment, the state of the seabed is expressed by tools, lighting, and the floor. The large tools 102, 103a, and 103b are modeling objects that resemble a reef. The large tools 104a and 104b are stone pillars that make you imagine a ruin that sinks to the seabed. The large tool 106 is a wall or screen-like tool depicting a distant view on the sea floor, and is painted in an underwater environmental color to make it feel as if the sea floor continues beyond the size of the stage device. The large tool 108 is a floor imitating the seabed. Reference numerals 101a, 101b, 101c, and 101d denote areas that can be taken by the initial position of the observer standing when the observer attaches / detaches the HMD 110, and reference numerals 105a, 105b, 105c, and 105d denote arrows indicating the initial direction of the observer. Although shown in FIG. 3 for explanation, the actual stage apparatus does not have lines or colors indicating this area.

  When the observer performs a pseudo-experience, the viewer first enters the stage apparatus, and the observer initial position area 101 (the following description is common to each of the areas 101a, 101b, 101c, and 101d. , A, b, c, and d are omitted), and the HMD 110 is mounted on its own head and the controller 120 is held in the hand.

  Here, the HMD 110 and the controller 120 are placed in a depression at the top of the large tool (103a or 103b) on the side of each observer initial position region 101. The large tool 103 is a molding that also serves as an HMD stand. The observer 100 can take out the HMD 110 from the HMD pedestal 103 and attach it to his / her head. Then, the observer 100 can move around the area including the observer initial position area 101 and can turn in an arbitrary direction or take an arbitrary posture. is there. The range of this area depends on the wiring length of the HMD 110 and the measurable range of the position and orientation sensor 131.

  When it is desired to end the pseudo-experience, the observer returns to the observer initial position area 101, removes the HMD 110, and exits the stage apparatus above the HMD table 103. Entry / exit from the stage equipment is performed from the front side of FIG. The front side depends on the installation location of this exhibition device, but in most cases it is connected to a passage or a space where people can accumulate to some extent. In the latter case, it is possible to create a queue for waiting for a simulated experience or create a place where a person who enjoys mixed reality space video viewed from the viewpoint of the video camera 184 displayed on the display 180 stays. This is a desirable form from the viewpoint of having many visitors enjoy the exhibition facility.

  The observer initial position area 101 is an area where the observer 100 attaches and detaches the HMD 110, and the observer initial direction 105 is a recommended line-of-sight direction at which the observer 100 starts pseudo-experience. Since there are various moldings in the stage, few people move around boldly with the unfamiliar HMD cable. Therefore, in most cases, it can be expected that the mixed reality space image is most frequently observed in the observer initial position area 101 in the pseudo-experience while facing the observer initial direction 105.

  Accordingly, the stage apparatus is assembled in consideration of the high quality of the mixed reality space image according to the arbitrary position and orientation in the observer initial position region 101. For example, the positions of the observer initial position area 101 and the background wall 106 are adjusted so that visitors viewing the queue and the exhibition facility cannot be seen in the real space image, or a plurality of reefs 103 are formed by the observer 100. So that you can feel the depth and stereoscopic effect by entering the field of view from different distances, or let other observers 100 enter the field of view and be aware that you are sharing your simulated experience with others. Or by creating a space between other observers and yourself and displaying the aquatic life that is a virtual space object in the space, or when the background wall is displayed on the HMD 110 Select a color that makes you feel as if it is spreading.

  The observer can perform a high-quality simulated experience by viewing the mixed reality space video in the stage apparatus described above.

  Next, processing performed by each of the computers 141 and 151 will be described.

  FIG. 7 is a flowchart of processing performed by the CPU of each computer executing the mixed reality video generation program installed in the computers 141 and 151.

  Note that the mixed reality video generation program is created using a general thread technique with parallel programming, and in parallel with the execution of this program, the mixed reality space state reception thread (step S710) and the video capture thread (step S720). ) Is working.

  The operation of the mixed reality space state reception thread in step S710 is slightly different between the one installed in the computer 141 and the one installed in the computer 151.

  The mixed reality space state reception thread installed in the computer 141 is a mixed reality space state (positions of the right eye and left eye of the observer) notified from the mixed reality space management program operating on the computer 151 via the network 160. Information (mixed reality space information) indicating the posture, the position and posture of the controller, the position and posture of each virtual object in the virtual space), and notifies the mixed reality video generation program operating on the computer 141 To do.

  The mixed reality space state reception thread installed in the computer 151 is a mixed reality space state (position and orientation of the video camera 184, each virtual in the virtual space, which is notified from the mixed reality space management program operating on the computer 151. The position / posture and state of the object) are received and notified to the mixed reality video generation program operating on the computer 151.

  On the other hand, the operation of the video capture thread in step S720 is slightly different between that installed in the computer 141 and that installed in the computer 151.

  The video capture thread installed in the computer 141 notifies the mixed reality video generation program operating on the computer 141 of the real space video received from the HMD 110 via the capture board 142.

  The video capture thread installed in the computer 151 notifies the real space video received from the video camera 184 via the capture board 152 to the mixed reality video generation program operating on the computer 151.

  In step S701, the state of the virtual space is updated based on the mixed reality space state obtained from the mixed reality space state receiving thread. In step S702, a virtual space image corresponding to the updated state is generated. Here, the image of the virtual space includes, for example, an aquatic life image drawn by CG or the like and a command display panel image described later, and a viewpoint (observer's right eye, left eye, or video camera 184). It is the image | video of the virtual space which can be seen according to the position and orientation. The generated virtual space video is temporarily stored in the memories 147 and 158 in the computer.

  In step S703, the video is drawn on the memories 147 and 158 based on the real space video data obtained from the video capture thread. In step S704, the virtual space video generated in step S702 is created on the video drawn in step S703. To generate a mixed reality space image and output it. The output destination is the display device 180 (when executed on the computer 151) or the display device 112 of the HMD 110 (when executed on the computer 141).

  The above processing is performed until an end instruction is input from the computer 141 or the computer 151. It should be noted that the series of processing relating to the generation of the video of the mixed reality space that can be seen according to the position and orientation of the desired viewpoint as described above is performed by a known technique. However, as will be described in detail later, in this embodiment, instead of simply generating a mixed reality space image that can be seen according to the position and orientation of a desired viewpoint, an image of a panel that displays a command related to the controller 120 is displayed in a virtual space. The place to include in the video is different from the conventional one. The generation of the video of this panel will be described later.

  FIG. 8 is a flowchart of processing performed by the CPU 157 executing the mixed reality space management program installed in the computer 151.

  Note that the mixed reality space management program is created using a general thread technique with parallel programming, and the position and orientation acquisition thread (step S840) operates in parallel with the execution of this program.

  The position / orientation acquisition thread in step S840 always communicates with the position / orientation sensor main body 131, data indicating the position / orientation in the sensor coordinate system of the observer's right eye and left eye, and data indicating the position / orientation in the sensor coordinate system of the controller 120. Is getting.

  In step S800, the data obtained from the position / orientation acquisition thread is received, and the position / orientation of the right eye and the left eye of each observer in the world coordinate system and the position / orientation of the controller 120 in the world coordinate system are obtained by a known technique. The position and orientation data is updated to the position and orientation data obtained this time. Thereby, the memory 158 always holds the latest data indicating the position and orientation of the right eye and left eye of each observer in the world coordinate system, and the data indicating the position and orientation of the controller held by each observer in the world coordinate system. become.

  If the position / orientation data obtained by the position / orientation acquisition thread indicates the position / orientation in the world coordinate system, in this step, the position / orientation data simply obtained from the previous position / orientation acquisition thread and held. Is updated to the data obtained from the current position / posture acquisition thread.

  Next, in step S802, the time set in the virtual space is updated, and in step S803, the state of the virtual space corresponding to the updated time is updated. For example, when it is set to perform a preset event at the time after the update, this event is generated. Even if no event occurs, for example, the position and orientation of each virtual object in the virtual space are updated, and the display position and display size of the command display panel, which will be described later, are updated.

  In step S808, the “data indicating the position and orientation of the observer's right eye and left eye in the world coordinate system”, “data indicating the position and orientation of the controller in the world coordinate system” obtained by the above processing, "State" is notified to all the mixed reality video generation programs and sound programs as mixed reality space information.

  The above processing is performed until an end instruction is input from the computer 151.

  FIG. 10 is a diagram illustrating an example of a mixed reality space image obtained by executing the above mixed reality space management program and mixed reality image generation program. This figure is a mixed reality space image that can be seen in accordance with the position and orientation of the video camera 184 or the observer's right eye (or left eye). The virtual object images 200a to 200f of the fish and the real object tool 102 are displayed. It can be viewed through the device 180 or the display device 112 of the HMD 110.

  Next, a program for sound reproduction will be described. FIG. 12 is a flowchart of processing performed by the CPU 157 executing a sound program installed in the computer 151.

  Note that the sound program is created by parallel programming using a general thread technique, and the mixed reality space state reception thread (step S1610) operates in parallel with the execution of this program.

  The mixed reality space state reception thread in step S1610 is the same as that in step S710. That is, the computer 151 receives a mixed reality space state (position and orientation of the video camera 184, position and orientation of each virtual object in the virtual space) notified from the mixed reality space management program operating on the computer 151. It notifies the sound program running above.

In step S1601, referring to information obtained from the mixed reality space state reception thread, a signal based on sound data corresponding to the state of the mixed reality space is generated by the sound board 155, and the signal is output to the speaker 184. Then, a process of reproducing sound such as sound effects is performed. This sound effect is, for example, a sound in which a virtual object in the virtual space moves around or a sound according to sound data corresponding to the event when an event occurs. When reproducing these sounds, sound data corresponding to these events and operations is read out to the memory 158. Besides this, BGM (background music) may be played, and the sound to be played is not limited to this.
The above processing is performed until an end instruction is input from the computer 151.

  It should be noted that sound reproduction is not an indispensable process because it is only for the purpose of increasing the immersion time in the mixed reality space for the observer.

  Next, display on the command display panel will be described. FIG. 11 is a diagram illustrating a display example of the command display panel in the display device 112 of the HMD 110. Reference numeral 1100 denotes a display screen of the HMD display device 112. That is, an image of the mixed reality space is displayed in the display screen 1100.

  Reference numeral 900 denotes an image of the frame portion of the command display panel, and the function of each button of the tenth button 122 in each direction is displayed as button images 901 to 904 inside this frame. That is, pressing the button at the 12 o'clock position of the ten o'clock button 122 functions as “HIT”, pressing the button at 3 o'clock as “RIGHT”, pressing the button at 6 o'clock as “SHOCK”, 9 When the button at the hour position is pressed, the “LEFT” function is executed. In the figure, the button image 901 is in a reversed state when the button at 12 o'clock is pressed (“HIT” is selected). In this state, for example, when the button at the 9 o'clock position of the cross key 112 is pressed, the color of the button image 903 is reversed, and the display shifts to a state in which “LEFT” is selected.

  That is, the command display panel 900 is an image of an explanation window related to the operation of the ten o'clock button 122.

  The display position of the video moves according to the movement of the position of the controller 120 in the display screen 1100. The display position determination process will be described.

  First, a position that is always relatively the same as the position of the controller 120 in the world coordinate system is determined in advance. For example, the position is 5 cm rearward and 5 cm upward (bias) from the tip of the controller 120. This position can be obtained by adding a bias to the obtained position because the position and orientation of the controller 120 in the world coordinate system is obtained by the mixed reality space management program.

  The position obtained by this is arranged near the position of the controller 120 so that the surface of the command display panel 900 can be seen from the observer's viewpoint when the command display panel 900 is arranged as a three-dimensional virtual object in the virtual space. It is an arrangement position when doing. In addition, it is preferable that this bias be a position near the position of the controller 120 in the world coordinate system and closer to the observer's viewpoint than this position. Thus, the command display panel 900 is always arranged in front of the controller 120 (a position closer to the observer's viewpoint position).

  Then, the position obtained by projecting the obtained position on the display screen 1100 (actually a projection plane provided in advance for displaying an object in the three-dimensional virtual space on the display surface of the display device) is obtained, and the projected position is The image of the command display panel 900 is displayed.

  Thus, if the controller 120 is visible on the display screen 1100, an explanation screen (command display panel 900) relating to the operation can be displayed together with the controller 120. This also moves the display position of the command display panel 900 in accordance with the movement of the image of the controller 120 in the display screen 1100, so that the observer can see the command display panel 900 together with the controller 120. You can see the explanation while looking at the controller 120.

  Of course, when the controller 120 is not visible on the display device 112 of the HMD 110, the command display panel 900 is not displayed on the display device 112.

  If the video of the command display panel 900 is displayed in a translucent form, the video of the controller 120 and other virtual objects cannot be completely seen by the display of the command display panel 900. The technique for displaying an image in a translucent manner is a well-known technique and will be described briefly. However, an α value may be set for each pixel constituting the image and the α value may be controlled. In addition to the transparency control, the same effect can be obtained by performing the control again.

  In addition, as described above, the command display panel 900 is always arranged in front of the controller 120 (a position closer to the observer's viewpoint), so that the controller 120 hides the display of the command display panel 900 or displays a command in the back. It does not happen that the front-rear relationship between the command display panel 900 and the controller 120 breaks down in a three-dimensional space by superimposing the panel 900 on the gun display.

  Further, since the command display panel 900 is always displayed in the vicinity of the controller 120, the controller 120 and the command display panel 900 do not cause a large difference with respect to the parallax at the time of performing stereo stereoscopic viewing. It can be carried out.

  The display form of the command display panel 900 is not limited to that shown in FIG.

  In addition, the application target of the technology according to the present embodiment is not just to present the operation method of the controller as a command display panel, but for the observer wearing the HMD to perform some interaction in the mixed reality space. If you want to see the description of the interface (indicator) that you use (it is not limited to being held in your hand, as long as you carry it) via the HMD, use this explanation along with the indicator that you can see through the HMD. It is also applicable when viewing.

  Also, the panel image for showing this explanation is stored in advance in the memory 147 as a two-dimensional image, and a mixed reality space image is generated, and the position of the pointing tool (the controller 120 in this embodiment) in this image is obtained. (The position of the controller 120 in the world coordinate system can be obtained by the above processing, and the position obtained by projecting it on the projection plane may be obtained). In this embodiment, the position may be displayed near the position. When a three-dimensional virtual object is created as described above and placed in the virtual space and displayed as an image on the display screen of the HMD display device, the three-dimensional virtual object is appropriately scaled so as to always have the same size (three-dimensional virtual object Is larger from the viewpoint position, the size of the three-dimensional virtual object is increased. Conversely, when the three-dimensional virtual object is closer to the viewpoint position, the three-dimensional virtual object is increased. May be performed to reduce the size of the object). In the latter case, this three-dimensional virtual object is such that the surface of the panel as the virtual object (the surface on which the description is described) is a surface whose normal vector is the viewing direction (posture) of the right eye and left eye of the observer. Be placed.

  Moreover, you may make it change suitably the text of the description which should be displayed on the image of this panel in real time.

  In the present embodiment, an aquarium is taken as an example of the mixed reality space, but the present invention is not limited to this.

  In this embodiment, the position and orientation of the controller 120 are measured. However, in order to determine the display position of the command display panel, it is sufficient to know the position of the controller 120 in the world coordinate system at least, so that only the position is measured. Anyway.

  The HMD used in this embodiment is a video see-through type, but an optical see-through type may be used. In this case, since the real space video is not shot, the synthesis processing of the real space video and the virtual space video is not performed.

  Further, in the present embodiment, the camera for imaging the real space is provided for the right eye and for the left eye. However, the present invention is not limited to this, and one camera may be used. In this case, the real space image captured by the one camera is presented to both the right eye and the left eye by superimposing the virtual space image. The video of the virtual space to be superimposed is visible from the viewpoint of this one camera.

  In this embodiment, the device that can be used as the head-mounted display device is not limited to the HMD. For example, a camera that captures an image of a real space, a display device, and a position and orientation sensor receiver are combined. It is also possible to use a device that can be mounted on the observer's head.

[Second Embodiment]
In the first embodiment, the computer 151 manages various states related to the mixed reality space, and the management results are sequentially notified to each computer 141. However, the present invention is not limited to such a system configuration. Absent. That is, the computer 141 provided for each observer may manage information necessary for generating a mixed reality space image to be presented to each observer.

[Other Embodiments]
An object of the present invention is to supply a recording medium (or storage medium) that records software program codes for realizing the functions of the above-described embodiments to a system or apparatus, and the computer of the system or apparatus (or CPU or MPU). Needless to say, this can also be achieved by reading and executing the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an operating system (OS) running on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Furthermore, after the program code read from the recording medium is written into a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer, the function is based on the instruction of the program code. It goes without saying that the CPU or the like provided in the expansion card or the function expansion unit performs part or all of the actual processing and the functions of the above-described embodiments are realized by the processing.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

1 is an external view of a mixed reality apparatus (exhibit apparatus) according to a first embodiment of the present invention. It is an external view of HMD110. It is a figure which shows the example of an external appearance of a stage device when there is no observer. It is a figure which shows the external appearance example of a stage device in case an observer exists. 1 is a block diagram showing a basic configuration of a video apparatus according to a first embodiment of the present invention including a computer 141. FIG. It is a block diagram which shows the basic composition of the video device concerning this embodiment containing the computer 151, and the relationship with a peripheral device. It is a flowchart of the process performed when CPU of each computer runs the mixed reality video generation program installed in the computers 141 and 151. 12 is a flowchart of processing performed by a CPU 157 executing a mixed reality space management program installed in a computer 151. It is a figure which shows the example of an external appearance of the controller 120. FIG. It is a figure which shows an example of a mixed reality space image | video obtained by running a mixed reality space management program and a mixed reality image generation program. 6 is a diagram illustrating a display example of a command display panel on the display device 112 of the HMD 110. FIG. 12 is a flowchart of processing performed by a CPU 157 executing a sound program installed in a computer 151.

Claims (13)

An image for superimposing on the real space a video of a virtual space seen from the viewpoint of an observer wearing the head-mounted display device on his / her head and presenting it on the display unit of the head-mounted display device A processing method,
A position and orientation calculation step for obtaining a position and orientation in the world coordinate system of the indicator based on data obtained from a device that measures the position and orientation of the indicator held by the observer;
When a virtual object in which the description of the pointing tool is described is arranged near the position of the pointing tool and closer to the viewpoint than the position of the pointing tool, an image of the virtual object is displayed on the display unit. An image processing method comprising: a display control step for displaying the image on the display. The display control step generates an image of the virtual object to be displayed on the display unit after appropriately changing the size of the virtual object so that the image of the virtual object is displayed with a predetermined size on the display unit. The image processing method according to claim 1, wherein the image is displayed on the display unit. The image processing method according to claim 1, wherein, in the display control step, an image of a virtual object on which an operation method of the pointing tool is described is displayed. An image for superimposing on the real space a video of a virtual space seen from the viewpoint of an observer wearing the head-mounted display device on his / her head and presenting it on the display unit of the head-mounted display device A processing method,
A position and orientation calculation step for obtaining a position and orientation in the world coordinate system of the indicator based on data obtained from a device that measures the position and orientation of the indicator held by the observer;
A display control step of displaying on the display unit an image in which the description of the pointing tool is described in the vicinity of the display position of the pointing tool on the display screen of the display unit after the superimposing process. Image processing method. The image processing method according to claim 4, wherein in the display control step, an image in which an operation method of the pointing tool is described is displayed. The image processing method according to claim 1, wherein the image is displayed in a translucent form. An information processing method for superimposing a virtual space image updated in response to an instruction from an operating device operated by an experienced person and a real space image,
A real space image acquisition step of acquiring a real space image obtained by photographing the real space;
A first detection result input step of inputting a detection result of a viewpoint position and / or a line-of-sight direction of the photographed real space image;
A virtual space image generation step of generating a virtual space image according to the viewpoint position and / or the line-of-sight direction;
A second detection result input step for inputting a detection result of the position and / or direction of the operating device operated by the experience person;
A virtual screen generating step for generating a virtual screen for assisting the operation according to the position and / or direction of the operating device;
An instruction input step of inputting an instruction of an experienced person using the operation device;
Means for updating the virtual space image in response to an instruction from the experience person;
An image processing method comprising: a mixed reality space image generating step of generating a mixed reality space image by superimposing the real space image and the virtual space image. The image processing method according to claim 7, wherein most of a virtual screen for assisting an operation is translucent or transparent. 9. The image processing method according to claim 7, wherein the arrangement of the menu in the virtual screen corresponds to the arrangement of the operation unit of the operation device. A program that causes a computer to execute the image processing method according to claim 1. A computer-readable storage medium storing the program according to claim 10. An image for superimposing on the real space a video of a virtual space seen from the viewpoint of an observer wearing the head-mounted display device on his / her head and presenting it on the display unit of the head-mounted display device A processing device comprising:
Measuring means for measuring the position and orientation of the pointing tool held by the observer;
Based on data indicating the position and orientation measured by the measurement means, a position and orientation calculation means for obtaining a position and orientation in the world coordinate system of the pointing tool;
When a virtual object in which the description of the pointing tool is described is arranged near the position of the pointing tool and closer to the viewpoint than the position of the pointing tool, an image of the virtual object is displayed on the display unit. An image processing apparatus comprising: display control means for displaying the image on the display. An image for superimposing on the real space a video of a virtual space seen from the viewpoint of an observer wearing the head-mounted display device on his / her head and presenting it on the display unit of the head-mounted display device A processing device comprising:
Measuring means for measuring the position and orientation of the pointing tool held by the observer;
Based on data indicating the position and orientation measured by the measurement means, a position and orientation calculation means for obtaining a position and orientation in the world coordinate system of the pointing tool;
Display control means for displaying, on the display unit, an image in which the description of the pointing tool is described in the vicinity of the display position of the pointing tool on the display screen of the display unit after the superimposing process. An image processing apparatus.

Images