JP2018067060A - Stereoscopic composite-reality sense providing program for virtual three-dimensional (3d) model - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereoscopic composite-reality sense providing program wich uses a virtual 3D model and is expected to realize force and impact close to a real model for the virtual 3D model.SOLUTION: The stereoscopic composite-reality sense providing program for a virtual 3D model causes a computer to function as: virtual 3D diorama arrangement means for virtually arranging a virtual 3D diorama 70 containing a virtual 3D model 71 in a real space (L); virtual 3D model operating means for operating the virtual 3D model by a controller; HMD position and posture determining means for determining the position and posture of an HMD in the real space; real image capturing means for capturing a real image for a right eye and a real image for a left eye; virtual diorama image creating means for creating a virtual diorama image for the right eye and a virtual diorama image for the left eye; composite image creating means for creating a composite image for the right eye and a composite image for the left eye; stereoscopic video information creation means for creating the stereoscopic video information; and stereoscopic video reproducing means for reproducing the stereoscopic video on a 3D display device.SELECTED DRAWING: Figure 3

Description

  The present invention provides a 3D mixed reality for a virtual 3D model, which allows a 3D video (3D video) to be further enjoyed by a composite video in which a virtual video for a virtual 3D model is superimposed on a real video. It relates to a feeling provision program.

  Conventionally, there are many users whose hobbies are various models such as railroad models. For example, a user who has a hobby of a railroad model obtains a series of rails (tracks) and trains (models) called N gauges and HO gauges, lays the rails in the room, and runs a motor-operated train on the rails. I enjoyed watching it from various angles at various positions.

  For example, a user who enjoys a model of an N-gauge train may create a so-called diorama in a room that includes a background such as a mountainous area or an urban area in order to give a more realistic feeling. In this case, the room is occupied by a diorama. In addition, even if it is a diorama only with rails without creating the background, general users who do not have a model dedicated room need to lay rails in the room to enjoy the model, After enjoying the model, you need to clear the rails. This is a very tedious and time-consuming task and halves the fun.

  In addition, since it is difficult for a user who enjoys a model such as an aircraft or a ship that can be operated wirelessly to operate the model in the room, he / she enjoys taking the model outdoors and operating the model. However, there is not always a place where the model can be operated safely and enjoyed without disturbing the surrounding people, and the range in which the model can be operated is often not sufficiently wide. In addition, when enjoying such a model, there is a possibility that the model may be damaged due to a crash or submergence.

  In recent years, virtual reality (VR) technology has progressed, and virtual reality can be enjoyed from a relatively large system such as a movie to a relatively small system such as a personal computer or a smartphone. Therefore, it is possible to enjoy the virtual model in virtual reality.

  For example, in Patent Document 1, each of a plurality of observers wears a head-mounted display (hereinafter referred to as HMD), and each has an operation input device that operates a virtual object that is visually recognized. An image processing apparatus that more easily associates an HMD and an operation input device is disclosed.

  Patent Document 2 discloses a virtual reality image display device that uses a transmissive display device to display a virtual image displayed on a display in a superposed real space. Note that when the virtual image is superimposed on the transmitted real space, if the user moves the viewpoint direction, the positional relationship between the position of the object in the real space and the position of the object in the virtual image is maintained due to a processing delay or the like. There are cases where it is not possible. If this positional relationship cannot be maintained, an irregular position shift occurs between the position of the object in the real space and the position of the object in the virtual image, and the user feels uncomfortable or sick. Patent Document 2 discloses a virtual reality image display device that greatly reduces the displacement of this position.

Japanese Patent No. 4757115 JP 2007-213407 A

  The invention described in Patent Document 1 can be expected to be able to enjoy the virtual model without trouble both in the room and outdoors by using the virtual model. However, a virtual image including a virtual object is superimposed on a real image, and the virtual object is operated using an operation input device that operates the virtual object. However, a stereoscopic image (three-dimensional image) cannot be provided. Compared to the actual model, there is a possibility that it lacks power and impact and cannot be enjoyed as much as expected.

  The invention described in Patent Document 2 can also be expected to be able to enjoy the virtual model without trouble both in the room and outdoors by using the virtual model. However, although the virtual video is superimposed on the real video that has passed through the transmissive display, as in Patent Document 1, a stereoscopic video (three-dimensional video) cannot be provided. Compared with a real model, Lack of power and impact, you may not be able to enjoy as much as you expected.

  The present invention was devised in view of the above points, and a three-dimensional model for a virtual three-dimensional model that can be expected to be thinned by the force and impact of a real model using a virtual three-dimensional model. It is an object to provide a mixed reality providing program.

  In order to solve the above-mentioned problem, a first invention of the present invention is an HMD that is a non-transmissive head mounted display that is mounted so as to cover both eyes of a user, a controller that can input instructions from the user, and a computer A stereoscopic mixed reality providing program for a virtual three-dimensional model using the HMD, wherein the HMD is a three-dimensional display device that allows a user to visually recognize a stereoscopic image, and a front view of the user's right eye At least one right imaging device that captures the direction, at least one left imaging device that captures the direction of the front line of sight of the user's left eye, and a position at which the computer can detect the position and orientation of the HMD in real space Posture detecting means. A virtual 3 including a virtual three-dimensional model read from a storage device by the computer or received by the computer via a communication line by cooperating the computer or the computer and the HMD. Virtual three-dimensional diorama placement means for virtually placing a three-dimensional diorama at a specified position in a real space, virtual three-dimensional model operation means for operating the virtual three-dimensional model based on an instruction input to the controller HMD position / orientation determination means for determining the position of the HMD and the attitude of the HMD based on a detection signal from the position / orientation detection means, and a real image for the right eye in the real space imaged by the right imaging device A real image capturing means for capturing the real image for the left eye in the real space imaged by the left imaging device; Based on the position and orientation of the HMD, it can be seen from the virtual diorama image for the right eye, which is an image of the virtual three-dimensional diorama that will be seen from the position of the right imaging device of the HMD, and from the position of the left imaging device of the HMD. Virtual diorama image for left eye, which is an image of the virtual three-dimensional diorama, and virtual diorama image creation means for creating the virtual diorama for right eye in the real image for right eye according to the determined position and orientation of the HMD. Composite image creation means for creating a composite image for the right eye in which an image is superimposed, and a composite image for the left eye in which the virtual diorama image for the left eye is superimposed on the real image for the left eye, the composite image for the right eye and the composite image for the left eye 3D video information creating means for creating 3D video information for reproduction on the 3D display device, and using the created 3D video information Te, stereoscopic image reproducing means for reproducing the stereoscopic image in the 3D display device, to function as a three-dimensional composite reality providing program for a virtual 3-dimensional model.

  Next, a second invention of the present invention is a three-dimensional mixed reality providing program for the virtual three-dimensional model according to the first invention, wherein the virtual three-dimensional model is a train, train or subway train. , Construction machines, vehicles or military vehicles, ships or warships, helicopters or aircraft or military aircraft, submarines, robots, rockets, spacecrafts, and the controller indicates the speed of the virtual three-dimensional model. , An instruction to advance or retreat the virtual three-dimensional model, an instruction to turn the virtual three-dimensional model to the right or left, an instruction to raise or lower the virtual three-dimensional model, and a smoke discharge position in the virtual three-dimensional model Instructions for discharging virtual three-dimensional smoke, instructions for discharging virtual three-dimensional steam from the steam discharge position in the virtual three-dimensional model, and exhausting in the virtual three-dimensional model An instruction to discharge virtual three-dimensional exhaust from the discharge position, an instruction to fire virtual three-dimensional shell from the bullet firing position in the virtual three-dimensional model, an instruction to operate in a specified direction of a predetermined part in the virtual three-dimensional model, At least one instruction can be input, and when the computer or the computer and the HMD cooperate to function as the virtual three-dimensional model operation means, according to the instruction input to the controller It is a stereoscopic mixed reality providing program for a virtual three-dimensional model that operates the virtual three-dimensional model.

  Next, a third invention of the present invention is a three-dimensional mixed reality providing program for the virtual three-dimensional model according to the first invention or the second invention, wherein the storage device stores the virtual At least one of motion corresponding sound information that is sound information corresponding to the motion of the three-dimensional model and instruction corresponding sound information that is sound information corresponding to the instruction input to the controller is stored, or the motion corresponding A sound generation program capable of generating at least one of sound information and instruction corresponding sound information is stored, and a sound output that outputs a sound corresponding to a sound signal based on the operation corresponding sound information and the instruction corresponding sound information When the apparatus is used and the computer or the computer and the HMD cooperate to function as the virtual three-dimensional model operation means, the storage device stores the operation-corresponding sound information. Or having the sound generation program capable of generating the motion-corresponding sound information, outputting the sound signal based on the motion-corresponding sound information according to the motion of the virtual three-dimensional model When the sound generation program that has the instruction corresponding sound information or can generate the instruction corresponding sound information is stored in the storage device, the instruction input to the controller A stereoscopic mixed reality providing program for a virtual three-dimensional model that causes the sound output device to output a sound signal based on the corresponding instruction corresponding sound information.

  Next, a fourth invention of the present invention is a three-dimensional mixed reality providing program targeting a virtual three-dimensional model according to any one of the first to third inventions, wherein the virtual 3 As a light source for a three-dimensional diorama, an azimuth angle that is a horizontal angle of a virtual sun that is a virtually set sun in the virtual three-dimensional diorama and an elevation angle that is a vertical angle of the virtual sun or the elevation angle. The virtual time information to be determined and the weather of the virtual diorama can be designated as clear weather or cloudy weather or virtual weather which is at least one weather of rain, wind or snow, and the computer or the computer and the HMD. When functioning as the virtual diorama image creation means in cooperation with each other, the designated azimuth angle and the designated elevation angle or the elevation angle based on the virtual time information The virtual three-dimensional diorama is virtually irradiated with virtual sunlight, which is virtual light when the virtual sun is arranged at a position corresponding to the virtual sun, and the designated virtual weather is virtual around A stereoscopic mixed reality providing program for a virtual three-dimensional model that creates the right-eye virtual diorama image and the left-eye virtual diorama image based on the virtual three-dimensional diorama reproduced in FIG.

  Next, a fifth invention of the present invention is a three-dimensional mixed reality providing program targeting a virtual three-dimensional model according to any one of the first to fourth inventions, wherein one or more The virtual three-dimensional diorama common to the plurality of HMDs using the plurality of HMDs and the controllers corresponding to the respective HMDs for each of the HMDs. A common virtual three-dimensional diorama including the virtual three-dimensional model can be set, and when the computer or the computer and the HMD cooperate to function as the virtual three-dimensional diorama placement means, the common When the virtual three-dimensional diorama is virtually arranged and functions as the virtual three-dimensional model operating means, each of the virtual three-dimensional diorama included in the common virtual three-dimensional diorama When the virtual three-dimensional model is operated corresponding to each of the controllers and functions as the HMD position / orientation determination unit, the position and orientation of each HMD is determined, and when functioning as the actual image capturing unit, When the real image for the right eye and the real image for the left eye are captured for each HMD and function as the virtual diorama image creation means, the common virtual 3D including a plurality of the virtual 3D models that are virtually arranged When the diorama is used to create the virtual diorama image for the right eye and the virtual diorama image for the left eye for each HMD and function as the composite image creation means, the composite image for the right eye and the left eye for each HMD When creating a composite image and functioning as the 3D image information creating means, the 3D image information is created for each HMD. When caused to function as the stereoscopic image reproducing means, on the three-dimensional display device corresponding to the HMD, to reproduce the stereoscopic image corresponding to the HMD, a stereo composite reality providing program for a virtual 3-dimensional model.

  According to the first invention, using an HMD including a right imaging device and a left imaging device, a right-eye composite image in which a right-eye real image and a right-eye virtual diorama image are superimposed, a left-eye real image, and a left-eye image A composite image for the left eye on which a virtual diorama image is superimposed is created separately and a stereoscopic image is reproduced on a three-dimensional display device. The user can enjoy the virtual three-dimensional model from various angles at various positions on the stereoscopic video (three-dimensional video) while instructing the operation of the virtual three-dimensional model from the controller. In addition, since there is no danger of the user actually colliding with the virtual 3D model, the user can enjoy the virtual 3D model by approaching the virtual 3D model. It can be expected that you can enjoy stereoscopic images that exceed the power and impact.

  According to the second aspect of the present invention, the user operates the controller to discharge smoke, fire bullets, etc., which are difficult with an actual model, to discharge virtual 3D smoke, fire virtual 3D shells, etc. Can be done. Therefore, it is possible to give a more realistic feeling and to expect a force and impact exceeding the actual model.

  According to the third invention, it is possible to give a more realistic feeling by outputting the sound of an actual train or an actual aircraft instead of the sound of the actual model. You can expect an impact.

  According to the fourth invention, the user desires even in a scene (environment around the diorama) that is difficult to realize with an actual model, such as an aircraft flying in the sunset sky or a train running in a snowstorm. The scene to be performed can be set freely. Therefore, it is easy to give the user a desired real feeling, and the force and impact exceeding the actual model can be expected.

  According to the fifth invention, a plurality of users can enjoy a battle or the like using each virtual three-dimensional model within the common virtual three-dimensional diorama, so that a plurality of users can sympathize with the enjoyment.

It is a figure explaining the whole structure of the three-dimensional mixed reality provision system which performs the three-dimensional mixed reality provision program for the virtual three-dimensional model of this invention. It is a perspective view explaining the example of the structure of HMD (Head Mounted Display), and the example of the structure of a controller. It is a figure explaining the example at the time of utilizing in the room the 3D mixed reality provision system which performs the 3D mixed reality provision program for the virtual three-dimensional model. It is a flowchart explaining the example of the process sequence of PC (equivalent to a computer) in a stereoscopic mixed reality provision system. It is a flowchart explaining the detail of the process sequence of process SB000 in the flowchart shown in FIG. It is a flowchart explaining the detail of the process sequence of process SB100 in the flowchart shown in FIG. It is a flowchart explaining the detail of the process sequence of process SB200 in the flowchart shown in FIG. It is a flowchart explaining the detail of the process sequence of process SB300 in the flowchart shown in FIG. It is a figure explaining the example which superimposes the virtual diorama image for right eyes on the real image for right eyes, and produces the composite image for right eyes. It is a figure explaining the example of the stereo image reproduced | regenerated by the three-dimensional display apparatus. It is a figure explaining the example which creates a tunnel automatically in the position where a rail (track) crosses the wall of a room. Even if the actual model cannot be enjoyed at various angles from various positions, the virtual three-dimensional diorama can be enjoyed from various positions at various angles. FIG. It is a figure explaining the example which takes out the 3D mixed reality provision system which performs the 3D mixed reality provision program for the virtual 3D model of this invention outdoors, and enjoys a virtual 3D diorama outdoors.

● [Overall Configuration of Stereo Mixed Reality Providing System 1 for Executing a Virtual Mixed Reality Providing Program for Virtual Three-Dimensional Model (FIG. 1), HMD 10 Structure, and Controller 20 Structure (FIG. 2)]
EMBODIMENT OF THE INVENTION Below, the form for implementing this invention is demonstrated using drawing. As shown in FIG. 1, the stereoscopic mixed reality providing program for the virtual three-dimensional model of the present invention (hereinafter referred to as the “stereo mixed reality providing program for the virtual three-dimensional model” is referred to as “stereo mixed reality providing”. 3D mixed reality providing system 1 that executes a program) is described as follows: HMD 10 (Head Mounted Display), controller 20, position and orientation detection unit 30, sound output device 40, PC 50 (corresponding to a computer, for example, a personal computer) Etc. The PC 50 can transmit and receive information (including various data) to and from the diorama providing server 80 via the Internet 60. As indicated by dotted lines, the HMD 10 and the controller 20 can use not only one set but also a plurality of sets.

  The HMD 10 is a non-transmissive head mounted display. As shown in FIG. 2, the main body unit 11, the three-dimensional display device 12, the right imaging device 13 </ b> R, the left imaging device 13 </ b> L, the user identification unit 14, and the viewpoint distance adjustment unit 15. , Position / orientation detecting means 16, belt 17, cables 18A, 18B and the like. The main body 11 is mounted so as to cover both eyes of the user.

  The three-dimensional display device 12 is provided inside the main body 11 and at a position in front of the user wearing the main body 11, and allows the user to visually recognize a stereoscopic image. When the three-dimensional display device is, for example, a liquid crystal shutter type, it has a liquid crystal display device that alternately displays a right eye image and a left eye image, a right eye liquid crystal shutter, and a left eye liquid crystal shutter. Further, when the three-dimensional display device is, for example, a parallax barrier method or a lenticular lens method, a liquid crystal display device and a parallax barrier (a right-eye image and a left-eye image are alternately displayed in the left-right direction) (a right-eye image) And the left-eye image are alternately displayed in the left-right direction) and a liquid crystal display device and a lenticular lens. When the three-dimensional display device is independent of the left and right sides, for example, as shown by a one-dot chain line in FIG. 2, a right-eye liquid crystal display device 12R that displays a right-eye image and a left-eye liquid crystal display device that displays a left-eye image. 12L. The 3D display device 12 may have any structure as long as it allows the user to visually recognize a 3D image, and the PC 50 can display 3D image information (in accordance with the structure of the 3D display device 12). In the case of the liquid crystal shutter type, an image and a shutter control signal are included).

  At least one right imaging device 13R is provided outside the main body 11 and at a position corresponding to the right eye of the user wearing the main body 11 of the HMD 10 (single or plural right imaging devices are provided. ), Image the direction of the front line-of-sight direction 13RV of the user's right eye. Also, the left imaging device 13L is provided at least one outside the main body 11 and at a position corresponding to the left eye of the user wearing the main body 11 (single or multiple left imaging devices are provided. ), Image the direction of the front line-of-sight direction 13LV of the user's left eye. It is more preferable that the interval between the right imaging device 13R and the left imaging device 13L is a configuration that can be adjusted according to the interval between the right eye and the left eye of the user.

  The user identification means 14 is an apparatus for setting identification information for identifying a user when a plurality of users use a plurality of HMDs 10. FIG. 2 shows an example in which the user identification means 14 is a rotary dial type and any one of the numbers 1 to 4 can be set. However, the configuration is not limited to the rotary dial type and may be set by a switch or the like. Also, the setting content is not limited to 1 to 4, and may be A to D or the like. For example, when two users use the HMD 10 respectively, the first user sets the user identification means 14 of the HMD 10 to be worn to “1”, and the second user sets the user identification means 14 of the HMD 10 to be worn “ 2 ”.

  The viewpoint distance adjusting unit 15 is an apparatus for adjusting the distance from the right eye and the left eye of the user wearing the main body 11 of the HMD 10 to the three-dimensional display device 12. Although FIG. 2 shows an example in which the viewpoint distance adjusting means 15 is a rotary dial type, the viewpoint distance adjusting unit 15 is not limited to the rotary dial type and may be configured to be adjustable with a slide lever or the like. When the 3D display device 12 is a parallax barrier method or a lenticular lens method, the distance from the user's right eye and left eye to the 3D display device 12 is finely adjusted so that the user can appropriately view the stereoscopic image. If possible, it is more preferable. In the case where the right-eye liquid crystal display device 12R and the left-eye liquid crystal display device 12L are left and right independent, the distance from the user's right eye and left eye to the three-dimensional display device, and the right-eye liquid crystal display device 12R and the left-eye liquid crystal display device. It is more preferable that the left and right spacing with the display device 12L can be finely adjusted, and a shielding plate is provided so that the left eye liquid crystal display device cannot be seen from the right eye (or the right eye liquid crystal display device cannot be seen from the left eye). It is more preferable.

  The position and orientation detection means 16 is a sensor for detecting the position and orientation of the HMD 10 in the real space, and a plurality of position and orientation detection means 16 are provided in the HMD 10. For example, the position / orientation detection means 16 is a magnetic sensor, an acceleration sensor, or a tilt sensor. For example, the magnetic sensor detects magnetism from the position / orientation detection unit 30 which is a (reference) magnetism generator, and outputs a detection signal to the position / orientation detection unit 30. Further, for example, the tilt sensor outputs a detection signal corresponding to each tilt angle with respect to the three orthogonal axes (X axis, Y axis, Z axis) directions in the HMD 10 to the position and orientation detection unit 30. Further, for example, the acceleration sensor outputs a detection signal corresponding to the acceleration in each of the three orthogonal axes (X axis, Y axis, Z axis) in the HMD 10 to the position and orientation detection unit 30.

  The position / orientation detection unit 30 is based on the detection signal from the position / orientation detection means 16 and the position (position of the orthogonal three axes in the X-axis, Y-axis, and Z-axis directions) and attitude (the rotation angle around the X-axis, This is a device for detecting a rotation angle around the Y axis and a rotation angle around the Z axis. In this embodiment, an example in which the position / orientation detection unit 30 is provided separately from the PC 50 will be described. However, the PC 50 or the HMD 10 may be provided with the function of the position / orientation detection unit 30. The position / orientation detection unit 30 is based on, for example, the detection signal from the magnetic sensor (position / orientation detection means) and the (reference) magnetism output by the position / orientation detection unit 30. X-axis, Y-axis, and Z-axis positions) and postures (a rotation angle around the X axis, a rotation angle around the Y axis, and a rotation angle around the Z axis) can be detected. Further, the position / orientation detection unit 30 is based on the detection signal from, for example, an inclination sensor (position / orientation detection means) or a gyro sensor (position / orientation detection means). Rotation angle, rotation angle around the Z axis) can be detected. Further, the position / orientation detection unit 30 is based on detection signals from, for example, an acceleration sensor (position / orientation detection means) and a gyro sensor (position / orientation detection means), and the three axes (X axis, Y axis, and Z axis) of the HMD 10. The amount of motion corresponding to each, that is, the moving direction and the moving distance can be detected. By integrating the movement distance, the position of the HMD 10 with respect to the initial position (X-axis direction position, Y-axis direction position, Z-axis direction position) can be detected. Then, the position / orientation detection unit 30 outputs position / orientation information including the detected position and orientation of the HMD 10 to the PC 50. The position / orientation detection unit 16 and the position / orientation detection unit 30 are not limited to the above example as long as they can detect the position and orientation of the HMD 10 in the real space.

  The belt 17 is attached to the main body 11 and is configured to be adjustable in length, and holds the main body 11 at a position covering both eyes of the user.

  The cable 18A is connected to the PC 50. The PC 50 captures image information from each of the right imaging device 13R and the left imaging device 13L via the cable 18A. Further, the PC 50 outputs stereoscopic video information to the three-dimensional display device 12 via the cable 18A. The cable 18B is connected to the position / orientation detection unit 30. The position / orientation detection unit 30 takes in a detection signal from the position / orientation detection means 16 via the cable 18B. 1 shows an example in which the HMD 10 and the PC 50 are connected via the cable 18A, and the HMD 10 and the position / orientation detection unit 30 are connected via the cable 18B. However, the cables 18A and 18B are abolished and wirelessly connected. Thus, information may be transmitted to and received from the PC 50 (and the position / orientation detection unit 30).

  The controller 20 is a device for a user to input an operation instruction of the virtual three-dimensional model. As shown in FIG. 2, the main body unit 21, the traveling direction instruction means 22, the speed instruction means 23, and the auxiliary operation instruction means 25A. ˜25D, user identification means 24 and the like. The main body 21 is held by the user and receives input of each instruction from the user. The controller 20 is connected to the PC 50 via the cable 28 and outputs input information corresponding to each input instruction to the PC 50. The input information includes the user number set by the user identification means 24. The controller 20 is configured to be able to input various operation instructions according to the virtual three-dimensional model to be controlled.

  Examples of virtual three-dimensional models include trains (eg, steam locomotives), trains (eg, trains), subway trains, construction machines (eg, power shovels), vehicles (eg, formula machines), military vehicles (eg, tanks), Various remote control operations are meaningful for users who enjoy models, such as ships, warships (for example, battleships), helicopters, aircraft, military aircraft (for example, fighters), submarines, robots, rockets, spacecraft (for example, space battleships) There is a model. The virtual three-dimensional diorama includes a virtual three-dimensional model. In addition to the virtual three-dimensional model, a virtual three-dimensional track (such as a rail for a train or a circuit course for a vehicle) or a virtual three-dimensional background (virtual three-dimensional). The space and surrounding environment in which the 3D model is operated, such as mountains, forests, grasslands, wilderness, urban areas, air, sea, sea, outer space, etc.) and virtual 3D background auxiliary parts (buildings, houses, trees, bridges) Etc.).

  The traveling direction instruction means 22 is for the user to input an instruction for the traveling direction of the virtual three-dimensional model. The traveling direction instructing means 22 shown in FIG. 2 shows an example in which the virtual three-dimensional model is a train, and an example in which instructions for forward, stop, and backward can be input, forward, backward, stop, It is also possible to input various traveling direction instructions according to the virtual three-dimensional model, such as right turn, left turn, ascending and descending. The appearance of the traveling direction instructing unit 22 and the types of traveling directions that can be instructed are not limited to the example in FIG.

  For example, when the virtual three-dimensional model is a train, a train, a subway train, or the like, the traveling direction instruction means 22 can instruct forward, backward, and stop. For example, when the virtual three-dimensional model is a construction machine, a vehicle, a military vehicle, a ship, a warship, or the like, the traveling direction instruction means 22 can instruct forward, backward, stop, right turn, and left turn. For example, when the virtual three-dimensional model is a helicopter, an aircraft, a military aircraft, a submarine, a robot, a rocket, a spacecraft, or the like, the traveling direction instruction means 22 instructs forward, backward, stop, right turn, left turn, rise, and drop. Is possible.

  The speed instruction means 23 is used by the user to input an instruction for the moving speed of the virtual three-dimensional model. The appearance and the like of the speed instruction means 23 are not limited to the example of FIG. Further, the traveling direction instruction means 22 and the speed instruction means 23 may be integrated.

  The auxiliary operation instruction means 25A to 25D are for inputting an instruction of auxiliary operation of the virtual three-dimensional model. For example, when the virtual three-dimensional model is a train, when the user operates the auxiliary operation instruction means 25A, virtual three-dimensional black smoke is discharged from the chimney (corresponding to the smoke discharge position) of the virtual three-dimensional model train, and the user performs an auxiliary operation. When the instruction unit 25B is operated, virtual three-dimensional steam is discharged from the steam outlet (corresponding to the steam discharge position) below the virtual three-dimensional model train, and when the user operates the auxiliary operation instruction unit 25C, the virtual three-dimensional model The sound of the train whistle is output from the sound output device 40. For example, when the virtual three-dimensional model is a construction machine, a vehicle, or a military vehicle, the virtual three-dimensional exhaust is discharged from the exhaust port (corresponding to the exhaust discharge position) of the virtual three-dimensional model when the user operates the auxiliary operation instruction means 25A. The For example, when the virtual three-dimensional model is a military vehicle, a warship, a military aircraft, a submarine, a robot, or a spacecraft, when the user operates the auxiliary operation instruction means 25D, the virtual three-dimensional model is ejected from the bullet outlet (corresponding to the bullet launch position). A virtual 3D shell is fired. For example, when the virtual three-dimensional model is a power shovel (construction machine), when the user operates the auxiliary operation instruction means 25D, the excavator arm (corresponding to a predetermined part) of the virtual three-dimensional model operates so as to excavate the ground ( Equivalent to movement in the specified direction).

  The user identification unit 24 is an apparatus for setting identification information for identifying a user (that is, identifying a virtual three-dimensional model to be operated) when a plurality of controllers 20 are used by a plurality of users. FIG. 2 shows an example in which the user identification means 24 is a rotary dial type and any one of the numbers 1 to 4 can be set. However, the configuration is not limited to the rotary dial type and may be set by a switch or the like. Also, the setting content is not limited to 1 to 4, and may be A to D or the like. For example, when two users each use the controller 20, the first user sets the user identification means 24 of the controller 20 to be used to “1”, and the second user identifies the user identification means of the controller 20 to be used. 24 is set to “2”. Although FIG. 1 shows an example in which the controller 20 and the PC 50 are connected via the cable 28, the cable 28 may be eliminated and information may be transmitted and received with the PC 50 wirelessly.

  The sound output device 40 is, for example, a wired or wireless speaker, earphone, headphone, or the like. When the sound is associated with any one of the auxiliary operation instruction units 25A to 25D of the controller 20 as described above, When the auxiliary operation instruction means associated with is operated by the user, the associated sound is output. The storage device of the PC 50 stores motion-corresponding sound information that is sound information according to the motion of the virtual three-dimensional model (for example, when the virtual three-dimensional model is a train, traveling sound and steam sound data according to the speed) or the controller 20. It has at least one of instruction corresponding sound information (for example, when the virtual three-dimensional model is a train, the whistle sound data associated with the auxiliary operation instruction means 25C) which is sound information corresponding to the input instruction. Alternatively, the storage device of the PC 50 stores a sound generation program (for example, a sound generator) that can automatically generate at least one of operation-corresponding sound information and instruction-corresponding sound information. If the storage device has the sound generation program capable of generating the motion-corresponding sound information or the motion-corresponding sound information, the PC 50 stores the motion-corresponding sound information according to the motion of the three-dimensional model. Based on the sound (sound signal), the sound output device 40 outputs the sound. Further, when the PC 50 has the instruction corresponding sound information or the sound generation program capable of generating the instruction corresponding sound information in the storage device, the instruction corresponding sound information corresponding to the instruction input to the controller A sound (sound signal) based on the above is output from the sound output device 40.

  The PC 50 (corresponding to a computer) is a personal computer, and position / orientation information corresponding to each HMD 10 is input from the position / orientation detection unit 30, a right-eye actual image and a left-eye actual image are input from each HMD 10, and each controller 20 Input information is input from. Then, the PC 50 creates stereoscopic video information corresponding to each HMD 10 and outputs each created stereoscopic video information to the corresponding HMD 10. Further, the PC 50 can send and receive information to and from the diorama providing server 80 via the Internet 60. As will be described later, the user selects a desired virtual 3D diorama from the virtual 3D diorama provided by the diorama providing server 80, receives the diorama information, and stores the diorama information in the storage device of the PC 50. Can be made.

  The diorama providing server 80 is, for example, a server of a manufacturer that sells virtual 3D diorama information (data), and provides a user with a plurality of diorama information corresponding to a plurality of virtual 3D diorama including a virtual 3D model. Yes. The virtual 3D diorama is, for example, a train, a virtual 3D rail, and a virtual 3D background that are virtual 3D models, a vehicle and virtual 3D circuit course that are virtual 3D models, and a military vehicle that is a virtual 3D model. Etc. Hereinafter, a virtual three-dimensional diorama including a train (so-called SL) as a virtual three-dimensional model and a rail as a virtual three-dimensional track will be described as an example.

  In FIG. 3, the example at the time of using the three-dimensional mixed reality provision system 1 in the living L is shown. In FIG. 3, reference numeral 71 is a virtual three-dimensional model train, and reference numeral 72 is a virtual three-dimensional track rail, both of which are not real, but are described as images that can be virtually visually recognized by the user U. ing. Real objects such as a television and a sofa are arranged in the living space L which is a real space. The user U arranges the PC 50 and the position / orientation detection unit 30 at an appropriate location in the living room L, wears the HMD 10, has the controller 20, and has a virtual three-dimensional diorama having a virtual three-dimensional model 71 and a virtual three-dimensional trajectory 72. 70 can be virtually arranged in the living room. Hereinafter, an example of the processing procedure of the PC 50 when the user U enjoys the virtual three-dimensional diorama will be described.

● [PC50 processing procedure (FIGS. 4 to 8)]
Next, the processing procedure of the PC 50 that executes the stereoscopic mixed reality providing program of the present invention will be described using the flowcharts shown in FIGS. In the following description, the explanation is based on the example of FIG. 3, and there is one user U to use, and in the living room L (indoor), a virtual three-dimensional model 71 is used as an example of a virtual three-dimensional diorama. An example using a virtual three-dimensional diorama 70 having a train and a rail that is a virtual three-dimensional track 72 will be described. As shown in FIG. 3, the user U places the PC 50, the position / orientation detection unit 30, and the sound output device 40 at appropriate positions in the living room L. Then, the user U sets the user identification means 14 of the HMD 10 to “1”, connects the cables 18A and 18B to the PC 50 and the position / orientation detection unit 30, and sets the user identification means 24 of the controller 20 to “1”. Connect the cable 28 to the PC 50. Then, when activated by the user U, the PC 50 advances the process to step S10.

  In step S10, the PC 50 displays an input waiting screen for the number of users, and proceeds to step S15.

  In step S15, the PC 50 determines whether or not the number of users (n) has been input. If the number of users has been input (Yes), the process proceeds to step S20, and if the number of users has not been input yet (No ) Returns to step S15. In this case, the user U inputs “1” as the number of users.

  When the process proceeds to step S20, the PC 50 stores the input number of users (in this case, “1”) in the storage device, and proceeds to step SB000. Note that the processing in steps S10 to S20 may be omitted when the PC 50 has a plug-and-play function in advance and can automatically recognize the connected device and automatically recognize the number of users.

  In step SB000, the PC 50 selects a (common) virtual three-dimensional diorama by the user U (m) and places the selected virtual three-dimensional diorama, and proceeds to step S25. The “(common) virtual three-dimensional diorama” handled in step SB000 is a diorama that does not include a virtual three-dimensional model. In step SB100 described later, a virtual three-dimensional model is included in the (common) virtual three-dimensional diorama. Be placed. Details of the process of step SB000 will be described later.

  In step S25, the PC 50 substitutes “1” (initial value) for m (user number) and proceeds to step SB100. Hereinafter, the user with the user number “1” is described as the user U (1), and the user with the user number m is described as the user U (m). The HMD 10 worn by the user U (1) is described as HMD10 (1), and the controller 20 operated by the user U (1) is described as controller 20 (1). Similarly, the HMD 10 of the user U (m) is described as HMD 10 (m), and the controller 20 of the user U (m) is described as the controller 20 (m).

  In step SB100, the PC 50 selects the virtual three-dimensional model (m) that the user U (m) operates with the controller 20 (m) by the user U (m) and the selected virtual three-dimensional model (m). Arrangement (arrangement in the virtual three-dimensional diorama arranged in step SB000) is performed, and the process proceeds to step S30. Details of the process of step SB100 will be described later.

  In step S30, the PC 50 adds “1” to m (user number), updates the value of m (user number) (preparation for the next user), and proceeds to step S35.

  In step S35, the PC 50 determines whether or not the updated value of m (user number) is less than or equal to the stored number of users n, and if m is less than or equal to n (Yes), Returning to step SB100 to select and arrange the virtual three-dimensional model (m) of the user U (m), if m is not less than n (No), the selection and arrangement of the virtual three-dimensional model of all users is completed. It is determined that the process has been completed, and the process proceeds to step S40.

  When the process proceeds to step S40, the PC 50 determines the azimuth angle, which is the horizontal angle of the virtual sun that is the light source for the virtual three-dimensional diorama, and the elevation angle or elevation angle, which is the vertical angle of the virtual sun (virtual time). Corresponding to information, for example, at the time of 12:00, an input waiting screen of elevation angle = 90 degrees is displayed. When the azimuth angle and elevation angle (or virtual time) are input, the PC 50 displays a screen for waiting for input of weather (at least one weather of fine weather, cloudy weather, rain, wind, and snow) that becomes the environment of the virtual three-dimensional diorama. To do. When the weather is input, the PC 50 proceeds to step S45. In addition, you may comprise so that step S40 can be skipped for the user who does not specifically desire the setting of a light source position or a weather.

  In step S45, the PC 50 displays an input waiting screen for starting the operation of the virtual three-dimensional model included in the virtual three-dimensional diorama (that is, starting the provision of stereoscopic mixed reality). When the operation start is input (Yes), the PC 50 proceeds to Step SB200, and when the operation start input is not yet (No), the PC 50 returns to Step S45.

  When the process proceeds to step SB200, the PC 50 operates the virtual three-dimensional model (m) of each user U (m) according to the instruction input from the controller 20 (m) of each user U (m). Proceed to step SB300. Details of the processing in step SB200 will be described later.

  In step SB300, the PC 50 provides each user U (m) with stereoscopic video information (m) for reproducing the stereoscopic video (m) on the 3D display device 12 of the HMD 10 (m) of each user U (m). The generated stereoscopic video information (m) is output to the HMD 10 (m) of the corresponding user U (m), and the process proceeds to step S50. Details of the processing in step SB300 will be described later.

  In step S50, the PC 50 determines the presence / absence of an end instruction. If the end instruction is input (Yes), the process ends. If the end instruction is not input (No), the process returns to step SB200. Continue to provide 3D mixed reality.

● [Step SB000 ((Common) Virtual 3D Diorama (Background, Scene) Selection and Placement) Processing (FIG. 5)]
Next, details of the processing of step SB000 in FIG. 4 will be described with reference to FIG. In step SB000 (particularly, steps SB055 to SB065) and step SB100 (particularly, steps SB155 to SB165) described below, the stereoscopic mixed reality providing program reads the PC 50 from the storage device (or PC 50). And 3D virtual 3D diorama including the virtual 3D model is functioned as virtual 3D diorama placement means for virtually placing the 3D virtual 3D diorama including the virtual 3D model at a specified position in the real space. When the PC 50 proceeds to step SB000, first, at step SB010, a (common) virtual three-dimensional diorama (background, scene) selection menu is displayed, and the process proceeds to step SB015. As an example of the display content, it is selected from virtual three-dimensional diorama (background, scene) stored in the storage device, or is accessed by the diorama providing server 80 by accessing the diorama providing server 80 via the Internet 60. Either the virtual dimensional diorama (background or scene) that is being displayed is selected or a screen waiting for input is displayed. Note that the virtual three-dimensional diorama handled in step SB000 is a background or scene (for example, a rail, a mountainous area, or an urban area) that does not include a virtual three-dimensional model. Also, the meaning of “(common)” in the (common) virtual three-dimensional diorama means that when one user U (m) is singular, one virtual three-dimensional diorama is selected and arranged for one user. Then, “(common)” is not given to the target virtual three-dimensional diorama. However, when there are a plurality of users, one virtual three-dimensional diorama is selected and arranged for a plurality of users. In this case, “(common)” is assigned to the target virtual three-dimensional diorama.

  In step SB015, the PC 50 determines whether or not “selection from the virtual three-dimensional diorama (background, scene) stored in the storage device” is selected, and the “virtual 3 stored in the storage device” is determined. When “selection from dimensional diorama (background, scene)” is selected (Yes), the process proceeds to step SB020A, and “from the virtual three-dimensional diorama (background, scene) provided by diorama providing server 80” When “select” is selected (No), the process proceeds to step SB020B.

  It is also possible to allow a user to create a virtual three-dimensional diorama (background, scene) from scratch. For example, the diorama providing server 80 includes a tool program for creating a virtual three-dimensional diorama (background, scene), various material data (background, background auxiliary parts), a diorama creation manual, and the like. The user can use the created virtual 3D diorama (background, scene) by creating a trick and a virtual 3D diorama (background, scene) over time and storing it in the storage device of the PC 50. .

  When the process proceeds to step SB020A, the PC 50 displays a list of diorama names of virtual three-dimensional diorama (background, scene) stored in the storage device on the screen, and then proceeds to step SB025A.

  In step SB025A, the PC 50 determines whether or not a selection instruction (input instruction) has been given from a list of diorama names displayed on the screen, and if there is a selection instruction (Yes), the process proceeds to step SB030A. If the selection instruction has not yet been made (No), the process returns to step SB025A.

  When the process proceeds to step SB030A, the PC 50 performs reading of the selected virtual three-dimensional diorama (background, scene), and when the reading ends, the process proceeds to step SB055.

  When the process proceeds to step SB020B, the PC 50 accesses the designated diorama providing server 80 via the Internet 60 and provides the virtual three-dimensional diorama (background, scene) provided from the diorama providing server 80. The list information is received, the received list information is displayed on the screen, and the process proceeds to step SB025B.

  In step SB025B, the PC 50 determines whether or not any virtual three-dimensional diorama (background, scene) selection instruction (input instruction) has been given from the list information displayed on the screen. If there is (Yes), the process proceeds to step SB030B, and if the selection instruction is not yet (No), the process returns to step SB025B.

  When the process proceeds to step SB030B, the PC 50 transmits transmission request information requesting transmission of information (data) of the selected virtual three-dimensional diorama (background, scene) to the diorama providing server 80, and from the diorama providing server 80, When the reception (reading) of the information (data) of the selected virtual three-dimensional diorama (background, scene) is performed and the reception ends, the process proceeds to step SB035B.

  In step SB035B, the PC 50 displays a screen waiting for input of the name of the received virtual three-dimensional diorama (background, scene). If the diorama name is input (Yes), the process proceeds to step SB040B, where the diorama name is input. If not yet (No), the process returns to step SB035B.

  When the process proceeds to step SB040B, the PC 50 associates the input diorama name with the information (data) of the received virtual three-dimensional diorama (background, scene), and the diorama name and virtual three-dimensional diorama (background, scene). Are stored in the storage device, and the process proceeds to Step SB055. The stored diorama name is displayed from the next time on the list of step SB020A (the list of stored virtual three-dimensional diorama (background, scene)).

  In step SB055, the PC 50 temporarily arranges the read (or received) virtual three-dimensional diorama (background, scene) at the position designated by the user U (m) in the real space, and proceeds to step SB060. For example, the PC 50 captures the right-eye actual image captured by the right imaging device 13R and the left-eye actual image captured by the left imaging device 13L from the HMD 10 (m) of the user U (m), and the right-eye actual image or A real image that is a real image for the left eye is displayed on the screen (liquid crystal display) of the PC 50. The user U (m) temporarily places a virtual three-dimensional diorama (background, scene) in the real space by instructing an arbitrary position in the real image displayed on the screen from an input device such as a mouse. When the virtual three-dimensional diorama (background, scene) is temporarily arranged, the PC 50 displays the virtual three-dimensional diorama (background, scene) superimposed on the screen displaying the real image.

  In step SB060, the PC 50 determines whether or not the adjustment of the arrangement position of the virtual three-dimensional diorama (background, scene) is input from the user U (m). If the adjustment is input (Yes), the PC 50 determines whether or not the adjustment is input (step SB065). If no adjustment input has been made (No), the process proceeds to step SB080. For example, when the PC 50 receives a sliding amount in each direction of the X axis, the Y axis, and the Z axis (see FIG. 3) of the orthogonal coordinate system with the position / orientation detection unit 30 as a reference, the PC 50 follows the input axis. Then, the virtual three-dimensional diorama (background, scene) superimposed on the real image is slid and displayed on the screen by a distance corresponding to the input slide amount. Further, for example, when the turning angle around the X axis, the turning angle around the Y axis, and the turning angle around the Z axis are input, the PC 50 has an angle corresponding to the inputted turning angle around the inputted axis. The virtual three-dimensional diorama (background, scene) superimposed on the real image is turned and displayed on the screen.

  When the process proceeds to step SB080, the PC 50 determines whether or not the adjustment of the arrangement position of the virtual three-dimensional diorama (background, scene) in the real space is input from the user U (m), and the adjustment is completed. If it is input (Yes), the process of step SB000 is terminated and the process proceeds to step S30 in FIG. If the completion of adjustment is not input (No), the PC 50 returns to Step SB060. For example, the PC 50 determines that the completion of adjustment has been input when the “placement complete” button displayed on a part of its screen is operated.

[Step SB100 (selection and arrangement of virtual three-dimensional model (m) of user U (m)) (FIG. 6)]
Next, details of the processing in step SB100 in FIG. 4 will be described with reference to FIG. When the PC 50 proceeds to step SB100, first, the virtual three-dimensional model selection menu is displayed at step SB110, and the process proceeds to step SB115. As an example of display contents, a virtual three-dimensional model selected from a virtual three-dimensional model stored in a storage device or accessed by the diorama providing server 80 via the Internet 60 and provided by the diorama providing server 80 Either select from the models or select the model and display the screen waiting for input. The virtual three-dimensional diorama handled in step SB100 is a background or scene that does not include the virtual three-dimensional model until the virtual three-dimensional model (1) is arranged in step SB155.

  In step SB115, the PC 50 determines whether or not “selection from the virtual three-dimensional model stored in the storage device” has been selected. If “selection” is selected (Yes), the process proceeds to step SB120A. If “selection from the virtual three-dimensional model provided by the diorama providing server 80” is selected (No), the process proceeds to step SB120B. .

  Note that the virtual three-dimensional model can be created by the user from scratch. For example, the diorama providing server 80 includes a tool program for creating a virtual three-dimensional model, various material data (a base of the virtual three-dimensional model, auxiliary parts, etc.), a model creation manual, and the like. The user can use the created virtual three-dimensional model by creating a trick and a virtual three-dimensional model over time and storing it in the storage device of the PC 50.

  When the process proceeds to step SB120A, the PC 50 displays a list of model names of the virtual three-dimensional model stored in the storage device on the screen and proceeds to step SB125A.

  In step SB125A, the PC 50 determines whether or not a selection instruction (input instruction) has been given from a list of model names displayed on the screen. If there is a selection instruction (Yes), the process proceeds to step SB130A. If the selection instruction has not yet been made (No), the process returns to step SB125A.

  When the process proceeds to step SB130A, the PC 50 reads the selected virtual three-dimensional model, and when the reading is completed, the PC 50 proceeds to step SB150.

  When the process proceeds to step SB120B, the PC 50 accesses the designated diorama providing server 80 via the Internet 60 and receives the list information of the provided virtual 3D model from the diorama providing server 80. Then, the received list information is displayed on the screen, and the process proceeds to Step SB125B.

  In step SB125B, the PC 50 determines whether or not any virtual 3D model selection instruction (input instruction) has been given from the list information displayed on the screen, and if there is a selection instruction (Yes) ) Proceeds to step SB130B, and if the selection instruction has not yet been made (No), the process returns to step SB125B.

  When the process proceeds to step SB130B, the PC 50 transmits transmission request information for requesting transmission of information (data) of the selected virtual three-dimensional model to the diorama providing server 80, and from the diorama providing server 80, the selected virtual 3 When the reception (reading) of the information (data) of the dimension model is performed and the reception ends, the process proceeds to step SB135B.

  In step SB135B, the PC 50 displays a screen waiting for input of the received virtual 3D model name. If the model name is input (Yes), the process proceeds to step SB140B, and if the model name has not yet been input ( No) returns to step SB135B.

  When the process proceeds to step SB140B, the PC 50 stores the model name and the virtual three-dimensional model information in the storage device by associating the input model name with the received virtual three-dimensional model information (data). Proceed to step SB150. The stored model names are displayed in the list of the step SB120A (list of stored virtual three-dimensional models) from the next time.

  When the process proceeds to step SB150, the PC 50 stores the model name in association with the user number “m (in this case,“ 1 ”)” (corresponding to the input instruction from the controller 20 of the user number “1”). Proceed to step SB155 (in order to reflect the model name in the operation of the virtual three-dimensional model). Hereinafter, the virtual three-dimensional model selected and arranged by the user (1) is referred to as a virtual three-dimensional model (1). Similarly, a virtual three-dimensional model selected and arranged by the user (m) is referred to as a virtual three-dimensional model (m).

  In step SB155, the PC 50 reads the virtual three-dimensional model (m) read (or received) in the real space (or in the virtual three-dimensional diorama (background, scene) arranged in step SB000). The process temporarily moves to the position instructed by U (m) and proceeds to step SB160. For example, the PC 50 captures the right-eye actual image captured by the right imaging device 13R and the left-eye actual image captured by the left imaging device 13L from the HMD 10 (m) of the user U (m). The PC 50 also has a virtual diorama image for the right eye that is an image of a virtual three-dimensional diorama that will be seen from the right imaging device 13R of the HMD 10 (m), and a virtual three-dimensional that will be seen from the left imaging device 13L of the HMD 10 (m). A virtual diorama image for the left eye, which is a diorama image, is created. The PC 50 creates a right-eye composite image in which the right-eye virtual diorama image is superimposed on the right-eye real image, and a left-eye composite image in which the left-eye virtual diorama image is superimposed on the left-eye real image, and the created right-eye image The composite image that is the composite image or the composite image for the left eye is displayed on the screen (liquid crystal display) of the PC 50. The user U (m) designates an arbitrary position in the composite image displayed on the screen from an input device such as a mouse, so that the virtual three-dimensional model (m) is displayed in the real space (or (common) virtual 3). Temporary placement in the dimensional diorama). When the virtual three-dimensional model (m) is provisionally arranged, the PC 50 creates the virtual three-dimensional model (m) including the temporarily arranged virtual three-dimensional model (m) when creating the right-eye virtual diorama image and the left-eye virtual diorama image. Create with a diorama.

  In step SB160, the PC 50 determines whether or not the adjustment of the placement position of the virtual three-dimensional model (m) is input from the user U (m). If the adjustment input is made (Yes), the process proceeds to step SB165. If no adjustment input is made (No), the process proceeds to step SB180. For example, when the PC 50 receives a sliding amount in each direction of the X axis, the Y axis, and the Z axis (see FIG. 3) of the orthogonal coordinate system with the position / orientation detection unit 30 as a reference, the PC 50 follows the input axis. Then, the virtual three-dimensional model (m) in the composite image is slid and displayed on the screen by a distance corresponding to the input slide amount. Further, for example, when the turning angle around the X axis, the turning angle around the Y axis, and the turning angle around the Z axis are input, the PC 50 has an angle corresponding to the inputted turning angle around the inputted axis. The virtual three-dimensional model (m) in the composite image is turned and displayed on the screen.

  When the process proceeds to step SB180, the PC 50 determines whether or not the completion of adjustment of the placement position of the virtual three-dimensional model (m) in the real space (or (common) virtual three-dimensional diorama) is input from the user U (m). If the completion of adjustment is input (Yes), the process of step SB100 is terminated and the process proceeds to step S30 in FIG. If the completion of adjustment is not input (No), the PC 50 returns to Step SB160. For example, the PC 50 determines that the completion of adjustment has been input when the “placement complete” button displayed on a part of its screen is operated.

● [Step SB200 (Operation of Virtual 3D Model) (FIG. 7)]
Next, details of the processing of step SB200 in FIG. 4 will be described with reference to FIG. In step SB200 (particularly, steps SB215 and SB220) described below, the stereoscopic mixed reality providing program causes the PC 50 to operate the virtual three-dimensional model based on an instruction input to the controller 20. It functions as an operating means. When proceeding to step SB200, the PC 50 first reads the value of “n” as the number of users in step SB210, substitutes the initial value “1” for the user number “m”, and proceeds to step SB215.

  In step SB215, the PC 50 captures input information (m) from the controller 20 (m) of the user U (m), and proceeds to step SB220. The PC 50 can recognize what instruction is input to the controller (m) based on the input information (m) from the controller 20 (m).

  In step SB220, the PC 50 operates the virtual three-dimensional model (m) corresponding to the user U (m) in accordance with the instruction input to the controller 20 (m) in the (common) virtual three-dimensional diorama, Proceed to step SB225. For example, the (common) virtual three-dimensional diorama includes a virtual three-dimensional model train (virtual three-dimensional model (m)) and a virtual three-dimensional rail, and the controller 20 (m) gives an instruction to move forward at a speed v. When input, the PC 50 advances the virtual three-dimensional model train at a speed v along the virtual three-dimensional rail.

  In step SB225, the PC 50 adds “1” to m (user number), updates the value of m (user number) (preparation for the next user), and proceeds to step SB230.

  In step SB230, the PC 50 determines whether or not the updated value of m (user number) is equal to or less than the stored number of users n, and if m is equal to or less than n (Yes), (common) ) Return to step SB215 to operate the virtual 3D model (m) corresponding to the next user U (m) in the virtual 3D diorama, and if m is not less than n (No), all users It is determined that the operation of the virtual three-dimensional model (m) of U (m) is completed, the process of step SB200 is terminated, and the process proceeds to step SB300 in FIG.

[Step SB300 (Reproduction of stereoscopic video on a three-dimensional display device) (FIG. 8)]
Next, the details of the process of step SB300 in FIG. 4 will be described with reference to FIG. When the PC 50 proceeds to step SB300, it first reads the value of “n” as the number of users in step SB310, substitutes the initial value “1” for “m” of the user number, and proceeds to step SB315.

  In step SB315, the PC 50 captures the position and orientation information (m) of the HMD 10 (m) of the user U (m) from the position and orientation detection unit 30, and detects the position and orientation of the HMD 10 (m) in the real space. Proceed to SB320. Based on the position / orientation information (m), the PC 50 uses, for example, the position of the position / orientation detection unit 30 in the real space as a reference for the X, Y, and Z axes (see FIG. 3) from the position / orientation detection unit 30. The position of the HMD 10 (m) in the real space in each direction and the turn of the HMD 10 (m) corresponding to each of the turning around the X axis, the turning around the Y axis, and the turning around the Z axis in the real space. The posture can be detected. Note that the function of the position / orientation detection unit 30 may be built in the PC 50 or the HMD 10. In step SB315, the stereoscopic mixed reality providing program causes the PC 50 to function as an HMD position / orientation determination unit that determines the position of the HMD 10 and the attitude of the HMD 10 in the real space based on the detection signal from the position / orientation detection unit. .

  In step SB320, the PC 50 causes the real image for the right eye (m) captured by the right imaging device 13R of the HMD 10 (m) of the user U (m) and the actual image for the left eye captured by the left imaging device 13L of the HMD 10 (m). (M) is captured, and the process proceeds to Step SB325. In step SB320, the stereoscopic mixed reality providing program uses the PC 50 as a real image capturing unit that captures the real image for the right eye captured by the right imaging device 13R and the real image for the left eye captured by the left imaging device 13L. Make it work.

  In step SB325, the PC 50 will be visible from the right imaging device 13R of the HMD 10 (m) based on the position and orientation of the HMD 10 (m) detected in step SB315 (including the virtual three-dimensional model (m)). A right-eye virtual diorama image (m), which is an image of a virtual three-dimensional virtual diorama, and a virtual three-dimensional virtual diorama that will be visible from the left imaging device 13L of the HMD 10 (m) (including the virtual three-dimensional model (m)). The left-eye virtual diorama image (m), which is an image, is created, and the process proceeds to step SB330. For example, when there are a plurality of users U (m) and a plurality of virtual three-dimensional models (m), a virtual diorama for right eye (m) including a (common) virtual three-dimensional diorama and a plurality of virtual three-dimensional models (1 to n) And a left-eye virtual diorama image (m). In step SB325, the stereoscopic mixed reality providing program causes the PC 50 to determine the virtual diorama for the right eye, which is an image of a virtual three-dimensional diorama that will be seen from the right imaging device 13R of the HMD 10, based on the determined position and orientation of the HMD 10. It functions as a virtual diorama image creating means for creating an image and a left-eye virtual diorama image that is a virtual three-dimensional diorama image that will be visible from the left imaging device 13L of the HMD 10. Note that the 3D mixed reality providing program uses virtual sunlight, which is virtual light when a virtual sun is arranged at a specified azimuth angle and elevation angle (or virtual time) (see the description of step S40), A virtual 3D diorama for the right eye and a virtual diorama for the right eye and a left eye based on the virtual 3D diorama that is virtually reproduced around the virtual 3D diorama specified by the virtual weather. A virtual diorama image is created on the PC 50. In addition, when creating the right-eye virtual diorama image and the left-eye virtual diorama image, the PC 50 also creates the virtual sun position and weather set in step S40. For example, according to the position of the virtual sun, the virtual three-dimensional model is provided with brightness, shadow, reflected light, and the like, and a right-eye virtual diorama image and a left-eye virtual diorama image to which a set climate background is added are created.

  In step SB330, the PC 50 combines the right-eye virtual diorama image (m) with the right-eye virtual diorama image (m) based on the position and orientation of the HMD 10 (m) detected in step SB315 (m). ) And a left-eye composite image (m) in which the left-eye virtual diorama image (m) is superimposed on the left-eye real image (m), and the process proceeds to step SB335. For example, as shown in FIG. 9, the PC 50 captures the right image based on the position and orientation of the HMD 10 (m) in the real image R10R that is the real image for the right eye (m) captured by the right imaging device of the HMD 10 (m). A composite image G10R that is a composite image for right eye (m) is created by superimposing a virtual diorama image V10R that is a virtual diorama image for right eye (m) that will be visible from the device. In step SB330, the stereoscopic mixed reality providing program causes the PC 50 to determine the right-eye composite image in which the right-eye virtual diorama image is superimposed on the right-eye real image and the left-eye actual image in accordance with the determined position and orientation of the HMD 10. And a composite image creating means for creating a composite image for the left eye in which the virtual diorama image for the left eye is superimposed on.

  In step SB335, the PC 50 reproduces the stereoscopic video (m) on the 3D display device 12 of the HMD 10 (m) based on the composite image for the right eye (m) and the composite image for the left eye (m). Information (m) is created and the process proceeds to step SB340. The stereoscopic video information (m) is created so as to match the playback method of the 3D display device 12 of the HMD 10 (m). For example, in the case of the parallax barrier method or the lenticular lens method, the stereoscopic image information (m) includes a part of the elongated right eye composite image (m) and a part of the elongated left eye composite image (m). They are arranged alternately in the left-right direction. For example, in the case of a liquid crystal shutter type, the stereoscopic video information (m) includes shutter control information in addition to image information. For example, when the right-eye liquid crystal display device and the left-eye liquid crystal display device are independent from each other, the right-eye liquid crystal display device displays the right-eye composite image (m), and the left-eye liquid crystal display device displays the left-eye composite image (m). Is displayed. In step SB335, the stereoscopic mixed reality providing program generates stereoscopic video information generating means for generating stereoscopic video information for reproducing the PC 50 on the three-dimensional display device based on the composite image for the right eye and the composite image for the left eye. To function as.

  In step SB340, the PC 50 outputs the created stereoscopic video information (m) to the HMD 10 (m) of the user (m), and sends the stereoscopic video information to the 3D display device 12 of the HMD 10 (m) of the user (m). The stereoscopic video (m) based on (m) is reproduced, and the process proceeds to Step SB345. In step SB340, the stereoscopic mixed reality providing program causes the PC 50 to function as stereoscopic video reproduction means for reproducing the stereoscopic video on the three-dimensional display device using the created stereoscopic video information.

  In step SB345, the PC 50 adds “1” to m (user number), updates the value of m (user number) (preparation for the next user), and proceeds to step SB350.

  In step SB350, the PC 50 determines whether or not the updated value of m (user number) is less than or equal to the stored number of users n, and if m is less than or equal to n (Yes), In order to reproduce the stereoscopic video (m) on the 3D display device of the HMD 10 (m) of the user U (m), the process returns to Step SB315. Further, when m is not less than n (No), the PC 50 completes the reproduction of the stereoscopic video (m) on the three-dimensional display device of the HMD 10 (m) of all the users U (m). And the process proceeds to step S50 in FIG.

● [Effects of the present application, other examples (FIGS. 10 to 13)]
By using the 3D mixed reality providing system 1 equipped with the 3D mixed reality providing program described in the present embodiment, the user U can input an instruction from the controller 20 as shown in FIG. The virtual three-dimensional model 71 can be operated as desired. Further, the user U can enjoy the virtual three-dimensional model 71 from various angles at various positions. Compared with the actual model, the virtual three-dimensional model 71 can be configured to be faithful to the real thing up to the details, so that a force and impact exceeding the actual model can be expected.

  In addition, since the virtual three-dimensional model 71 does not actually collide with the user U, the user U can approach the virtual three-dimensional model 71 with all the way. In addition, since it can be enjoyed with a “stereoscopic” video, it is possible to enjoy a powerful and impactful stereoscopic video as shown in the example of FIG. Further, as shown in the example of FIG. 10, the user can perform the auxiliary operation of the controller 20 for the discharge of the virtual three-dimensional smoke 71Y, the discharge of the virtual three-dimensional steam 71Z, the output of the horn sound, etc., which are difficult with an actual model. It can be performed simply by inputting an instruction from the instruction means 25A to 25D, and the force and impact exceeding the actual model can be enjoyed.

  In addition, since the position of the virtual sun can be set freely, it is possible to shine a sepia sunset light from the desired direction, and the climate can be set, so the background scene can be made snowstorm, or a real model Even if it is difficult or impossible in the surrounding environment, the 3D mixed reality providing system described in this embodiment can be realized in a virtual environment and excite the atmosphere of a virtual three-dimensional diorama. be able to.

  FIG. 11 shows an example in which a virtual three-dimensional diorama is arranged so that a virtual three-dimensional trajectory 72 (rail) of the virtual three-dimensional diorama intersects the wall L1 of the living space L in the real space. In this case, the PC 50 determines an intersection where the wall L1 of the living room L1 intersects with the virtual three-dimensional trajectory 72, and automatically creates virtual three-dimensional tunnel entrances 73 and 74 at the determined intersection. Then, the PC 50 erases the display of the virtual three-dimensional trajectory at a position farther from the virtual three-dimensional tunnel entrances 73 and 74, and when the virtual three-dimensional model moves to a position farther than the virtual three-dimensional tunnel entrance, Turn off the display. In this way, even an arrangement impossible with an actual model can be made possible. Further, since the virtual 3D trajectory 72 can be arranged so that the virtual 3D trajectory 72 is at a predetermined height from the floor of the living room L, the virtual 3D can be viewed at an angle looking up from below, which is difficult with an actual model. You can enjoy the model.

  FIG. 12 shows an example in which a virtual three-dimensional diorama having a submarine 71B as a virtual three-dimensional model and a deep sea 75 as a background environment is arranged in the living space L of the real space. Thus, even if it is very difficult to enjoy the operation with an actual model, it can be easily enjoyed. In addition, when a real model is operated in actual water, it is very difficult for the user to enjoy it from various angles at various positions by approaching the model. However, by using the three-dimensional mixed reality providing system described in the present embodiment, the user can use the virtual three-dimensional model submarine 71B submerged in the virtual deep sea 75 while staying in the living space L of the real space. I can enjoy it.

  FIG. 13 shows an example of taking the 3D mixed reality providing system 1 described in the present embodiment out of the park or the like and enjoying various virtual three-dimensional models in various real spaces. . The example of FIG. 13 shows an example where the user U1 enjoys operating a virtual three-dimensional model military aircraft 71C (fighter) in the outdoors, and the user U2 is a virtual three-dimensional model vehicle 71D (formula machine). An example is shown in which a user U3 enjoys the game by operating a virtual three-dimensional model warship 71E (battleship) in a park pond or the like. Since the virtual three-dimensional model military aircraft 71C (and the vehicle 71D and the warship 71E) is not real, it does not cause trouble to the surrounding people without colliding. In addition, the use of earphones and headphones does not disturb the surrounding people. Moreover, since the user U1 (and U2 and U3) can also see the situation of the surrounding real space, even if the user U1 (and U2 and U3) moves around, the user U1 (and U2 and U3) may collide with surrounding people or fall into a pond. And you can enjoy it safely.

  In addition, adding visual and acoustic effects to the virtual 3D model's movements can make it more realistic and less than the force and impact of a real model, or can exceed the force and impact of a real model. I can expect. Note that visual effects and acoustic effects are more preferably added as follows, for example, according to the type of virtual three-dimensional model (train, vehicle, aircraft, ship, etc.).

  For example, when the virtual three-dimensional model is a military aircraft 71C, an engine sound corresponding to the magnitude of the speed instructed from the controller 20 is output from a sound output device (for example, an earphone). Further, for example, a virtual three-dimensional exhaust (or contrail) generation instruction is associated with one of the auxiliary operation instruction units 25A to 25D of the controller 20, and the exhaust port (exhaust discharge position) of the virtual three-dimensional model military aircraft 71C. The virtual three-dimensional exhaust (or contrail) can be discharged from the equivalent). Also, for example, a virtual three-dimensional bullet firing instruction is associated with any one of the auxiliary operation instruction means 25A to 25D of the controller 20, and the virtual bullet is ejected from the bullet launch port (corresponding to the bullet firing position) of the military aircraft 71C of the virtual three-dimensional model. Enable to fire 3D shells with sound.

  For example, when the virtual three-dimensional model is the vehicle 71D, an engine sound corresponding to the magnitude of the speed instructed from the controller 20 is output from a sound output device (for example, an earphone). Further, for example, a tire slip sound is output from a sound output device (for example, an earphone) according to a right turn or left turn instruction input from the controller 20 and the magnitude of the speed. Further, for example, a virtual three-dimensional exhaust gas generation instruction is associated with any one of the auxiliary operation instruction units 25A to 25D of the controller 20, and the virtual three-dimensional model 71D corresponds to the virtual three-dimensional model from the exhaust port (corresponding to the exhaust discharge position) of the vehicle 71D. Enable to discharge dimension exhaust.

  For example, when the virtual three-dimensional model is the warship 71E, an engine sound corresponding to the magnitude of the speed instructed from the controller 20 is output from a sound output device (for example, an earphone). Further, for example, the state of the wave generated at the tip of the hull is changed or the trajectory is left in the navigation route according to the right turn or left turn instruction input from the controller 20 and the magnitude of the speed. Further, for example, a virtual three-dimensional bullet firing instruction is associated with any one of the auxiliary operation instruction means 25A to 25D of the controller 20, and the virtual three-dimensional model warship 71E has a bullet launch port (corresponding to the bullet firing position) to the virtual 3D. Dimensional shells can be fired with a firing sound. If the impact of the bullets between the virtual three-dimensional models is reached, the process of damaging the landing part can be performed together with the explosion sound, and development into a battle game or the like is also possible.

  In the description of the present embodiment, the example in which the processing illustrated in FIGS. 4 to 8 is executed by the PC 50 has been described, but the PC 50 and the HMD 10 are executed in cooperation (part of the processing is executed by the HMD 10). You may make it do.

  In the description of the present embodiment, when there are a plurality of users U (m), each user U (m) displays a (common) virtual 3D diorama stereoscopic image including a plurality of virtual 3D models (1 to n). The example which can be visually recognized was demonstrated. However, for example, user U (1) and user U (2) respectively arrange virtual three-dimensional diorama (1) and virtual three-dimensional diorama (2), and user U (1) only has virtual three-dimensional diorama (1). It is also possible to set so that the user U (2) can visually recognize only the virtual three-dimensional diorama (2). In this case, even if a plurality of users arrange a plurality of virtual three-dimensional diorama in a narrow real space such as a living room, they can enjoy each virtual three-dimensional diorama.

  In the description of the present embodiment, an example in which a virtual three-dimensional diorama includes a virtual three-dimensional model has been described. However, a virtual three-dimensional diorama may include a virtual three-dimensional background using an actual model. For example, a virtual N-dimensional rail and train, and a virtual three-dimensional diorama including a virtual three-dimensional background (mountainous area, urban area, etc.) and climate (climate such as rain, snowstorm, etc.) may be used.

  In the example of FIG. 1 and the description of the present embodiment, an example in which a single HMD and controller or a plurality of HMDs and controllers are connected to one PC 50 has been described. And a plurality of users can enjoy a battle or the like using one (common) virtual three-dimensional diorama. That is, for one or more PCs 50, a plurality of HMDs (m) and a virtual 3D corresponding to the HMD (m) using each controller (m) corresponding to each HMD (m). You can enjoy a battle in a (common) virtual three-dimensional diorama including the model (m).

  The three-dimensional mixed reality providing program for the virtual three-dimensional model of the present invention is not limited to the processing procedure described in the present embodiment, and various modifications, additions, and the like are within the scope of the present invention. Can be deleted.

  The numerical values used in the description of the present embodiment are examples, and are not limited to these numerical values.

1 3D mixed reality providing system 10 HMD (head mounted display)
12 three-dimensional display device 13R right imaging device 13RV (right eye) frontal line-of-sight direction 13L left imaging device 13LV (left eye) frontal line-of-sight direction 14 user identification means 16 position and orientation detection means 20 controller 22 travel direction instruction means 23 speed instruction means 24 User identification means 25A to 25D Auxiliary operation instruction means 30 Position and orientation detection unit 40 Sound output device 50 PC (personal computer)
60 Internet 70 Virtual 3D Diorama 71 Virtual 3D Model 72 Virtual 3D Trajectory 80 Diorama Provision Server

Claims (5)

An HMD that is a non-transmissive head-mounted display worn so as to cover both eyes of the user;
A controller capable of inputting instructions from the user;
A three-dimensional mixed reality providing program for a virtual three-dimensional model using a computer,
The HMD is a three-dimensional display device that allows a user to visually recognize a stereoscopic image, at least one right imaging device that images the front line-of-sight direction of the user's right eye, and at least that images the front line-of-sight direction of the user's left eye. One left imaging device, and position and orientation detection means that enables the computer to detect the position and orientation of the HMD in the real space,
The computer, or the computer and the HMD cooperate,
A virtual that virtually arranges a virtual three-dimensional diorama, including a virtual three-dimensional model, read from a storage device by the computer or received via a communication line by the computer at a specified position in the real space. 3D diorama placement means,
Virtual three-dimensional model operating means for operating the virtual three-dimensional model based on an instruction input to the controller;
HMD position / posture determination means for determining the position of the HMD and the posture of the HMD in real space based on a detection signal from the position / posture detection means;
A real image capturing means for capturing the real image for the right eye in the real space captured by the right image capturing device and the real image for the left eye in the real space captured by the left image capturing device;
Based on the determined position and orientation of the HMD, from the virtual diorama image for the right eye that is an image of the virtual three-dimensional diorama that will be seen from the position of the right imaging device of the HMD, and from the position of the left imaging device of the HMD Virtual diorama image creation means for creating a left-eye virtual diorama image that is an image of the virtual three-dimensional diorama that will be visible;
According to the determined position and orientation of the HMD, a right-eye composite image in which the right-eye virtual diorama image is superimposed on the right-eye real image, and a left-eye virtual diorama image on the left-eye real image. A composite image creating means for creating a composite image;
3D image information creating means for creating 3D image information to be reproduced on the 3D display device based on the composite image for the right eye and the composite image for the left eye;
Using the created stereoscopic video information, the 3D display device functions as a stereoscopic video playback unit that plays back the stereoscopic video.
A 3D mixed reality providing program for virtual 3D models. A three-dimensional mixed reality providing program for the virtual three-dimensional model according to claim 1,
The virtual three-dimensional model is at least one of a train or train or subway train, a construction machine, a vehicle or military vehicle, a ship or warship, a helicopter or aircraft or military aircraft, a submarine, a robot, a rocket, a spacecraft,
The controller includes an instruction for the speed of the virtual three-dimensional model, an instruction to move the virtual three-dimensional model forward or backward, an instruction to turn the virtual three-dimensional model to the right or left, an ascent of the virtual three-dimensional model, Indication of lowering, instruction of virtual 3D smoke discharge from the smoke discharge position in the virtual 3D model, instruction of virtual 3D steam discharge from the vapor discharge position in the virtual 3D model, in the virtual 3D model Instructions for discharging virtual three-dimensional exhaust from the exhaust discharge position, instructions for discharging virtual three-dimensional shells from the bullet firing position in the virtual three-dimensional model, instructions for movement of a predetermined part in the virtual three-dimensional model in a specified direction , At least one instruction can be entered,
When the computer or the computer and the HMD cooperate to function as the virtual three-dimensional model operation means, the virtual three-dimensional model is operated according to an instruction input to the controller.
A 3D mixed reality providing program for virtual 3D models. A three-dimensional mixed reality providing program for a virtual three-dimensional model according to claim 1 or 2,
The storage device stores at least one of motion-corresponding sound information that is sound information corresponding to the motion of the virtual three-dimensional model or instruction-corresponding sound information that is sound information corresponding to an instruction input to the controller. Or a sound generation program capable of generating at least one of the action corresponding sound information and the instruction corresponding sound information is stored,
Using a sound output device that outputs a sound according to a sound signal based on the action-corresponding sound information and the instruction-corresponding sound information,
When the computer or the computer and the HMD cooperate to function as the virtual three-dimensional model operation means,
In the case where the storage device has the sound generation program that has the motion corresponding sound information or can generate the motion corresponding sound information, the motion corresponding to the motion of the virtual three-dimensional model Outputting a sound signal based on sound information to the sound output device;
In the case where the storage device has the instruction corresponding sound information or the sound generation program capable of generating the instruction corresponding sound information, the instruction corresponding to the instruction input to the controller Outputting a sound signal based on the sound information to the sound output device;
A 3D mixed reality providing program for virtual 3D models. A three-dimensional mixed reality providing program for the virtual three-dimensional model according to any one of claims 1 to 3,
As a light source for the virtual three-dimensional diorama, an azimuth angle that is a horizontal angle of the virtual sun that is the sun virtually set in the virtual three-dimensional diorama, and an elevation angle that is an angle in the vertical direction of the virtual sun or Virtual time information for determining the elevation angle;
The weather of the virtual diorama can be specified as clear weather or cloudy weather or virtual weather which is at least one weather of rain or wind or snow,
When the computer or the computer and the HMD cooperate to function as the virtual diorama image creation means,
Virtual sunlight, which is virtual light when the virtual sun is arranged at a position corresponding to a designated azimuth angle and a designated elevation angle or an elevation angle based on the virtual time information, is virtually irradiated. The right-eye virtual diorama image and the left-eye virtual diorama image on the basis of the virtual three-dimensional diorama, which is the virtual three-dimensional diorama around which the designated virtual weather is virtually reproduced. Create
A 3D mixed reality providing program for virtual 3D models. A three-dimensional mixed reality providing program for the virtual three-dimensional model according to any one of claims 1 to 4,
For one or more computers, using a plurality of the HMDs and the controllers corresponding to the HMDs,
A common virtual three-dimensional diorama that is common to a plurality of the HMDs and that includes the virtual three-dimensional model for each HMD can be set;
The computer, or the computer and the HMD cooperate,
When functioning as the virtual three-dimensional diorama arranging means, the common virtual three-dimensional diorama is virtually arranged,
When functioning as the virtual three-dimensional model operation means, each virtual three-dimensional model included in the common virtual three-dimensional diorama is operated corresponding to each controller,
When functioning as the HMD position and orientation determination means, determine the position and orientation for each HMD,
When functioning as the real image capturing means, the real image for the right eye and the real image for the left eye are captured for each HMD,
When functioning as the virtual diorama image creating means, the virtual diorama image for the right eye and the left eye for each HMD using the common virtual three-dimensional diorama including a plurality of virtual three-dimensional models that are virtually arranged Create virtual diorama images for
When functioning as the composite image creation means, the right eye composite image and the left eye composite image are created for each HMD,
When functioning as the stereoscopic video information creation means, the stereoscopic video information is created for each HMD,
When functioning as the 3D video playback means, the 3D display device corresponding to each HMD is caused to play back the 3D video corresponding to each HMD.
A 3D mixed reality providing program for virtual 3D models.

Images