JP2017146762A - Image display type simulation service providing system and image display type simulation service providing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display type simulation service that allows persons interested in somesthetic experience to go around by themselves, makes it possible for the persons to attain somesthetic experience of a virtual space, resolves a picture-sickness symptom peculiar to a three-dimensional stereoscopic viewing, and makes it possible to avoid risks of usage in a real world by displaying images in combination of the real world with virtual space.SOLUTION: An image display type simulation service providing system is configured to: extract arbitrary image information from preliminarily shot image information; create three-dimensional virtual image information with reference to the extracted arbitrary image information; consecutively measure positions of persons or objects, or poses thereof by a motion capture device 200; perform synthesis processing of synthesizing image information actually shot by a camera as to measurement objects of the motion capture device 200 with the three-dimensional virtual image information with reference to information on the measurement, and create synthesis image information; and output the synthesis image information to an image display device 300 to display the output synthesis image information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image display type simulation service providing system and an image display type simulation service providing method that enable a simulated experience with a composite image.

  Conventionally, model rooms have been used in the sale of apartment houses such as condominiums. The model room is set up as a business meeting place with a large space near the apartment building construction site. There are actually some variations in the room types of condominiums sold, but not all of them are prepared as model rooms. Usually, one standard room type model room is prepared. It is what is done.

  As for other room types, the model room tourer himself had to imagine the standard room type model room, receiving explanations from dealers and referring to brochures and drawings.

  Patent Document 1 listed below discloses a technique for creating and displaying a CG image in which a building such as a building schedule is created as an image (CG image) by computer graphics and displayed as a stereoscopic image. According to Patent Document 1, three-dimensional shape data relating to a building is created in advance by a CAD system, and a virtual viewpoint position set for the three-dimensional shape data is determined according to an input operation by an operator. CG image data corresponding to the CG image representing the state of the building viewed from the desired position is generated by calculation based on the three-dimensional image data and the viewpoint position by stopping or moving the shape data in the desired direction. When the viewpoint position is stopped based on the CG image data, the CG image related to the building is displayed as a still image, and when the viewpoint position is moving, it is displayed as a moving image and the viewer wears stereoscopic glasses. Thus, a still image or a moving image can be recognized as a stereoscopic image.

  In Patent Document 2 below, a building such as a building schedule can be viewed as a still image or a moving image in a general home using a display device such as a home game machine and a television or a head mounted display (HMD). The technique regarding the display method of the building using the game machine to make is disclosed. In this document, image data relating to a building is recorded on a recording medium of a game machine, and the viewpoint position with respect to the image data is stopped or desired by operating an operation means attached to the game machine. The CG image representing the building is displayed as a still image or a moving image on the display device linked with the game machine by performing a predetermined calculation based on the viewpoint position and the image data.

JP-A-9-106464 Japanese Patent Laid-Open No. 9-147141

  There is only one standard room type in the model room provided near the conventional apartment building construction site. For other room types, the description or drawings of the seller or The only way for visitors to imagine from pamphlets was to imagine. In particular, it was difficult to accurately grasp the floor plan and sunshine situation, which are important elements in living, under such circumstances.

  In what is disclosed in Patent Document 1, a viewer wearing a 3D glasses is assumed to be positioned at a fixed point, and a CG image is recognized as a 3D image by an operation by an operator. For this reason, it is impossible to confirm the situation where the viewer himself / herself actually moves around.

  In what is disclosed in Patent Document 2, by operating the operating means and instructing the moving direction, the customer who is presumed to be located at a fixed point is walking around the interior of the building. In the same way, each part of the building can be observed and evaluated independently. However, it is impossible to confirm the situation when the customer actually moves around.

  By the way, recently, virtual reality (VR) technology, in which the surrounding environment image is generated by CG and the viewer sees the image, the virtual reality technology (VR) technology has been rapidly developed due to technological progress such as improvement of computer computing performance. It is developing. There are a variety of systems that apply VR, and training simulation systems such as flight simulators used by airline pilots and drive simulators used to improve driving skills at driving schools are typical. is there. The reality is that simulations closer to reality are possible due to improvements in VR technology.

  In addition, it is said that the goodness of HMD that has been attracting attention recently is the immersive feeling that covers the entire field of view. This is because the HMD wearer can feel the presence of being in a virtual world. By using this HMD, the range of use of training simulators has also expanded.

  However, in 3D stereoscopic vision, there is a strong tendency for symptoms similar to motion sickness. For example, when a user sees a VR image that moves violently while sitting in a chair, the information that the human sense is “not moving” is sent from the image in the brain. This is due to the confusion that causes the confusion that “there should be”. For this reason, symptoms of sickness appear and there is a risk of health damage.

  Further, since the HMD covers the entire field of view, there is a high possibility that the actual surrounding environment cannot be grasped. The image displayed on the HMD viewed by the wearer is a reproduction of the virtual space by CG, and is different from the environment around the actual wearer. For example, when performing a VR experience in a small room of about 6 tatami mats, if a large virtual space of 100 tatami mats or more is generated, if the wearer runs around in a wide space, there is a risk of colliding with an actual wall. . In order to avoid such a situation, the wearer is currently required to sit on a chair or the like so as not to move at a fixed position. This also causes sickness and narrows the range of use of HMD.

The present invention has been made in view of the above circumstances, and solves the above-described problems, and does not assume that the sensation person is positioned at a fixed point, so that the sensation person can move around and experience the virtual space. It is an object to provide a service providing system and an image display type simulation service providing method.
In addition, the present invention eliminates video sickness symptoms peculiar to three-dimensional stereoscopic vision, and displays an image by combining the real space and the virtual space, thereby making it possible to avoid danger in use in the real world. An object is to provide a type simulation service providing system and an image display type simulation service providing method.

  In order to achieve the above object, the present invention has the following configuration.

(1) Reference is made to image information storage means for storing various image information, image information extraction means for extracting arbitrary image information from the image information storage means, and the arbitrary image information extracted from the image information extraction means 3D virtual image information generating means for generating 3D virtual image information, and 3D virtual image information generated by the 3D virtual image information generating means and real image information are combined to generate a composite image A simulation management apparatus comprising synthetic image information generation means for generating information;
A motion capture device capable of continuously measuring the position and orientation of a person or object in time and sending the measurement information to the simulation management device;
An image display device for outputting and displaying the composite image information sent from the simulation management device;
An image display type simulation service providing system comprising: an imaging device provided in the image display device, capable of capturing at least the front of the image display device, and capable of transmitting the captured image information to the simulation management device. ,
The composite image information generating means of the simulation management apparatus recognizes the captured image information transmitted from the imaging apparatus as the actual image information and combines the composite image information based on the measurement information transmitted from the motion capture apparatus. The image display device is configured to output and display the composite image information.

(2) In the configuration of (1), the motion capture device is configured to have an optical sensor type, inertial sensor type, ultrasonic sensor type or magnetic sensor type motion capture function.

(3) The image display type simulation service providing method of the present invention extracts arbitrary image information from various image information created or photographed in advance, and refers to the extracted arbitrary image information to perform tertiary processing. Original virtual image information is generated, and the position and posture of a person or object are continuously measured with a motion capture device, and the measurement information of the motion capture device is actually measured with reference to the measurement information from the motion capture device. The image information captured by the camera and the three-dimensional virtual image information are combined to generate combined image information, and the combined image information is output and displayed on a display device.

(4) In the configuration of (3) above, a display device that is mounted on the measurement target using the imaging information in front of the imaging device as image information actually captured by the camera using the imaging device provided in the display device The above composite image information is output and displayed.

According to the above configuration, the sensation person (measurement target) can move around and experience the virtual space without being located at a fixed point.
In addition, according to the above configuration, since the sensation person does not need to be positioned at a fixed point, it eliminates video sickness symptoms peculiar to three-dimensional stereoscopic vision and displays an image by combining the real space and the virtual space, thereby realizing the real world It is possible to avoid the dangers of use in Japan.
Furthermore, according to the said structure, since it can implement | achieve, without requiring a large-scale simulator installation, it is very useful economically and practically.

According to the present invention, an experienced person can move around and experience a virtual space without being located at a fixed point.
In addition, according to the present invention, it eliminates video sickness symptoms peculiar to three-dimensional stereoscopic vision, and by displaying an image by combining a real space and a virtual space, it is possible to avoid dangers in use in the real world, It has a practically excellent effect.

It is a figure which shows notionally the whole structure of the image display type simulation service provision system concerning one Embodiment of this invention. It is a figure which shows the schematic structure of a simulation management apparatus in connection with the embodiment. It is a figure which shows typically the external appearance structure concerning an image display apparatus and an imaging device in connection with the embodiment, (a) is from the front side, (b) is a figure from the left side. It is a figure which shows the general flow of the whole process of the image display type simulation service providing method according to the embodiment. It is a figure which shows typically the mode of the positional information acquisition by the optical sensor type motion capture apparatus in connection with the embodiment. It is a figure which shows the flow of the process of the positional information acquisition by the optical sensor type motion capture apparatus in connection with the embodiment. It is a figure which shows typically the mode of the positional information acquisition in connection with the embodiment by an inertial sensor type motion capture device. It is a figure which shows the flow of the process of the positional information acquisition by the inertial sensor type motion capture apparatus in connection with the embodiment. It is a figure which shows typically the mode of the positional information acquisition in connection with the same embodiment by an ultrasonic sensor type motion capture apparatus. It is a figure which shows the flow of the process of position information acquisition in connection with the same embodiment by an ultrasonic sensor type motion capture device. FIG. 4 is a diagram schematically illustrating a method for generating a composite image according to the embodiment.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  The image display type simulation service providing system 10 according to the present embodiment includes a simulation management device 100, a motion capture device 200, an image display device 300, a photographing device 400, and an operation device 500. In this system 10, an HMD is adopted as the image display device 300, and a virtual space (virtual space) is combined with a motion capture device 200 that detects the position of a person (measurement target, that is, a sensible person) wearing the HMD in real time. For example, a model room of an apartment house such as an apartment can be experienced.

  As shown in FIG. 2, the simulation management apparatus 100 includes a central control processing unit 110, an input unit 120, an output unit 130, a storage unit 140, an image information extraction unit 150, a 3D virtual image information generation unit 160, and a composite image information generation unit. 170, each of which is connected via a signal 180.

  The central control processing unit 110 has a function of managing control processing of the entire system with reference to predetermined program information stored in the storage unit 140. In addition, the central control processing unit 110 receives information transmitted from the outside via the input unit 120 and stores it in the storage unit 140, and outputs information stored in the storage unit 140 and input information from the input unit 120. It has a function of sending to the outside via the unit 130.

  The input unit 120 has a function of receiving information transmitted from the outside and storing it in the storage unit 140 under the control of the central control processing unit 110. For example, it is configured to receive information transmitted from the motion capture device 200 or the imaging device 400 and store it in the storage unit 140. Further, the input unit 120 has a function of storing, in the storage unit 140, input information by key input from an internal keyboard or the like under the control of the central control processing unit 110.

  The output unit 130 has an output function for outputting various types of information stored in the storage unit 140 or sending the information to the outside under the management of the central control processing unit 110. For example, the function is configured to send the composite image information stored in the storage unit 140 to the image display device 300.

  The storage unit 140 is a readable / writable memory device. The storage unit 140 includes a program information storage area 140a, an image information storage area 140b, a photographing information storage area 140c, a composite image information storage area 140d, a measurement information storage area 140e, a work area 140f, and the like.

  Thus, the program information storage area 140a is a storage area for storing program information that the central control processing unit 110 refers to for various control processes and management.

  The image information storage area 140b is a storage area for storing various types of image information that have been taken or created in advance. The image information storage area 140b is also a storage area for storing various image content information, menu information, and the like.

  The shooting information storage area 140c is a storage area for storing shot image information. That is, this is a storage area for storing photographed image information photographed by the photographing apparatus 400 and transmitted via the input unit 120.

  The composite image information storage area 140d is a storage area for storing composite image information. That is, it is a storage area in which the composite image information generated by the composite image information generation unit 170 is stored.

  The measurement information storage area 140e is a storage area for storing measurement information related to the position and orientation of a person or an object measured continuously in time by the motion capture device 200.

  The work area 140f is an area for performing various kinds of work on information to be installed and information to be transmitted.

  The image information extraction unit 150 is based on information transmitted through the input unit 120 from the image information stored in the image information storage area 140b of the storage unit 140 under the control of the central control processing unit 110. It has a function of extracting arbitrary image information.

  The three-dimensional virtual image information generation unit 160 has a function of generating three-dimensional virtual image information with reference to arbitrary image information extracted by the image information extraction unit 150 under the control of the central control processing unit 110. It is. Here, the generated three-dimensional virtual image information is generated as a pair of three-dimensional virtual image information for right eye and left eye.

  The composite image information generation unit 170 performs composite processing of the 3D virtual image information generated by the 3D virtual image information generation unit 160 and the real image information under the control of the central control processing unit 110 to generate composite image information. It has the function to produce | generate as. That is, the composite image information generation unit 170 recognizes the captured image information captured by the imaging apparatus 400 as actual image information, and generates 3D virtual image information based on the measurement information transmitted from the motion capture apparatus 200. The three-dimensional virtual image information generated by the unit 160 and the captured image information from the image capturing apparatus 400 are combined and synchronized, and the combined image information generated by the combining process is stored in the combined image information storage area 140d. Is. Here, the generated composite image information is generated by combining two sets of right-eye and left-eye composite image information.

  The composite image information generated in this way is configured to be sent out to the image display device 300 via the output unit 130 by the simulation management device 100 under the control of the central control processing unit 110.

  The motion capture device 200 has a function of digitally measuring the position and orientation of a person or an object in a time-continuous manner, and is configured to detect position information of the image display device 300 and send the measurement information to the simulation management device 100. Has been. The motion capture device 200 includes a plurality of non-contact optical motion capture cameras 202 and a marker (sensor) 204 attached to the image display device 300.

  In this embodiment, an optical sensor type motion capture device is used, but an inertial sensor type motion capture device, an ultrasonic sensor type or a magnetic sensor type non-contact type motion capture device may be used. Of course.

  The image display device 300 is an HMD that can be detachably worn by a wearer (sensory user), and has a function of outputting and displaying composite image information sent from the simulation management device 100.

  The imaging device 400 includes a right-eye camera 420 and a left-eye camera 440 and is provided in a front portion of the image display device 300 so that the front of the image display device 300 can be captured. Image information captured by the imaging apparatus 400 is configured to be sent to the simulation management apparatus 100. Thus, when the wearer wears the image display device 300, the right-eye camera 420 is positioned in front of the wearer's right eye, and the left-eye camera 440 is positioned in front of the wearer's left eye.

  The operation device 500 has a light function and a hand camera function that can be operated by a wearer (sensory person) of the image display device 300, and can display captured image information as a picture-in-picture in a video image of the image display device 300. It has the function to do.

  The image display-type simulation service providing system 10 is configured so that a third party other than the wearer can simultaneously check image information viewed by the wearer of the image display device 300 on a separate monitor. Yes. Thereby, it is functionally configured so that the information pointed out by the wearer can be shared while also being viewed by a third party.

  The effect | action is demonstrated below about the said structure.

  When the wearer of the image display device 300 selects a desired model room from the menu displayed on the image display device 300 via the operation device 500 (see step S100 in FIG. 4), image information extraction of the simulation management device 100 is performed. The unit 150 extracts corresponding image information from the image information storage area 140b of the storage unit 140 (step S200). That is, the image information of the room of the corresponding model room that has been photographed or created in advance is prepared.

  At the same time, the position of the right eye and the position of the left eye are calculated based on the measurement information from the motion capture device 200 (step S300).

  With reference to the calculated position information of the right eye and the left eye, three-dimensional virtual image information is created for the extracted image information (step S400). That is, the three-dimensional virtual image information generation unit 160 generates the right-eye and left-eye three-dimensional virtual image information at a position corresponding to the wearer in the model room.

  Thus, the right-eye and left-eye photographed image information sent from the photographing device 400 is acquired (step S500), and image synthesis with the right-eye and left-eye three-dimensional virtual image information created in step S400 is performed. Processing is performed by the composite image information generation unit 170 (step S600).

  The synthesized composite image information is sent to the image display device 300 and output and displayed (step S700). Thereby, the wearer can feel as if he / she is located in the model room.

  If the wearer moves (Y in step S800), the process proceeds to step S300, where a new image composition process is performed and output and displayed.

  When the operation device 500 is operated (step S850), processing corresponding to the operation is performed (step S900).

  The function of the optical sensor type motion capture device 200 will be described in detail below with reference to FIGS.

In the optical sensor type, for example, the marker 204 attached to the wearer A is photographed by a plurality of cameras 202 installed around the room (step S602 in FIG. 6), and the position of the marker 204 is analyzed by analyzing the image. Information is calculated (step S604). Thus, the position of the photographing apparatus 400 is calculated from the position information (step S606). The position information calculated in this way is used for the position information acquisition process in step S300.
The optical sensor type motion capture device can obtain very accurate data.

  Optical sensor type motion capture devices are provided by, for example, Vicon Motion Systems, UK, Motion Analysis, USA, and Natural Point, USA.

  By the way, as the motion capture device, an inertial sensor type and an ultrasonic sensor type are provided in addition to the optical sensor type.

  Hereinafter, functions of the inertial sensor type motion capture device will be described with reference to FIGS.

In the inertial sensor type, for example, a plurality of inertial sensors 206 are mounted on the body of the wearer B, the movement amount of each sensor 206 is received (step S802 in FIG. 8), and the movement of the wearer B is determined from the received movement amount. Is calculated (step S804). Thus, the position of the photographing apparatus 400 is calculated from the position information (step S806). The position information calculated in this way is used for the position information acquisition process in step S300.
The inertial sensor type motion capture device is not limited in the place of use, and can be used in a desired place instead of a preset place.

  For example, an inertial sensor type motion capture device is provided as an MVN motion capture system by Xsens of the Netherlands.

  Next, functions of the ultrasonic sensor type motion capture device will be described with reference to FIGS.

In the ultrasonic sensor type, for example, a plurality of ultrasonic transmitters 208 are installed on the ceiling of a room and the ultrasonic wave from the ultrasonic transmitter 208 is received by the ultrasonic receiver 210 attached to the body of the wearer C. (Step S1002 in FIG. 10), and the position information of the receiver is calculated from the phase difference of the received ultrasonic signals (step S1004). Thus, the position of the photographing apparatus 400 is calculated from the position information (step S1006). The position information calculated in this way is used for the position information acquisition process in step S300.
The ultrasonic sensor type motion capture device is easier to move than the optical sensor type and can be moved easily, and can calculate position information with higher accuracy than the inertial sensor type.

  By the way, the composition processing in step S600 will be described below using a model room as an example.

  For example, the position information of the eyes of the HMD wearer is calculated from the position information calculated by the optical sensor processing. Then, a CG image corresponding to the calculated eye position is generated. Here, furniture and the like to be arranged in the model room are generated by CG (see FIG. 11A).

  Since the cameras 420 and 440 of the imaging device 400 provided in the front part of the image display device 300 are configured to match the position of the wearer's eyes, the images captured by the cameras 420 and 440 are actually eyes. It will match the image you see. Here, an actual indoor landscape is photographed (see FIG. 11B).

  Since the generated CG image and the actual image taken by the camera have the same resolution, a composite image can be obtained by superimposing each pixel. Thus, the image is displayed on the image display device 300 and the wearer feels it (see FIG. 11C).

  According to the above-described embodiment, the HMD wearer (sensation person) can move around and experience the virtual space without being positioned at a fixed point. In addition, since it is not necessary for the sensible person to be positioned at a fixed point, video sickness symptoms peculiar to 3D stereoscopic vision can be eliminated. Further, since the image is displayed by combining the real space and the virtual space, it is possible to avoid danger in use in the real world. In addition, it is extremely useful because you can experience a space that assumes various scenes.

  For example, a condominium room can be confirmed for each variation by using this system, even if the room is completely different from the property being sold, such as a conference room or reception room in a real estate dealer. Is. At that time, tables, chairs, etc. are installed in the actual conference room or reception room, and if these cannot be recognized, there is a possibility that the experienced person may collide. In order to avoid such a situation, this system can avoid the danger by synthesizing and displaying the image of the conference room or reception room photographed with the camera and the CG image of the apartment building generated by CG. become.

  Furthermore, according to the above embodiment, it is not necessary to prepare a conventional large-scale showroom or exhibition hall, and can be realized in a small-scale space without requiring a large-scale simulator facility, so that it is economical and practical. It is extremely useful.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 10 ... Image display type simulation service provision system 100 ... Simulation management apparatus 140 ... Storage part 140b ... Image information storage area 140c ... Shooting information storage area 140d ... Composite image information storage area 140e ... Measurement information storage area 150 ... Image information extraction part 160 ... three-dimensional virtual image information generation unit 170 ... composite image information generation unit 200 ... motion capture device 202 ... camera 204 ... marker 300 ... image display device 400 ... imaging device 420 ... right-eye camera 440 ... left-eye camera 500 ... operating device

Claims (4)

The image information storage means for storing various image information, the image information extraction means for extracting arbitrary image information from the image information storage means, and the tertiary with reference to the arbitrary image information extracted from the image information extraction means 3D virtual image information generating means for generating original virtual image information, and 3D virtual image information generated by the 3D virtual image information generating means and real image information are combined to generate composite image information. A simulation management device comprising synthetic image information generating means for performing,
A motion capture device capable of continuously measuring the position and orientation of a person or object in time and sending the measurement information to the simulation management device;
An image display device for outputting and displaying the composite image information sent from the simulation management device;
An image display type simulation service providing system comprising: an imaging device provided in the image display device, capable of capturing at least the front of the image display device, and capable of transmitting the captured image information to the simulation management device. ,
The composite image information generation means of the simulation management device recognizes the captured image information sent from the photographing device as the actual image information and combines it based on the measurement information sent from the motion capture device. Generate image information,
An image display type simulation service providing system, wherein the image display device outputs and displays the composite image information.   2. The image display type simulation service providing system according to claim 1, wherein the motion capture device has an optical sensor type, inertial sensor type, ultrasonic sensor type or magnetic sensor type motion capture function. Arbitrary image information is extracted from various image information created in advance or taken,
3D virtual image information is generated by referring to the extracted arbitrary image information,
Measure the position and posture of a person or object continuously with a motion capture device,
With reference to the measurement information from the motion capture device, the image information actually captured by the measurement target of the motion capture device is combined with the three-dimensional virtual image information to generate composite image information,
An image display type simulation service providing method characterized in that the composite image information is output and displayed. The composite image information is output and displayed on a display device attached to the measurement target,
4. The image display type simulation service providing method according to claim 3, wherein the photographing information provided in front of the photographing device is used as the image information actually photographed by the camera by the photographing device provided in the display device.