WO2022107294A1

WO2022107294A1 - Vr image space generation system

Info

Publication number: WO2022107294A1
Application number: PCT/JP2020/043282
Authority: WO
Inventors: 晃弘安藤; 智絵水上
Original assignee: 株式会社ハシラス
Priority date: 2020-11-19
Filing date: 2020-11-19
Publication date: 2022-05-27
Also published as: JPWO2022107294A1

Abstract

[Problem] To provide a VR image space generation system which makes it possible to give users a feeling of being in a virtual space and for the users to communicate with each other in the virtual space, collect a plurality of users in remote areas in one VR space, give the users the feeling of watching and experiencing various materials, including independent readable image/video data, generated in a general file format in the virtual space by projecting the materials into the VR image space as is, or switch between the pages of these materials, the materials themselves, and the scenes by operating both hands and fingers at the same time. [Solution] Provided is a VR image space generation system which generates a VR image for constructing a virtual space accessible by one or a plurality of users, comprising: a VR viewing device worn by the user; an image generation means for generating an initial image that can be displayed on the VR viewing device; a position information acquisition means for acquiring position information on each VR viewing device; a VR image generation means for generating a VR image by synthesizing an avatar image with an initial image generated by the image generation means, on the basis of each piece of position information acquired by the position information acquisition means; and an image output means for outputting a VR image to the VR viewing device.

Description

VR video space generation system

The present invention relates to a VR video space generation system for constructing a virtual space that can be used by a user by VR (Virtual Reality), and in particular, the user can freely move and operate in the generated VR video space. In addition, it is generated in a general file format including PNG, MP4, etc., and can be easily replaced without changing the VR video space generation system main body, readable independent, plane projection video, 3D video Or, by projecting image / video data (hereinafter referred to as "material") such as dome images as they are in the VR image space, the user can feel as if they are watching and experiencing them in the virtual space. It is possible to give, and the page of these materials, the material itself, and the scene switching, that is, the processing such as play / stop / frame advance / switching of the above material including image / video data can be performed simultaneously with both hands or a controller. The present invention relates to a VR video space generation system that enables a user to effectively experience presentations, amusements, exhibitions, trainings, etc. using various materials in a VR space by performing the operation of.

Conventionally, many technologies related to virtual reality (VR) that enable users to obtain various simulated experiences by watching videos have been developed and are used in various situations. Technology related to virtual reality has made progress and is being used in all fields, and many VR technologies have been developed that make it possible to get the feeling of being in the virtual space while staying indoors. Has been used.

Virtual reality refers to the technology for making users recognize and perceive in a virtual space created by a computer as if they were in that space, and the technology for creating such an environment and for that purpose. Various business tools and amusement devices using such techniques have been developed to give convenience and entertainment to users, and have the possibility of further use and utilization in the future.

As a system for creating a space using virtual reality technology, for example, Japanese Patent Application Laid-Open No. 2018-142722 exists. Here, as a technology related to a system that provides a support method that demonstrates superiority to competitors in presentations of buildings, etc., a 3D CG perspective, which is two panoramic images that are stereograms based on the original drawing data, is generated. , The 3D CG perspective is converted into a panoramic VR image, and the converted panoramic VR image is associated with each other to generate a presentation document related to the building, and then associated with the presentation document in which the presentation document is used. A technique relating to a system for displaying a panoramic VR image on a display device is disclosed.

It is possible that this technology will certainly make it possible to give presentations in VR space, but it is not possible to have an immersive VR experience in 6DOF, which is limited to viewing in 3DOF. When users participate in a presentation, there is a fatal problem that it is not possible to adopt a complicated structure such as communicating with each other. In addition, there is a problem that it is not possible to take a configuration in which a plurality of users in a remote place are gathered in one VR space. Furthermore, when creating a VR space other than a presentation such as a building, it is necessary to reconstruct it from scratch with dedicated software, and it is not possible to use an independent readable material as it is in a general format. In the VR space, there is a problem that the VR experience person cannot intuitively switch the page of the material, the material itself, or the scene.

Therefore, by allowing the user to freely move and operate in the VR video space, the user feels as if he / she is in the virtual space, and multiple users communicate with each other in the virtual space. It is possible to take a picture or collect multiple users in a remote place in one VR space, and it is possible to effectively project readable independent material into the VR space by a general file format, and in the VR space. In, it is desired to develop a VR video space generation system for generating a VR video space for constructing a virtual space in which a VR experience person can intuitively switch between the page of the material, the material itself, and the scene. rice field.

Japanese Unexamined Patent Publication No. 2018-147272

The present invention is a VR video space generation system for constructing a virtual space that can be used by users by virtual reality, and in particular, one or more users are free to use the generated VR video space in the generated VR video space. A readable, independent, planar projection image, or dome image that can be moved and operated, and is generated in a general file format and can be easily replaced without changing the system itself. By projecting the material consisting of images and videos such as these into the VR video space as it is, it is possible to give the user the feeling of actually watching and experiencing them in the virtual space. It is possible to communicate with each other in a virtual space, gather multiple users in a remote place in one VR space, and switch between these pages, materials themselves, and scenes, that is, image / video data. By performing processing such as play / stop / frame advance / switching of the above materials including by operating both hands at the same time or by operating the controller, presentations, amusements, exhibitions, training, etc. using various materials can be effectively performed in the VR space. It is an object of the present invention to provide a VR video space generation system that enables a user to experience it.

In order to achieve the above object, the VR video space generation system according to the present invention is a VR video space generation system that generates VR video for constructing a virtual space accessible to one or more users, and is a user. A VR viewing device including one or a plurality of VR viewing devices, a video generating means for generating an initial image that can be displayed on the VR viewing device, a position information acquiring means for acquiring the position information of each of the VR viewing devices, and the above. Based on each position information acquired by the position information acquisition means, it is generated by the VR image generation means that generates a VR image by synthesizing an avatar image with the initial image generated by the image generation means, and the VR image generation means. It is configured to include a video output means for outputting the VR video to a VR viewing device.

Further, the VR video space generation system numerically defines an area defining means that numerically defines a region that can be moved by the user as XYZ coordinates, and each position information acquired by the position information acquisition means numerically as XYZ coordinates. Each position information defined by the position definition means is introduced (applied) as a coordinate value into the position definition means and the area defined by the area definition means, and the area and each position information are introduced into the video. The configuration includes a mapping means associated with the initial image generated by the generation means.

Further, the VR video space generation system according to the present invention is a VR video space generation system that generates VR video for constructing a virtual space accessible to one or a plurality of users, and is one or a plurality of VR video space generation systems worn by the user. A VR viewing device consisting of a VR viewing device, a video generation means for generating an initial image that can be displayed on the VR viewing device, and an avatar image of each user who attaches the VR viewing device to the initial video generated by the video generation means. It is composed of a VR image generation means for generating a VR image and a video output means for outputting the VR image generated by the VR image generation means to a VR viewing device. Further, the VR video space generation system numerically defines the area defining means for numerically defining the area in which the user can move as XYZ coordinates and the position information of the user who wears the VR viewing device as XYZ coordinates. Each position information defined by the position definition means is introduced (applied) as a coordinate value into the position definition means and the area defined by the area definition means, and the area and each position information are introduced into the video. The configuration includes a mapping means associated with the initial image generated by the generation means.

Further, the position defining means acquires information relating to the position of a specific part of the body of the user wearing the VR viewing device, calculates information regarding the position of each other part of the body, and generates the VR image generation means. However, it is configured to draw based on the above information and generate a VR image to be displayed on the VR viewing device.

Further, the initial image is composed of an image selected from any of a plane projection image, a 3D image, and a dome image.
In addition, the initial video is created in a general format and is a plane projection containing readable, independent, user-instructed illustrations, 2D images, and textual information that can be easily replaced without changing other systems. Any one or a video, a presentation image consisting of a 2D moving image that operates according to the user's instruction, a 3D image including a CG stereoscopic model that operates according to the user's instruction, an all-sky image, or a part thereof. It is a configuration that includes a plurality.
Further, the initial video is configured to be composed of independent video data that can be read by the video generation means.

Further, the VR viewing device includes a sensor that detects the position and movement of the fingers of both hands of the user who wears the device, and the sensor detects the movement of the user's finger and then acquires the movement information as tracking information. At the same time, the VR video space generation system holds a plurality of action information consisting of finger movement information, and each of the action information is associated with a specific change process of the initial video, and the VR is described. The video space generation system is configured to perform change processing of the associated initial video when the tracking information matches any of the action information.

Further, the action information is composed of instruction information generated by simultaneously operating the fingers of both hands of the user.
Further, the initial video includes an image such as a presentation and a moving image, and the action information is associated with the progress and backward processing of the initial video.

Since the present invention has the configuration as described in detail above, it has the following effects.
1. 1. Since the position information of the VR viewing device is acquired and the avatar image is synthesized with the initial image so as to correspond to the position information, it is possible to reflect the current position of the user in the VR image space and display it. , Multiple people can experience and share the VR video space at the same time. Further, by performing the calibration process, even a user at a remote place can experience and share the VR video space in the same manner.
2. 2. Since the area defining means and the position defining means define the area of the VR video space and the position of the user in the area, it is possible to match the actually movable area with the area in the virtual space.

3. 3. Since the VR video space generation system can be configured without using location information acquisition means, even users who do not acquire the user's realistic current location information or continuously acquire the current location information can also virtualize in the virtual space. It becomes possible to participate in the VR space using the specific location information.
4. Since the position defining means can be configured to numerically define each position information of the user who wears the VR viewing device as XYZ coordinates without using the position information acquisition means, when the user is in a remote place in Japan or abroad. However, it will be possible for multiple people to experience and share the VR video space at the same time.

5. Since the position defining means is configured to acquire information related to the position of a specific part of the user's body and calculate the position of other parts of the body, which one is used for each VR viewing device based on such accurate position information. It is possible to draw and display in detail whether such an image should be displayed.
6. Since the initial image is composed of a plane projection image, a 3D image, a dome image, etc., it is possible to experience all types and types of materials in the VR space.

7. As initial images, 3D including illustrations that are switched according to the user's instructions, 2D images and a planar projection image that includes text information, a 2D moving image that operates according to the user's instructions, a 3D model of CG that operates according to the user's instructions, etc. A VR video space where multiple users can share presentations, amusements, exhibitions, and training that include various materials because the configuration includes one or more of video, all-sky video, or some of them. It will be possible to do it within.
8. As the initial video, the video that is switched according to the user's instruction is displayed at a fixed position on the VR viewing device, so these materials are shared by multiple users as images / videos for presentations, amusements, exhibitions, and training. It becomes possible to display in space.

9. One VR video space generation system because the initial video is configured so that the video generation means conforms to a general file format and is generated from readable independent material that can be easily replaced without changing the system body. It is possible to display various VR video spaces and let the user experience them.
10. Since the sensor acquires the movement of the user's finger as tracking information and contrasts it with the preset action information, the user moves the finger to give a processing instruction such as the movement / change of the initial image. It becomes possible.

11. Since the action information is composed of instruction information generated by simultaneously moving the fingers of both hands of the user, it reacts only to the processing instruction by moving both hands, and the risk of issuing an erroneous processing instruction can be reduced.
12. Since the action information is configured to include instruction information corresponding to the progress and backward processing of the initial video, presentations, amusements, exhibitions, and trainings that involve sequential changes of images in the VR video space can be performed only with the movements of the user's hands and fingers. It is possible to proceed with.

Hereinafter, the VR video space generation system according to the present invention will be described in detail based on the examples shown in the drawings. FIG. 1a is a schematic diagram of a VR video space generation system according to the present invention, and FIG. 1b is a schematic diagram of a VR video space generation system provided with an external computer. FIG. 2 is a schematic view showing a display example of a VR image, and FIG. 3 is a diagram showing area / position information. FIG. 4a is a schematic diagram of a VR video space generation system provided to a remote user, and FIG. 4b is a schematic diagram of a VR video space generation system provided to a remote user provided with an external computer. FIG. 5 is a schematic diagram of a VR video space generation system that performs switching processing of video materials and scenes.

As shown in FIGS. 1a and 1b, the VR video space generation system 1 according to the present invention includes a VR viewing device 100, a video generation means 200, a position information acquisition means 300, a VR video generation means 400, and a video output. It is a system for constructing a virtual space using virtual reality (VR) technology, which comprises means 500 and means. It generates a VR video for constructing a virtual space that can be accessed by one or more users, and one or more users can freely move and operate in the generated VR video space, and it is also virtual. It is a system that makes it possible to give the user the feeling of being in space. The VR video space generation system according to the present invention makes it possible to realize user communication in a virtual space, presentation, experience of attraction, exhibition, training, and the like.

In addition to virtual reality, the "VR" image displayed in the present invention also includes AR (Augmented Reality), MR (Mixed Reality), and SR (Substitutional Reality). It is a concept that includes.

The VR viewing device 100 is a device composed of one or a plurality of devices worn by a user, and is a device for playing / displaying a video used for playing a VR video 20 generated based on the initial video 10. .. The device of this embodiment mainly consists of a goggle-type projection device, but is not limited to this, and may be worn by a user, for example, a retinal projection method for directly forming and projecting an image on the retina. Any projection device that is possible and gives a feeling of immersion in the image can be appropriately selected and used.

In this embodiment, as shown in FIG. 1a, the VR viewing device 100 is equipped with a storage means 610 including an arithmetic unit (not shown) and an arbitrary storage medium, and the storage means 610 is equipped with an initial image 10 And the VR image 20 generated from this is stored. Further, the VR viewing device 100 is a projection device for enabling viewing of VR images, and in the present embodiment, it is mainly configured as a goggle type projection device, but the present invention is not limited to this. However, it is possible to use a projection device having another structure. Further, the image displayed on the VR viewing device 100 is mainly composed of an image that can be viewed in whole or in part at 360 degrees, and (1) a plane projection version (text, illustration and / or 2D image). It is possible to configure either (2) a 3D image or (3) a three-dimensional spherical dome image.

As shown in FIG. 1b, the VR viewing device 100 can be configured to be wirelessly or wiredly connected to the computer 600 or the cloud. In the case of this configuration, the computer 600 or the cloud is equipped with a storage means 610 composed of an arithmetic unit (not shown) and an arbitrary storage medium, and the storage means 610 is equipped with an initial image 10 and a VR generated from the initial image 10. It is a configuration in which the video 20 is stored, and is also a configuration used for acquiring, calculating, and managing the position information and the like of the VR viewing device 100. At this time, the initial video 10 and the VR video 20 generated from the initial video 10 can be managed, calculated, and held by the VR viewing device 100.

The image generation means 200 is a device that generates an initial image 10 that can be displayed on the VR viewing device 100. In this embodiment, the initial image 10 is composed of materials (images, moving images, etc.) such as a plane projection image, a 3D image, or a dome image, and can be, for example, a 360-degree all-sky image. Yes, it can be a flat image, it can be any of 3 dof and 6 dof, and it can be an image with disparity. The image generation means 200 initially changes these basic image data into a flat output image (360 degree image or the like is also a kind of flat output image) which is a format that can be displayed on the VR viewing device 100. Generate video 10.

In this embodiment, the image generation means 200 is configured such that the arithmetic unit of the VR viewing device 100 performs arithmetic processing based on various data stored in the storage medium 610 incorporated in the VR viewing device 100 to generate the initial image 10. However, the format is not limited to this, and for example, a computer 600 or a cloud is provided outside, and an arithmetic unit performs arithmetic processing based on various data stored in the storage medium 610 on the computer 600 or the cloud. It is also possible to have a configuration that generates the initial video 10. Further, in the present embodiment, the video generation means 200 is configured to include software that the arithmetic unit reads from the storage medium 610 and processes the video, but the video generation means 200 is not limited thereto.

The position information acquisition means 300 is a means for acquiring each position information P of the VR viewing device 100. The user of the VR video space generation system 1 according to the present invention is equipped with the VR viewing device 100, and grasps the current position information regarding a specific part of the body of each user by each VR viewing device or an external sensor. The position of each other part of the body is calculated based on the position information P, the VR image generation means draws based on the information, and the avatar V of each user is displayed on the VR viewing device 100. Can be displayed.

In this embodiment, the position information acquisition means 300 is equipped in each VR viewing device 100, and the position information P is calculated by the camera of the VR viewing device 100, but the present invention is not limited to this. It is also possible to install a sensor on the outside and use other technologies such as tracking by laser irradiation. Further, in the present embodiment, the position information acquisition means 300 is configured to include software that is read from the storage medium 610 by the arithmetic unit incorporated in the VR viewing device 100 and processed, but is not limited thereto.

Further, as shown in FIGS. 1a and 1b, the position information acquisition means 300 can include a configuration for acquiring, calculating and managing the position information of the user in a remote place. In this case, the position information of the user in a remote place is acquired, the position information acquisition means 300 performs the calibration process, and the position information is assigned to the VR space in which another user exists so as to exist in that space. Perform video processing. When moving in a remote place, the calibration process is performed in the same manner, and the image processing is performed so as to move in the same way in the VR space. With this configuration, one or more users in remote areas can experience and share the VR image space at the same time via the avatar image V, even if the users are in remote positions. , It is possible to freely move and operate in the generated VR image space, and by gathering multiple users in a remote place in one VR space and communicating with each other, the experience in the space can be experienced. It became possible to share.

In this case, the arithmetic unit shifts the center of the calibrated world of each user (for example, if it moves a few centimeters to the right and a few centimeters to the left in the real world, it moves from the center of the world in VR space to the right. (Move a few centimeters, a few centimeters to the left), or when the user overlaps with another user's avatar, landscape, or object, it is possible to make it transparent (thin), etc. It is also possible to incorporate a configuration that performs video processing that enhances the user's immersive feeling in VR.

The above-mentioned calibration process may be configured by the position defining means 320 described later. With this configuration, each position information P of the VR viewing device 100 can be numerically defined as the coordinates consisting of the XYZ axes.

The VR image generation means 400 is a means for synthesizing the avatar image V with the initial image 10 generated by the image generation means 200 to generate the VR image 20. The VR image generation means 400 specifies a position and an orientation (or a posture) to be arranged in the initial image 10 based on the position information P of each VR viewing device 100 acquired by the position information acquisition unit 300. Then, a VR image 20 is generated by synthesizing the avatar image V of each user with the initial image 10. As a result, a VR image 20 in a state in which the realistic position of each user wearing the VR viewing device 100 is reflected in the initial image 10 is generated, and as shown in FIG. 2, each user is a VR image. It is possible to configure a state of participating in a virtual space consisting of 20. That is, the user can view the image of the virtual space displayed by another user and includes the image of the virtual space including the avatar of another person that can be seen from the position in the virtual space where he / she is, and he / she also enters the virtual space. It becomes possible to obtain a feeling as if it were, and the immersive feeling in the VR image 20 is enhanced.

In the present embodiment, the VR image generation means 400 is configured to include software that is read from the storage medium 610 and processed by the arithmetic unit incorporated in the VR viewing device 100, but is limited to this. However, for example, a computer 600 or a cloud may be provided externally, and the VR image generation means 400 installed in the computer 600 or on the cloud may perform arithmetic processing to generate the VR image 20. Further, it is also possible to perform each process after migrating only a part of each function constituting the VR video space generation system according to the present invention to the cloud. Further, the computer 600 or the cloud can acquire and manage the location information P, and the VR video 20 can be configured to be generated by the VR viewing device 100 that has acquired the information, and VR can be generated in any other manner. It is possible to select a configuration for generating the moving image 20.

The video output means 500 is a means for outputting the VR video 20 generated by the VR video generation means 400, in which the avatar video V of each user is synthesized with the initial video 10, to each VR viewing device 100. Since the position and orientation of each VR viewing device 100 are different, the images output to the VR viewing device 100 will be different (see FIG. 2). In the present embodiment, the video output means 500 is configured by software that is read from the storage medium 610 by the arithmetic unit incorporated in the VR viewing device 100 and processed, but is not limited to this configuration, for example. It is also possible to provide an external computer 600 and have the video output means 500 equipped in the computer 600 perform arithmetic processing to output the VR video 20 to the VR viewing device 100 by wire or wirelessly. It is also possible to perform the processing on the cloud.

Next, the details of the embodiment of the VR video space generation system 1 will be described. As shown in FIG. 1a, the VR video space generation system 1 according to the present invention prepares in advance after the arithmetic unit analyzes the video of the space acquired by the camera mounted on the VR viewing device 100 worn by the user. It is configured to project the spatial image onto the VR viewing device 100 in association with the image of the VR space. Further, as described above, the camera mounted on each VR viewing device 100 acquires the depth information, creates a mesh model that virtually exists according to the viewpoint position, and acquires and calculates the position information P. Then, a flat output image (360 degree image or the like is also a kind of flat output image) that can be seen from the angle is generated and displayed.

With this configuration, the virtual space that each user sees through the VR viewing device 100 corresponds to the scenery seen in the real space including other users, and becomes the VR image 20 that combines the image of the virtual space with the avatar image V. ..

That is, with this configuration, it is possible to give the user the feeling of being in the virtual space, and the real world and the virtual world are mixed, so that the generated VR image space is one or more. It has become possible for multiple users to freely move and operate, and for multiple users to communicate with each other in the virtual space.

As an embodiment of the present invention, an embodiment in the case where the VR viewing device 100 is configured to centrally manage various information will be described. In this case, as shown in FIG. 1a, the VR video space generation system 1 includes an area defining means 310, a position defining means 320, and an associating means 410 in the VR viewing device 100.

The area defining means 310 is, for example, as shown in FIG. 3, a means for defining a user-movable area F in the virtual space generated by the VR video space generation system 1. In this embodiment, for example, it is conceivable to numerically define it as a three-dimensional coordinate consisting of the XYZ axes. At this time, the area F coincides with a certain area provided in the real space. The area is the actual space in which the user arbitrarily moves or sits.

The area defining means 310 defines a virtual space that matches this space as the area F. The region F is numerically defined as, for example, coordinates consisting of XYZ axes, but is not limited to this, and it is possible to configure the VR space to be grasped and managed by using other region management means. ..

In this embodiment, the area defining means 310 is composed of software that is read from the storage medium 610 by the arithmetic unit of the VR viewing device 100 and processed, and in particular, is configured to include a software module or the like incorporated in the position information acquisition means 300. However, the present invention is not limited to this, and it may be configured to be processed on independent software, software embedded in separately provided hardware, or the cloud.

The position defining means 320 is a means for numerically defining each position information P of the VR viewing device 100 as coordinates having XYZ axes. When the X value, Y value and / or Z value of the user's position information exceeds the maximum value, an exception handling such as displaying a warning message on the VR viewing device 100 can be considered.

In this embodiment, the position defining means 320 is composed of software that is read from the storage medium 610 by the arithmetic unit of the VR viewing device 100 and processed, and in particular, is composed of a software module or the like incorporated in the position information acquiring means 300. However, the present invention is not limited to this, and it is of course possible to use independent software, software embedded in separately provided hardware, or a configuration for processing on the cloud.

The associating means 410 is a means for introducing and applying one or a plurality of position information Ps to the area F and then associating them with the initial image 10. Specifically, the associating means 410 introduces (applies) the position information P defined by the position defining means 320 as a coordinate value to the area F defined by the area defining means 310. Then, the area F and each position information P are associated with the initial image 10 generated by the image generation means 200. Specifically, for example, in this embodiment, a configuration using a technique related to inverse kinematics is possible. The details of this mapping will be described later.

For example, when there are a plurality of users wearing the VR viewing device 100 in a real space, each position information P exists in the real space in the area F defined to match the real space. Defined to match the user's position. Then, in order to apply each of the information to the initial image 10, the avatar image V of each user is displayed at a position where the avatar image V of each user is actually present in the initial image 10.

In this embodiment, the associating means 410 is composed of software that is read from the storage medium 610 by the arithmetic unit of the VR viewing device 100 and processed, and in particular, is composed of a software module or the like incorporated in the VR video generating means 400. However, the present invention is not limited to this, and it is of course possible to use independent software, software embedded in separately provided hardware, or a configuration for processing on the cloud.

As an embodiment of the present invention, it is possible to have a configuration for managing various information on a computer 600 or a cloud. In this case, as shown in FIG. 1b, the VR video space generation system 1 is configured to include an area defining means 310, a position defining means 320, and an associating means 410 in the computer 600.

As yet another embodiment, it is also possible to manage the position information P by the position defining means 320 on the computer 600 or the cloud, and perform all other arithmetic processing on the VR viewing device 100.

As another embodiment of the present invention, the VR video space generation system 2 can be configured without using the position information acquisition means 300 as shown in FIGS. 4a and 4b. That is, the configuration is such that the current and current realistic position of each VR viewing device 100 worn by each user is not acquired, and the position defining means 320 calculates the user's virtual position information P in the virtual space and XYZ axes. The VR image generation means 400 uses the position information P to numerically define the coordinates as the coordinates consisting of the coordinates, and the VR image generation means 400 attaches the VR viewing device 100 to the initial image 10 generated by the image generation means 200 as an avatar image of each user. V is combined to generate a VR image 20. At this time, it is conceivable to set an arbitrary position so that the avatars of users in remote areas who are not there do not overlap each other.

As an example according to the present invention, as shown in FIG. 4a, a configuration in which the VR viewing device 100 centrally manages various types of information will be described. In this configuration, the position defining means 320 arbitrarily sets the position information P on the assumption that the position information P is at an arbitrary place in the real space while communicating with the other VR viewing device 100. It is numerically defined as the coordinates consisting of the XYZ axes. With this configuration, one or more users in remote areas can experience and share the VR image space at the same time via the avatar image V, even if the users are in remote positions. , It is possible to freely move and operate in the generated VR image space, and by gathering multiple users in a remote place in one VR space and communicating with each other, the experience in the space can be experienced. It will be possible to share.

In this case, the arithmetic unit performs a process of shifting the center of the calibrated world of each user, and a process of making (thinning) the avatar, landscape, or object of another user when the user overlaps with the user. In addition, it is also possible to incorporate a configuration for performing video processing that enhances the user's immersive feeling in VR. The calibration process may be performed by the position defining means 320. With this configuration, each position information P of the VR viewing device 100 can be numerically defined as the coordinates consisting of the XYZ axes.

Further, as another embodiment, as shown in FIG. 4b, it is possible to provide a configuration in which an external computer 600 is provided or a cloud is used to centrally manage various types of information. In this configuration, it is assumed that the computer 600 or the position defining means 320 on the cloud communicates with each VR viewing device 100 and then the position information P is located at an arbitrary place in the real space. It is configured to be arbitrarily set and numerically defined as the coordinates consisting of the XYZ axes.

The position defining means 320 according to the present invention is configured to acquire information related to the position of a specific part of the body of the user wearing the VR viewing device 100 and to calculate the position of each other part of the body. .. The associating means 410 associates the area F with the position information P including these information with the initial image 10 generated by the image generating means 200. The VR image generation means 400 draws based on these information and generates a VR image 20 to be displayed on the VR viewing device 100. With this configuration, the VR image 20 displayed on the VR viewing device 100 worn by the user becomes accurate according to the movement of the user, and the user is immersed in the VR image 20 as if the virtual space is real. It is possible to get a feeling.

In this embodiment, the technique related to inverse kinematics can be used to generate the avatar video V. Inverse kinematics is a technique for calculating the position and rotation angle of a higher-level object by designating the target position of a lower-level object in an object having a hierarchical structure, and is used for calculating the operation of the avatar video V. In this embodiment, the VR viewing device 100 or the computer 600 incorporates an arithmetic unit, and the position defining means 320 acquires information related to the position of a specific part of the user's body. The arithmetic unit performs arithmetic processing on this information using the inverse kinematics technique, identifies the positions of other parts of the body, and the mapping means 410 performs the mapping processing, and then the VR viewing device 100. It is a configuration to generate a VR image 20 to be displayed in.

With the above configuration, some users experience the virtual space superimposed on the real space, and other users participate in the virtual space from the real space provided in a remote place. It is possible to make a configuration that does.

For example, in a situation where a plurality of users watch a presentation in a virtual space, the user in the presentation venue watches the presentation by the VR viewing device 100 in the virtual space superimposed on the real space, and is in a remote place. Similarly, the user also watches the presentation by the VR viewing device 100 in the same virtual space superimposed on the real space. Since the avatar image V is displayed on the VR image 20 for the user in the remote place, the user in the presentation venue can recognize that the user in the remote place is also in the same place. On the other hand, a user in a remote place can get the experience of being in the presentation venue through the VR viewing device 100.

In this embodiment, the VR video space generation systems 1 and 2 have a configuration corresponding to 6 dof or 3 dof. The 3DOF is a VR viewing device 100 corresponding to three movements of the X-axis, the Y-axis, and the Z-axis, and senses the rotation and tilt of the head equipped with the VR viewing device 100. Further, the 6 dof has a configuration corresponding to 6 movements in which movements in the X-axis, Y-axis, and Z-axis directions are added in addition to the movements of 3 dof. By supporting 6DOF, it is possible to obtain a high immersive feeling in the virtual space when the user wearing the VR viewing device 100 makes any movement. In addition, a sufficient immersive feeling can be obtained even if it is compatible with 3DOF.

In particular, by letting the user experience 6DOF and then showing the existing 3DOF image, the user can enjoy the 3DOF image while maintaining the immersive feeling of 6DOF. That is, it is possible to use the illusion of the experience of 6DOF to show a large number of existing 3DOF images while maintaining the immersive feeling of 6DOF.

Also, for example, an image equivalent to what is shown in the range that can be seen in the captured 3dof image is CG generated in 6dof, and then the 3dof is guided to the position where the 3dof was photographed in the 6dof experience, and the immersive feeling can be maintained. It is possible to replace 6dof with 3dof in a suitable position and show this continuously to show 3dof while maintaining the immersive feeling of 6dof.

Also, in this embodiment, it is possible to share voice. That is, the voice uttered by the user, the sound flowing in the VR image, the music, and the like are simultaneously reproduced by the speakers and the like provided in the VR viewing device 100 of all the users. This makes it possible for all users who participate in the same VR space to have an experience of sharing the space.

That is, the VR video space generation systems 1 and 2 according to the present invention give each user the feeling of being in a virtual space, allowing the users to communicate with each other in the virtual space, and a plurality of users in remote locations. Can be collected in one VR space, and it is possible to experience presentations, amusements, exhibitions, training, etc. in the VR space.

The VR video space generation system 1 according to the present invention can be operated only locally without using an external network such as the Internet. That is, it is possible to give a presentation or the like using the VR video space generation system 1 even in a situation where there is no network environment that can be connected to the outside such as the Internet.

In this embodiment, the initial video 10 is generated in a general file format including PNG, MP4, etc., and can be easily replaced without changing the main body of the VR video space generation system, and is a readable independent plane. It is composed of projected images, 3D images, dome images, and the like. With this configuration, it has become possible for users who use the VR video space generation systems 1 and 2 to experience materials of all formats and types in the VR space.

Further, the initial video 10 is generated in a general file format, and can be easily replaced without changing the VR video space generation system main body. It is a readable independent illustration, 2D image and characters that are switched according to the user's instruction. It is possible to configure a configuration including a plane projection image including information and a plane projection image composed of a 2D moving image that operates according to a user's instruction. Further, it is possible to configure the configuration to include a stereoscopic model of CG that operates according to the user's instruction, or to include a spherical image that operates according to the user's instruction or a part of the image.

Each of these images is generated by a general file format such as PNG or MP4, and is fixedly embedded in the VR space (in the initial image 10) or always displayed in the direction in which the user turns his face. It is an independent image that can be displayed or any display method can be selected. Moreover, since it can be embedded arbitrarily, it can be easily replaced without changing other systems.

With this configuration, the user can view the plane projection image, 3D image, and spherical image embedded in the virtual space, in order to introduce the structure and specifications of automobiles, real estate, etc. Presentations, amusements, exhibitions, and training can now be easily experienced and shared by users. In addition, it has become possible for multiple users to experience and share various attractions.

Further, the initial video 10 can be configured to be composed of independent video data that can be read by the video generation means 200. With this configuration, the video generation means 200 can read an arbitrary initial video 10 from various existing materials including image / video data generated in a general file format, and one VR video space. It has become possible to display an arbitrary VR video space desired by the generation system and let the user experience it.

For example, in the VR video space generation systems 1 and 2 according to the present invention, presentation materials created in another format are inserted at a fixed position and displayed in the VR space in which the user participates (for example, the VR space is composed of an auditorium). , Presentation materials are fixed and embedded behind the lecture hall). As a result, the user can share the presentation in the VR space, and the presentation material can be easily replaced.

Also, for example, it is possible to include an image in the VR space. With this configuration, for example, when experiencing exercise training involving physicality with the VR video space generation system 1 and 2 according to the present invention, the video of the instructor embedded in the virtual space is always displayed in front of the user. It is possible to provide a VR video space generation system 1 and 2 that can always display a projected image at a specific position (for example, in front of the eyes) regardless of the user's posture and is highly convenient for the user. Is now possible.

As shown in FIGS. 3 and 5, the VR viewing device 100 includes a sensor 110 in this embodiment. The sensor 110 is a sensor that detects the position and movement of the fingers of both hands of the user wearing the VR viewing device 100. The sensor 110 detects the movements of the hands and fingers of the user wearing the VR viewing device 100, and then acquires the information related to the movements as the tracking information T. The tracking information T is information related to the movements of the fingers of both hands of the user, and is composed of information related to a series of movements of the hands and fingers within a certain period of time. The sensor 100 may be installed in a device other than the VR viewing device 100 to detect the positions and movements of the fingers of both hands of the user who wears the VR viewing device 100.

For example, the path of a hand or finger from a certain point to a certain point, such as the movement of drawing a single line vertically or horizontally or the movement of drawing a figure of eight, is traced for each of the left and right hands and acquired and saved as tracking information T. do.

In addition, the VR video space generation systems 1 and 2 hold the action information A. The action information A is information that tracks a series of finger movements within a certain period of time, and in this embodiment, a plurality of patterns of action information A are retained. In this embodiment, the action information A is composed of data stored in the storage medium 610 of the VR viewing device 100 or the computer 600.

In this embodiment, each of the plurality of action information A is configured to be associated with a specific change process of the initial image 10. Further, when the acquired tracking information T matches any of the stored action information A, the change processing of the associated initial video 10 is performed.

For example, assume that the initial video 10 includes a presentation image and the like, and the action information A is associated with the progress and backward processing of the initial video. At this time, the action of moving the finger from right to left is registered and held as the action information A, and the action information A changes the image so as to sequentially switch the presentation image displayed in a part of the initial image 10. Assuming that it is associated with the processing, when the user wearing the VR viewing device 100 moves the finger, the sensor 110 detects the position and movement of the finger, and the information related to the operation is used as the tracking information T. get. The VR video space generation systems 1 and 2 perform processing for comparing the tracking information T that traces the movement of the finger and the action information A, and when it is determined that they are the same, the associated presentation video is sequentially switched. It is a configuration that processes images.

In addition, it is possible to register and retain all movements as action information A, such as the movement of making a circle by contacting the thumb with the index finger or middle finger. With this configuration, when one of the users participating in the virtual space gives a presentation, attraction, exhibition, or training, the presenter can switch the video material or scene sequentially by simply moving his or her finger. It has become possible to provide smoother and more appealing presentations, attractions, exhibitions and training.

In this embodiment, the action information A is configured to include instruction information generated by simultaneously operating the fingers of both hands of the user. For example, it is assumed that the initial video 10 includes a presentation video composed of a plurality of images, and the action information A is associated with the progress and backward processing of the presentation video. At this time, the action of contacting the thumb and the index finger of both hands or the thumb and the middle finger is registered as the action information A, and this action is made to correspond to the progress (or backward) processing of the presentation image. When a user giving a presentation in a virtual space touches the thumb and index finger of both hands, or the thumb and middle finger, the VR video space generation systems 1 and 2 detect tracking information T that traces the movement of the finger. At the same time as the acquisition, comparison with the action information A is performed, it is confirmed that they match, and the progress processing of the presentation video associated with the action information A is performed.

By using the tracking information T consisting of only one hand, it is possible to instruct the change processing of the initial image 10 simply and smartly, but if the user's operation is ambiguous, there is a big problem that it leads to a malfunction. there were. By adopting this configuration in which both hands are used at the same time, it is possible to prevent / suppress, for example, a malfunction in the screen switching process at the time of presentation and an unintended change in the image by the user.

With the above configuration, the user can feel as if he / she is in the virtual space, communicate with each other in the virtual space, and multiple users in remote areas gather in one VR space. By projecting various materials, including independent images and videos that can be chilled, generated in a general file format, and readable, into the VR video space as they are, you can watch them in the virtual space. By giving the user the feeling of experiencing, and switching between the pages of these materials, the materials themselves, and the scenes by operating both hands at the same time, presentations, amusements, exhibitions, training, etc. in VR space, etc. It became possible to experience and share with a very high sense of reality.

Schematic diagram of the VR video space generation system according to the present invention Schematic diagram of a VR video space generation system equipped with an external computer Schematic diagram showing a display example of a VR image Diagram showing area / location information Schematic diagram of VR video space generation system provided to users in remote areas Schematic diagram of a VR video space generation system provided to users in remote areas equipped with an external computer Schematic diagram of a VR video space generation system that switches video materials and scenes

1.2 VR video space generation system 10 Initial video 20 VR video 100 VR viewing device 110 Sensor 200 Video generation means 300 Position information acquisition means 310 Area definition means 320 Position definition means 400 VR video generation means 410 Correspondence means 500 Video output means 600 Computer 610 Storage means A Action information F Area P Location information T Tracking information V Avatar

Claims

The VR video space generation system (1) that generates VR video for constructing a virtual space accessible to one or more users is
A VR viewing device (100) consisting of one or more worn by the user,
An image generation means (200) that generates an initial image (10) that can be displayed on the VR viewing device, and
The position information acquisition means (300) for acquiring the position information (P) of each of the VR viewing devices, and
A VR image generation means (400) that generates a VR image (20) by synthesizing an avatar image (V) with an initial image generated by the image generation means based on each position information acquired by the position information acquisition means. When,
A VR video space generation system comprising: a video output means (500) for outputting a VR video generated by the VR video generation means to a VR viewing device.
The VR video space generation system (1) is
Area defining means (310) that numerically defines the area (F) that can be moved by the user as XYZ coordinates, and
The position defining means (320) that numerically defines each position information (P) acquired by the position information acquiring means as XYZ coordinates, and
Each position information (P) defined by the position definition means is introduced (applied) as a coordinate value into the area (F) defined by the area definition means, and the area and each position information are introduced into the video. The VR video space generation system according to claim 1, further comprising an associating means (410) associated with the initial video (10) generated by the generation means.
The VR video space generation system (2) that generates VR video for constructing a virtual space accessible to one or more users is
A VR viewing device (100) consisting of one or more worn by the user,
An image generation means (200) that generates an initial image (10) that can be displayed on the VR viewing device, and
A VR image generation means (400) that generates a VR image (20) by synthesizing an avatar image (V) of each user who attaches a VR viewing device to the initial image generated by the image generation means.
A VR video space generation system comprising: a video output means (500) for outputting a VR video generated by the VR video generation means to a VR viewing device.
The VR video space generation system (2) is
Area defining means (310) that numerically defines the area (F) that can be moved by the user as XYZ coordinates, and
A position defining means (320) that numerically defines each position information (P) of the user who wears the VR viewing device as XYZ coordinates, and
Each position information (P) defined by the position definition means is introduced (applied) as a coordinate value into the area (F) defined by the area definition means, and the area and each position information are introduced into the video. The VR video space generation system according to claim 3, further comprising an associating means (410) associated with the initial video (10) generated by the generation means.
The position defining means (320) acquires information related to the position of a specific part of the body of the user who wears the VR viewing device (100), and then calculates the position of each other part of the body.
Any of claims 1 to 4, wherein the VR image generation means (400) generates a VR image (20) that is drawn based on the information and displayed on the VR viewing device (100). The VR video space generation system described in C.
The VR image space generation according to any one of claims 1 to 5, wherein the initial image (10) is composed of an image selected from any of a plane projection image, a 3D image, and a dome image. system.
The initial video (10) is an illustration that is switched according to the user's instruction, a plane projection video that includes a 2D image and text information, a plane projection video that consists of a 2D video that operates according to the user's instruction, and a CG stereoscopic model that operates according to the user's instruction. 3. The image according to any one of claims 1 to 6, wherein the image comprises any one or a plurality of a 3D image, a whole celestial sphere image operated according to a user's instruction, or a part of the image. VR video space generation system.
The initial video (10) is an illustration that is switched according to the user's instruction, a plane projection video that includes a 2D image and text information, a plane projection video that consists of a 2D video that operates according to the user's instruction, and a CG stereoscopic model that operates according to the user's instruction. 3. The claim is characterized in that any one or more of a 3D image, an all-sky image operated according to a user's instruction, or a part of the image is displayed at a fixed position of the VR viewing device (100). The VR image space generation system according to any one of 1 to 7.
The initial video (10) is generated in a general file format, and can be easily replaced without changing the VR video space generation system main body, and is independent video data that can be read by the video generation means (200). The VR video space generation system according to any one of claims 1 to 8, further comprising.
The VR viewing device (100) includes a sensor (110) that detects the position and movement of the fingers of both hands of the user who wears the device, and the sensor detects the movement of the user's finger and then outputs the operation information. Acquired as tracking information (T) and
The VR video space generation system holds a plurality of action information (A) composed of finger movement information, and each of the action information is associated with a specific change process of the initial video (10). ,
The VR image space generation system is characterized in that when the tracking information (T) matches any of the action information (A), the associated initial image change processing is performed. The VR video space generation system according to any one of claims 9.
The VR video space generation system according to claim 10, wherein the action information (A) consists of instruction information generated by simultaneously operating the fingers of both hands of the user.
The initial image (10) includes all or a part of a plane projection image, a 3D image, or a dome image, and the action information (A) corresponds to the progress and / or backward processing of the initial image (10). The VR video space generation system according to claim 10 or 11.