JP2016045815A - Virtual reality presentation system, virtual reality presentation device, and virtual reality presentation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem in which: a virtual reality presentation system has been sought, which enables a user to mistakenly recognize a virtual reality world as a real world and become absorbed in the virtual reality world.SOLUTION: A virtual reality presentation system of the present invention includes: user state detection means for detecting the state of a user; video creation means for creating a video for presenting virtual reality on the basis of the state of the user; video display means for displaying the created video for presenting virtual reality; determination means for determining if the state of the user detected by the user state detection means satisfies a predetermined standard; and video display control means for, when determined that the state of the user satisfies the standard, controlling the video display means to display the video for presenting virtual reality.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a virtual reality presentation system, a virtual reality presentation device, a virtual reality presentation method, a virtual reality service providing system, a virtual reality service providing server, a virtual reality service providing method, an information processing system, an information processing server, an information processing method, and information. The present invention relates to a processing program and a head mounted display device.

In recent years, various electronic devices have been developed as devices for displaying images in addition to conventional stationary displays such as televisions, and are referred to as mobile phone terminals, tablet terminals, and head mounted displays (HMDs). Electronic devices have already been developed.

  Prior art document 1 discloses a virtual reality presentation device for displaying CG (Computer Graphics) on a real landscape and displaying it in HMD.

JP 2006-338163 A

In the conventional display, the video content that the user wants to view is selected and displayed on the display. For example, an application screen that the user wants to use can be selected by selecting a TV channel and displaying content broadcast on the channel, or by inputting instructions to the computer using an operation input device such as a mouse or a touch panel. On the display.

In these conventional displays, after receiving an input operation from the user, the content data designated in a short time can be read and displayed on the screen, which is a good display having a good response. This indicates that the conventional display is merely a display device that displays video content specified by the user.

An HMD device is a kind of display, but when used as a virtual reality presentation device that presents virtual reality, a function different from a simple display device is required. The criteria for whether or not it is an excellent virtual reality presentation device is not how quickly images can be displayed, but how it can eliminate the boundary between reality (virtual) and virtual reality (virtual) perceived by the user. is there. That is, it can be said that the device is so good that it is impossible to determine whether the user is currently viewing a real landscape or a virtual landscape presented by the virtual reality presentation device.

In order to make a virtual landscape be recognized as a real landscape, or to make the virtual scene an illusion that the user is a real landscape in another expression, first, the behavior of the image displayed on the HMD device is a reality. Must be consistent with the scenery's behavior. If the movement of the user's head causes a discrepancy in the behavior between the background scenery and the displayed video, the user can confirm that part of the currently visible landscape is a virtual video. It will be recognized. Therefore, the HMD device needs to be controlled to change the video displayed in the video display area in conjunction with the movement of the user's line of sight.

However, there is another factor that makes it difficult to recognize a virtual video as a real video. That is, there is a discontinuity in which the scenery actually sensed by the user is switched from real to virtual at the timing when video display is started in the video display area of the HMD device.

In FIG. 39, the transition of the level which recognizes that it is a virtual reality is shown in the user whose virtual reality is shown by the virtual reality presentation apparatus. In FIG. 39, when the threshold value Th0 is below, the user recognizes that the current view is virtual, and when the threshold value Th0 is exceeded, the user is viewing the current view in real. It shows that there is an illusion.

The upper diagram of FIG. 39 shows an example in which the user recognizes that it is virtual at time T1 in a state where the virtual reality presented by the user is erroneously recognized as real. For example, when the user's head has moved greatly at time T1, the consistency between the image displayed in the image display area and the background scenery is lost, and part or all of the scenery that the user has viewed so far has been lost. Recognize that it was virtual.

On the other hand, the lower part of FIG. 39 shows the degree of virtual recognition of the user when presentation of virtual reality is started at the time T0. In FIG. 39, the user clearly recognizes that the display of the virtual reality object has started in the display area at time T0, and that the video currently being viewed is not real but virtual. It has recognized.

Thus, at the stage of starting the presentation of virtual reality, discontinuity from real to virtual occurs, and the user recognizes that a virtual video is being shown. For example, when presenting virtual reality by displaying a CG dog so that it blends into the scenery, if the dog suddenly appears in the field of view, the discontinuity, that is, the moment that cannot occur physically in the real world This causes the user to recognize that the discomfort due to the phenomenon of movement is virtual.

If the user clearly recognizes that it is virtual at the start of presentation of virtual reality, the recognition that the displayed object is virtual will not be dispelled even if time passes, and the user will enter the virtual reality world The problem of being unable to get in occurs.

In view of the above problems, a virtual reality presentation device, an HMD device, a virtual reality presentation system, and a virtual reality presentation method that reduce the discontinuity of the boundary between virtual and real and allow the user to deeply enter the virtual reality world The purpose is to provide.

A virtual reality presentation system according to an aspect of the present invention includes a user state detection unit that detects a user state, a video generation unit that generates a virtual reality presentation video based on the user state, and the generated It is determined that the video display means for displaying the virtual reality presentation video, the determination means for determining whether the user status detected by the user status detection means satisfies a predetermined criterion, and the criterion is satisfied. Video display control means for performing control to display the virtual reality presentation video on the video display means.

According to another aspect of the present invention, a virtual reality presentation device includes a user state detection unit that detects a user state, a video display unit that displays a video, and a user state detected by the user state detection unit. A virtual reality presentation controlled based on the state of the user detected by the user state detection unit when it is determined that the reference is satisfied Display control means for performing control to display a video for use on the video display means.

The virtual reality presentation method according to another aspect of the present invention includes a user state detection step for detecting a user state, and a determination step for determining whether the detected user state satisfies a predetermined criterion, A video generation step for generating a virtual reality presentation video based on the detected user state, and a control for displaying the virtual reality presentation video when it is determined that the criterion is satisfied. A video display control step for performing, and a video display step for displaying the generated video for virtual reality presentation based on the control.

ADVANTAGE OF THE INVENTION According to this invention, the discontinuity of the boundary of virtual and real can be reduced, and it can enable a user to penetrate deeply into the virtual reality world.

It is a block diagram which shows the structure of the virtual reality presentation apparatus which concerns on Embodiment 1. FIG. FIG. 3 is a flowchart illustrating a process flow in the virtual reality presentation device according to the first embodiment. FIG. 5 is a flowchart illustrating a detailed flow of processing in the virtual reality presentation device according to the first embodiment. 6 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 2. FIG. 6 is a block diagram showing another configuration of the virtual reality presentation system according to Embodiment 2. FIG. It is a figure explaining the relationship between the image | video shown with the virtual reality presentation apparatus when not performing display start determination, and the scenery which a user senses. It is a figure explaining the relationship between the image | video shown with the virtual reality presentation apparatus in the case of performing display start determination, and the scenery which a user senses. It is a figure explaining the relationship between the image | video shown with a virtual reality presentation apparatus, and the scenery which a user senses when a user changes a visual field. It is a block diagram which shows the structure of the virtual reality presentation apparatus which concerns on Embodiment 3. FIG. It is a figure which shows an example of virtual reality service provision start notification information. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 3. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation device according to Embodiment 4. FIG. 10 is an external perspective view of a virtual reality presentation device according to Embodiment 4 from diagonally forward and diagonally rear. FIG. 10 is a cross-sectional view of the virtual reality presentation device according to Embodiment 4 in the vicinity of the display panel. FIG. 10 is a cross-sectional view near a display panel in a modification of the virtual reality presentation device according to Embodiment 4. FIG. 10 is a cross-sectional view near a display panel in a modification of the virtual reality presentation device according to Embodiment 4. FIG. 10 is a cross-sectional view near a display panel in a modification of the virtual reality presentation device according to Embodiment 4. FIG. 10 is a cross-sectional view near a display panel in a modification of the virtual reality presentation device according to Embodiment 4. FIG. 10 is a cross-sectional view near a display panel in a modification of the virtual reality presentation device according to Embodiment 4. It is another example of the user information management file which manages the information regarding the registered user. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 4. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation device according to Embodiment 5. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 5. It is an example of the hierarchical structure of virtual reality content. It is an example of the hierarchical structure of a virtual reality control program. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 6. FIG. 20 is a block diagram showing another configuration of the virtual reality presentation system according to Embodiment 6. FIG. 20 is a block diagram showing another configuration of the virtual reality presentation system according to Embodiment 6. It is a figure which shows the relationship between the image | video for virtual reality presentation before a virtual reality presentation, after a virtual reality presentation start, and after adding a virtual reality object, and the virtual reality world which a user senses. FIG. 20 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 7. FIG. 20 is a block diagram showing a configuration of a virtual reality presentation device according to Embodiment 7. It is a figure which shows the relationship between the image | video for virtual reality presentation before a virtual reality presentation, after a virtual reality presentation start, and after adding a virtual reality object, and the virtual reality world which a user senses. FIG. 20 is a block diagram showing another configuration of the virtual reality presentation system according to Embodiment 7. FIG. 20 is a block diagram showing another configuration of the virtual reality presentation device according to Embodiment 7. It is an example of the management file which manages the correlation of the layer of virtual space, and a virtual reality object. FIG. 20 is a block diagram showing a configuration of a virtual reality presentation device according to Embodiment 8. FIG. 20 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 8. FIG. 20 is a block diagram illustrating a configuration of a virtual reality presentation device according to Embodiment 9. It is a figure explaining the time transition of an illusion degree. It is a block diagram which shows the structure of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 10. FIG. It is a block diagram which shows the structure of the virtual reality presentation system which concerns on Embodiment 11. FIG. It is a block diagram which shows the structure of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is an example of the management file which manages the correspondence of a virtual reality object and virtual reality. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 12. FIG. It is an example of the management file which registered the virtual reality service which a user can use. It is an example of the management file which manages the correspondence of a virtual reality service, the virtual reality object used by the said virtual reality service, and its virtual reality. It is a block diagram which shows the structure of the virtual reality presentation system which concerns on Embodiment 13. FIG. It is an example of the management file which manages the correspondence of a virtual reality object and the virtual reality according to surrounding situations. It is a block diagram which shows another structure of the virtual reality presentation system which concerns on Embodiment 13. FIG. This is a management file that summarizes the correspondence between objects and surrounding situations. It is a block diagram which shows the structure of the virtual reality presentation system which concerns on Embodiment 14. FIG. It is an example of virtual reality service provision start notification information. It is a figure explaining the virtual reality object divided and arrange | positioned at several layers. It is a figure explaining the relationship between a user position and layer order. It is a figure which shows the image | video for virtual reality presentation according to the layer before and behind a correction | amendment process. FIG. 20 is an external perspective view of a virtual reality presentation device according to a fourteenth embodiment from obliquely forward and obliquely rearward. FIG. 17 is a cross-sectional view of the virtual reality presentation device according to the fourteenth embodiment near a display panel. FIG. 29 is a cross-sectional view near the display panel in a modification of the virtual reality presentation device according to the fourteenth embodiment. FIG. 29 is a cross-sectional view near the display panel in a modification of the virtual reality presentation device according to the fourteenth embodiment. FIG. 29 is a cross-sectional view near the display panel in a modification of the virtual reality presentation device according to the fourteenth embodiment. FIG. 29 is a cross-sectional view near the display panel in a modification of the virtual reality presentation device according to the fourteenth embodiment. FIG. 29 is a cross-sectional view near the display panel in a modification of the virtual reality presentation device according to the fourteenth embodiment. FIG. 16 is an external perspective view from diagonally forward and diagonally backward in a modified example of the virtual reality presentation device according to the fourteenth embodiment. It is a figure explaining the difference between the virtual reality world and the real world. It is a figure explaining the correlation with the virtual reality of the virtual reality world, and the illusion degree of a user. FIG. 16 is a conceptual diagram schematically showing an overall configuration of a person / organization related to the virtual reality service according to the fifteenth embodiment. It is a figure which shows an example of the user registration information which a virtual reality service provision company manages. 1 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 1. FIG. It is a figure explaining the difference between the real world and the virtual reality world. It is a figure explaining the structure of virtual reality content. It is an example of the management file which manages a virtual reality object. It is an example of model data of a virtual reality object. It is a figure explaining the classification | category of a virtual reality object control program. 6 is a block diagram showing a configuration of a modified example of the virtual reality presentation system according to Embodiment 1. FIG. It is an example of virtual reality service history information. It is an example of virtual reality history information. It is an external appearance perspective view from the diagonally forward and diagonally rear of the virtual reality presentation device according to the second embodiment. . FIG. 10 is a cross-sectional view of the virtual reality presentation device according to Embodiment 2 near a display panel. 6 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 2. FIG. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 3. It is an example of virtual reality service provision start notification information. It is an example of the image | video for virtual reality presentation according to layer produced | generated for every layer. It is a figure explaining the change of a layer order. It is a figure explaining the video for virtual reality presentation for every layer in which the display position of the virtual reality object was changed before and after video correction processing. It is a figure which shows the combination of the virtual reality related data memorize | stored in the database. It is a block diagram which shows the whole virtual reality presentation system structure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, parts denoted by the same reference numerals have substantially the same functions. For the sake of clarification of the invention, explanations of overlapping parts are omitted as appropriate.

<Embodiment 1>
FIG. 1 is a block diagram showing a configuration of a virtual reality presentation device 100 according to Embodiment 1 of the present invention. The virtual reality presentation device 100 includes a user state detection unit 110, a display start determination processing unit 120, a display control unit 130, and a display unit 140.

The user state detection unit 110 detects the state of the user who is using the virtual reality presentation device 100. The user state detection unit 110 includes, for example, an acceleration sensor attached to the virtual reality presentation apparatus 100, an internal camera that captures the user's facial expression, other inclination sensors that detect the inclination of the user, A GPS receiver that detects the position of The user status detection unit 110 outputs the detection result to the display start determination processing unit 120 as user status information indicating the user status.

The display start determination processing unit 120 determines whether the user state detected by the user state detection unit 110 satisfies a predetermined criterion. For example, when the user state detection unit 110 is an internal camera that captures part or all of the user's face, the display start determination processing unit 120 determines whether the user's eyes are closed for a predetermined time or more. If the user state detection unit 110 is an acceleration sensor, the display start determination processing unit 120 determines whether the direction in which the user is facing has changed by a predetermined angle or more.

When the display start determination processing unit 120 determines that the user state indicated by the detection result in the user state detection unit 110 satisfies a predetermined criterion set in advance, the display control unit 130 displays the display unit Control to display a predetermined video on 140 is performed. For example, the display control unit 130 controls the display position and display time of the virtual reality presentation video displayed on the display unit 140.

The display unit 140 displays a virtual reality presentation video based on the control of the display control unit 130. The video displayed on the display unit 140 is a still image or a moving image of a virtual reality object that is an object for presenting virtual reality.

Here, virtual reality (VR) refers to a world with reality created artificially by combining computer graphics (CG) and sound effects.

The virtual reality object refers to a virtual person or a building represented in the virtual reality world by being arranged in a virtual space that is a virtual space artificially set. By setting the virtual space so as to correspond to the real space around the user, the virtual reality world is presented to the user.

Each type of virtual reality object is composed of model data including 3D polygon data defining the appearance shape, texture data pasted on the surface of the shape formed by the polygon data, and the like. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint point data and the vertex of the polygon data are associated, and changing the position and angle of the joint point changes the position of the vertex of the associated polygon data, and the appearance of the virtual reality object The shape changes. A plurality of types of texture data are also prepared, and each texture data is identified by an identification number. An image of the virtual reality object is generated by pasting the selected texture onto the polygon surface.

The video data displayed on the display unit 140 may be stored in a storage unit separately provided in the virtual reality presentation device 100 in a state where it can be read and displayed as it is, or the virtual reality is stored in the storage unit. A configuration may be used in which model data of an object is stored, and the model data is drawn to generate a display video. Alternatively, the encoded data of the video displayed on the display unit 140 may be received from the outside of the apparatus. As the display unit 140, for example, a transmissive liquid crystal display or an organic EL display can be used.

FIG. 2 is a flowchart showing the operation of the virtual reality presentation device 100. The user state detection unit 110 detects a user state (step S101). For example, the user's state such as the user's facial expression, the eye open / closed state, the user's moving speed and acceleration, and other directions facing the user are detected at predetermined intervals. Although it depends on the type of user status to be detected, the detection cycle is once every 10 μs to 1 second, more preferably once every 100 μs to 100 ms, and more preferably once every 1 ms to 10 ms. A configuration in which sampling is performed at a cycle is preferable because the user's state can be detected in real time.

When a user state is newly detected in step S101, the display start determination processing unit 120 determines whether the detected user state satisfies a predetermined criterion (step S102). The display start determination processing unit 120 satisfies a predetermined criterion based not only on the user status at one timing detected in step S101 but also on the user status specified based on the detection results at a plurality of timings. It may be determined whether or not.

As a result of the determination in step S102, when it is determined that the current user state does not satisfy the criteria for starting the display of the virtual reality presentation video, the process returns to step S101 to detect the user state Wait until the next timing. On the other hand, if the result of determination in step S <b> 102 is that the current state of the user satisfies a preset criterion for starting video display, the display control unit 130 displays a predetermined video on the display unit 140. Control to display is performed (step S103).

For example, it is assumed that the user's eyes are closed for a predetermined time or more is set as the video display start condition. FIG. 3 is a flowchart showing an operation flow of the virtual reality presentation device 100 when the determination criterion is provided.

The user state detection unit 110 detects a user state (step S101). The display start determination processing unit 120 checks whether the current mode is the first mode or the second mode (step S102a).

When the current mode is the first mode, the display start determination processing unit 120 determines whether the user's eyes are closed based on the detection result in step S101 (step S102b). If the eyes are not closed, the process returns to step S101. On the other hand, when the eyes are closed, the display start determination processing unit 120 sets the current mode to the second mode (step S102c). Subsequently, the display start determination processing unit 120 starts measuring a predetermined time by setting a timer, and returns to step S101 (step S102d).

On the other hand, as a result of the determination in step S102a, if the current mode is the second mode, the display start determination processing unit 120 determines whether the user's eyes are closed (step S102e). As a result of the determination, if the user's eyes are not closed, the user can easily recognize that the displayed video is virtual from the discontinuous change in the scenery due to the start of video display. Does not proceed and returns the current mode to the first mode and returns to step S101 (step S102f).

On the other hand, when it is determined that the user's eyes are closed as a result of the determination in step S102e, the display start determination processing unit 120 determines whether the time measurement started in step S102d is completed (step S102g). ). If the time measurement has not been completed as a result of the determination, it is determined that the conditions for starting the video display are not yet ready, and the process returns to step S101.

On the other hand, if the time measurement is completed as a result of the determination in step S102g, the display start determination processing unit 120 assumes that the conditions for starting the video display are satisfied, and returns the current mode to the first mode ( In step S102h), the display control unit 130 is instructed to start displaying video.

As described above, the process in steps S102a to 102h determines whether or not the user's state satisfies the criteria for starting video display. Note that the reason for measuring the predetermined time is that if the eyes are simply closed, the eyes may be closed due to blinking. This is because it is not preferable in imprinting. If the scenery that can be seen in the blink of an eye is different, the discontinuity of the scenery creates a sense of incongruity, making it difficult to recognize the virtual as real. On the other hand, if you close your eyes for a certain period of time, the scenery you have seen will fade from the brain, so even if the video is multiplexed in a form that blends into the real scenery, the user can see that it is virtual. The possibility of recognizing it is reduced, and the user can naturally enter the virtual reality world.

The display control unit 130 reads the video to be displayed based on the determination result in the display start determination processing unit 120, and displays the read video on the display unit 140 (step S103).

As described above, the virtual reality presentation device 100 according to the first embodiment includes the user state detection unit that detects the user state, the display unit that displays the video, and the user state detected by the user state detection unit. Determining means for determining whether or not a predetermined criterion is satisfied, and display control means for performing control to display a predetermined video on the display means when it is determined that the criterion is satisfied.

With this configuration, it is possible to reduce the discontinuity of the boundary between the virtual and the real and to allow the user to unconsciously enter the virtual reality world.

In other words, in the past, development of technology to display images to users so that the unnaturalness is eliminated and real and virtual are fused by real-time control so that the behavior of the displayed video matches the actual landscape is being promoted. It was. However, technology development that suppresses the unnaturalness caused by discontinuity that occurs when video display is started has not been sufficiently advanced, and when the video for virtual reality is displayed, the user is assumed to be virtual. Was immediately recognized and the video was perceived as being virtual. This hinders entry into the virtual reality world and is a hindrance to let the user fully misunderstand that the virtual is real even if the behavior of the displayed video blends into the real landscape. It was.

On the other hand, according to the virtual reality presentation device described in the first exemplary embodiment, the timing for displaying the video based on the state of the user so that the user does not clearly recognize that it is virtual at the time of displaying the virtual video. Be controlled. Since the display of the video starts in a situation where the user does not feel unnaturalness, even if the user feels a little uncomfortable with the display of the video, the user has certainty that the video is virtual Not. Therefore, by matching the display with the actual scenery by subsequent display control, the user's alertness gradually fades, and the user can be drawn into the virtual reality world.

Note that the configuration of the virtual reality presentation device described above may be configured by a single device, or may be configured as a virtual reality presentation system by a plurality of devices.

<Embodiment 2>
FIG. 4 is a block diagram illustrating a configuration of the virtual reality presentation system 2000 according to the second embodiment. The virtual reality presentation system 2000 includes a user state detection unit 110, a video display start determination processing unit 120, a display control unit 130, a display unit 140, a virtual reality presentation control unit 2010, a display video generation processing unit 2020, Is provided.

The user state detection unit 110 includes an internal camera 111, a tilt sensor 112, and an acceleration sensor 113. The internal camera 111 captures part or all of the user's face. The tilt sensor 112 is a sensor that detects the tilt of the user's head. The acceleration sensor 113 is a sensor that detects acceleration applied to the user's head.

Part or all of the information detected by the user state detection unit 110 is sent to the video display start determination processing unit 120 as user state information. Also, part or all of the information detected by the user state detection unit 110 is sent to the display video generation processing unit 2020 as user state information.

The virtual reality presentation control unit 2010 performs overall control related to virtual reality presented to the user. When the virtual reality presentation control unit 2010 determines to start presenting virtual reality to the user, the virtual reality presentation control unit 2010 instructs the video display start determination processing unit 120 to perform video display start determination, and the display video generation processing unit 2020 generates a display video. Instruct.

The video display start determination processing unit 120 starts video display based on the user status detected by the user status detection unit 110 when receiving a video display determination instruction for virtual reality presentation from the virtual reality presentation control unit 2010. Determine if you can. Specifically, the video display start determination processing unit 120 determines whether the user status detected by the user status detection unit 110 satisfies a predetermined criterion. For example, it is determined whether the user's eyes are closed for a predetermined time or more based on the user's face image captured by the internal camera 111. Further, for example, based on the detection results of the inclination sensor 112 and the acceleration sensor 113, it is determined whether the direction in which the user is facing has changed by a predetermined angle or more.

The display video generation processing unit 2020 starts a process of generating a video to be displayed on the display unit 140 based on a virtual reality video generation start instruction from the virtual reality presentation control unit 2010. The display video generation processing unit 2020 sequentially generates virtual reality video controlled based on the detection result in the user state detection unit 110 and outputs the generated video to the display control unit 130. Examples of the virtual reality video include a virtual reality object video.

When the video display start determination processing unit 120 determines that the user state indicated by the detection result in the user state detection unit 110 satisfies a predetermined criterion set in advance, the display control unit 130 displays Control for displaying the virtual reality presentation video generated by the video generation processing unit 2020 on the display unit 140 is performed.

As described above, according to the virtual reality presentation system according to the second embodiment, the virtual reality presentation video controlled based on the user state generated by the display video generation processing unit 2020 is displayed in the display control unit 130. It is displayed on the display unit 140 based on the control. Based on the virtual reality presentation start control performed by the virtual reality presentation control unit 2010, the display video generation processing unit 2020 generates a virtual reality video and outputs the generated video to the display control unit 130. The unit 130 does not immediately display the video generated by the display video generation processing unit 2020 on the display unit 140 but starts display on the display unit 140 based on the determination result from the display start determination processing unit 120.

With this configuration, since the display video is generated by the display video generation processing unit 2020 so that it can be displayed at any time, the video display start determination processing unit 120 has a user state set in advance. When it is determined that the predetermined criterion is satisfied, the virtual reality video can be immediately displayed on the display unit 140.

Note that the functions constituting the virtual reality presentation system 2000 described above may be incorporated into a virtual reality presentation device worn by the user, or some functions may be incorporated into an external information processing server to provide virtual reality presentation. It is good also as a structure by which the video for virtual reality presentation is displayed on the display part 140 in a virtual reality presentation apparatus by communicating with an apparatus.

FIG. 5 is a block diagram showing a configuration of the virtual reality presentation system 2000 when an external information processing server performs a part of processing necessary for presentation of virtual reality. The virtual reality presentation system 2000 is a device worn by a user, and provides a virtual reality service 200 that provides a virtual reality service that is a service for presenting virtual reality, and a virtual reality presentation device 200 that directly presents virtual reality to the user. A server 2100; a voice analysis server 2200 that analyzes voice input by the virtual reality presentation device 200; and a virtual reality content storage database 2300 that stores virtual reality content.

The virtual reality presentation device 200 illustrated in FIG. 5 includes a user state detection unit 110, a video display start determination processing unit 120, a display control unit 130, a display unit 140, a voice input unit 210, and device-side virtual reality presentation control. Unit 220, communication unit 230, and video data temporary storage unit 240.

The voice input unit 210 is a so-called microphone, and collects voices uttered by the user wearing the virtual reality presentation device 200 and voices from the user's surrounding environment and inputs them as voice data. The voice input unit 210 outputs the input voice data to the communication unit 230.

The device-side virtual reality presentation control unit 220 performs various controls for virtual reality presentation required on the virtual reality presentation device 200 side. The device-side virtual reality presentation control unit 220 receives the virtual reality service provision start notification information, which is a notification of the provision of virtual reality service transmission transmitted from the virtual reality service providing server 2100, when the communication unit 230 receives virtual reality service provision start notification information. Control for starting presentation. Specifically, when receiving the virtual reality service provision start notification information, the device-side virtual reality presentation control unit 220 starts virtual reality presentation processing at least in the user state detection unit 110 and the video display start determination processing unit 120, respectively. Give instructions.

The user state detection unit 110 starts a user state detection process based on a virtual reality presentation process start instruction from the device-side virtual reality presentation control unit 220, and uses the detection result as user state information to display a video display start determination processing unit 120. And output to the communication unit 230.

The video display start determination processing unit 120 starts the video display start determination process based on the virtual reality presentation process start instruction from the device-side virtual reality presentation control unit 220. Specifically, when receiving a virtual reality presentation process start instruction, the video display start determination processing unit 120 starts a determination process to determine whether the user state detected by the user state detection unit 110 satisfies a predetermined criterion. To do.

The communication unit 230 communicates with the virtual reality service providing server 2100 via the communication network 10 such as the Internet. The communication unit 230 includes a wireless transmission unit 231 that wirelessly transmits various types of information generated in the device 200 to the outside of the device, and a wireless reception unit 232 that wirelessly receives various types of information transmitted from the outside of the device.

The wireless transmission unit 231 wirelessly transmits at least the user state information generated by the user state detection unit 110 and the voice data input by the voice input unit 210 to the virtual reality service providing server 2100. In addition, some control information generated by the device-side virtual reality presentation control unit 220 is wirelessly transmitted to the virtual reality service providing server 2100.

The wireless reception unit 232 wirelessly receives at least encoded data of virtual reality video generated by the virtual reality service providing server 2100. The wireless reception unit 232 outputs the wirelessly received video encoded data to the video data temporary storage unit 240.

The video data temporary storage unit 240 stores the encoded video data wirelessly received by the wireless reception unit 232 in a storage area for virtual reality presentation. The video data temporary storage unit 240 stores the latest generated video encoded data by overwriting the previously stored video encoded data based on the sequentially encoded video encoded data.

When the video display start determination processing unit 120 determines that the user state indicated by the detection result in the user state detection unit 110 satisfies a predetermined criterion set in advance, the display video generation processing unit 2140 Control is performed to display the generated virtual reality presentation video on the display unit 140.

Based on the determination result from the video display start determination processing unit 120, the display control unit 130 reads out the virtual reality presentation video encoded data temporarily stored in the video data temporary storage unit 240 and performs a decoding process. In addition, control is performed to display the video based on the obtained video data on the display unit 140. The display unit 140 displays a virtual reality presentation video according to display control performed by the display control unit 130.

  The virtual reality service providing server 2100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, and a display video generation processing unit 2140.

The communication unit 2110 communicates with the virtual reality presentation device 200 via the communication network 10 such as the Internet. The communication unit 2110 receives at least user state information and audio data transmitted from the virtual reality presentation device 200 via the communication network 10 and a virtual reality presentation generated by the display video generation processing unit 2140. A transmission unit 2112 that transmits the encoded video data to the virtual reality presentation device 200 via the communication network 10. The reception unit 2111 outputs the received audio data to the audio context acquisition unit 2120 and outputs user state information to the display video generation processing unit 2140.

The voice context acquisition unit 2120 acquires the context of the input voice data. Specifically, the voice context acquisition unit 2120 transfers voice data to the voice analysis server 2200 and receives a voice context extracted as a result of the voice analysis processing in the voice analysis server 2200, thereby inputting the voice of the virtual reality presentation device 200. The voice context input by unit 210 is acquired, and the acquired context is output to server-side virtual reality presentation control unit 2130.

The voice analysis server 2200 is a server for specifying a context indicated by input voice data, and includes a voice analysis processing unit 2210 and a context extraction unit 2220.

The voice analysis processing unit 2210 analyzes the voice data transferred from the virtual reality service providing server 2100 according to a predetermined voice analysis algorithm, and divides the voice data into a plurality of voice units.

For example, the voice analysis processing unit 2210 divides the voice data into unit voices by performing a matching process between the voice unit data included in the voice database storing a plurality of voice units and the input voice data. The voice unit data included in the voice database is assigned a code such that the unit voice “I” is code No004 and the unit voice “ko” is code No021, for example. The voice analysis processing unit 2210 divides the input voice data into unit voices, thereby converting the input voice data into a sequence of codes. The voice database is not shown.

The context extraction unit 2220 extracts the above-described audio data transferred from the information processing server 2200, that is, the context indicated by the audio input by the audio input unit 210 of the virtual reality presentation device 200.

For example, the context extraction unit 2220 accesses a word / grammar database storing a plurality of word data and grammar data defining a grammatical structure, and determines how many of the series of codes obtained by the analysis processing in the speech analysis processing unit 2210 A meaningful part is extracted by performing matching with word data stored in the database. The context extraction unit 2220 extracts the context indicated by the input audio data based on the series of processes.

For example, if the input voice data is a voice saying “I want to play with cats”, the voice analysis processing unit 2210 performs “ne”, “ko”, “to”, “ta”, “wa”, “Mu”, “Re”, “Ta”, “I”, “NA” are divided into unit sounds. Since a code is assigned to each unit voice, the voice analysis processing unit 2210 puts the voice data in a list of codes such as “072/02/041/036/124/085/099/036/004/043”. Convert.

The context extraction unit 2220 performs matching processing between the code converted from the input voice data and a code indicating a word in the word / grammar database to convert the code into a list of words. For example, by performing the matching process, the context extraction unit 2220 extracts “072021”, “034”, “036224085099”, and “0360004043” for each group, and “Ob: Cat, Of: and Va: Deflection. , Vc: taina "and organize it into a subject, object, predicate, particle, etc. The context extraction unit 2220 determines that the input voice indicates a context of “I want to play with a cat” by the processing. The context extraction unit 2220 transmits the extracted context data to the voice context acquisition unit 2120 of the virtual reality service providing server 2100.

The server-side virtual reality presentation control unit 2130 starts providing the virtual reality service when the context extracted by the voice analysis server 2200 requests to start the virtual reality service. Specifically, the server-side virtual reality presentation control unit 2130 generates virtual reality service provision start notification information and outputs it to the communication unit 2110. Also, the server-side virtual reality presentation control unit 2130 reads out virtual reality content related to the virtual reality service provided from the virtual reality content storage database 2300 and outputs the virtual reality content to the display video generation processing unit 2140.

The display video generation processing unit 2140 generates a virtual reality presentation video to be displayed on the virtual reality presentation device 200 based on the virtual reality content read by the server side virtual reality presentation control unit 2130. The display video generation processing unit 2140 generates a video for virtual reality presentation based on the user state information. The display video generation processing unit 2140 sequentially generates virtual reality video controlled based on the detection result in the user state detection unit 110 from the read virtual reality content, performs encoding processing, and encodes the video. Generate data. The display video generation processing unit 2140 outputs the generated video encoded data to the communication unit 2110.

In this way, by generating a virtual reality presentation video on the virtual reality service providing server side having relatively high processing performance, a more detailed virtual reality presentation video can be generated on the virtual reality presentation device side. It is possible to display.

FIG. 6 shows a virtual reality world presented in a conventional virtual reality presentation device that does not include the video display start determination processing unit 120. At the timing of T0 in FIG. 6, the real scenery is the scenery of the room where the sofa is placed, and the scenery can be seen through the transmissive front panel which is the display unit of the virtual reality presentation device. Since nothing is displayed on the display unit 140, the user can see only the actual scenery.

It is assumed that the user makes a request “I wish I had a dog” at the timing of T1. The voice is input to the virtual reality presentation device. The virtual reality presentation device reads the video content of the dog from the video content storage database and displays it on the display unit at the timing of T2 to start presenting virtual reality.

Here, the virtual reality presentation device immediately displays the video of the dog on the display unit when the reading of the video content is completed. If the user opens his eyes at this time, the dog suddenly appears on the sofa. It will be. In the real world, the appearance of an object as if it moves instantaneously is highly unrealistic and far from physical phenomena that are empirically recognized in life. Therefore, the user clearly recognizes that the dog is not a real dog but a dog displayed on the display unit. Once it is recognized that it is virtual, the user's alertness will not be lost even if time passes, and it will be difficult for the user to enter the virtual reality world. That is, the illusion that a dog that is visible in front of you is a real world dog cannot be realized.

Next, the virtual reality presented by the virtual reality presentation device 200 according to the second embodiment will be described with reference to FIG. The timings of T0 and T1 are the same as those in FIG. In the virtual reality presentation device 200, since the video display start determination processing unit 120 determines whether to start displaying the video, the video of the dog, which is a virtual reality object, is displayed by the display video generation processing unit 2140 at the timing T1. Even if it is generated as a virtual reality video and stored in the video data temporary storage unit 240, it is not immediately displayed on the display unit 140.

Based on the video image of the user's eyes taken by the internal camera 111, the video display start determination processing unit 120 determines that the user has closed his eyes for a time equal to or greater than a predetermined reference value. An image of a dog that is a virtual reality object read out as an image is displayed.

The user can see the dog at the timing T4 when the eyes are opened, but can make an illusion that the dog has strayed while the eyes are closed. Such a physical phenomenon is a phenomenon that can actually occur sufficiently, and the fact that appears in front of the user is prioritized. Therefore, even if there is a little uncomfortable feeling, the uncomfortable feeling fades with time, and the displayed virtual dog is recognized as a real dog and can enter the virtual reality world. it can.

In the above description, the video display start determination processing unit 120 determines whether the user's eyes are closed for a predetermined time or more based on part or all of the image data of the user's face taken by the internal camera 111. Although the case where it is determined whether to start displaying video based on the determination result has been described, the present invention is not limited to this.

For example, the video display start determination processing unit 120 may be configured to display a virtual reality presentation video when the user's field of view changes. For example, the video display start determination processing unit 120 may be configured to start displaying video when the user rotates the face and rotates the face by a predetermined angle or more.

In FIG. 8, the video content is read at the timing of T1, but the orientation of the user is not rotated by a predetermined angle or more, and it is unnatural to start displaying the video in this state. No video is displayed.

At the timing of T3, the user's direction is rotated 45 degrees from the beginning, and the dog's paw as the video content is starting to be displayed at the lower right of the display panel. That is, as a condition for starting the virtual reality video display, the user's direction is rotated 45 degrees to the right as a condition for starting the video display. The video display start determination processing unit 120 specifies the current direction of the user based on the tilt detected by the tilt sensor 112 and the acceleration detected by the acceleration sensor 113. The video display start determination processing unit 120 instructs the display control unit 130 to start displaying the virtual reality video, assuming that the display start condition is satisfied when the direction of the user exceeds the reference value of 45 degrees.

Furthermore, at the timing of T4, the orientation of the user is rotated 60 degrees from the beginning, and the display control unit 130 displays the display unit 140 based on the display position, display angle, and display size calculated in real time according to the rotation of the user. Displays a video of a dog, which is a virtual reality object.

At the timings T5 and T6, the orientation of the user has been rotated by 90 degrees and 120 degrees, respectively, from the beginning, and the entire video of the dog that is a virtual reality object is displayed on the display unit 140.

With this configuration, the user can change the field of view so that the video can be seen. Therefore, the scenery that the user sees is not discontinuous at the point of contact between the real and virtual, as it is when a dog is actually present. It is gone. Therefore, the user does not recognize that the displayed video of the dog is virtual, but recognizes that it is a real dog. Therefore, the user can enter the virtual reality world.

Note that the video display start determination processing unit 120 described above may be disposed on the virtual reality service providing server 2100 side.

<Embodiment 3>
There are various types of virtual reality presented to the user. The display start timing of the virtual reality presentation video is also preferably determined according to the type of virtual reality presented to the user.

FIG. 9 is a block diagram illustrating a functional configuration of the virtual reality presentation device 300 according to the third embodiment. The virtual reality presentation device 300 includes a user state detection unit 110, a video display start determination processing unit 120, a display control unit 130, a display unit 140, a voice input unit 210, a device-side virtual reality presentation control unit 220, A communication unit 230, a video data temporary storage unit 240, an ambient condition detection unit 310, and a video display start condition setting unit 320 are provided.

As the user state detection unit 110 that detects the user state, the virtual reality presentation device 300 includes an internal camera 111 that captures part or all of the face of the user wearing the device, and an inclination that detects the inclination of the user's head. The sensor 112 and the acceleration sensor 113 which detects the acceleration of a user's head are provided.

The video data of the user's face captured by the internal camera 111, the tilt information indicating the tilt detected by the tilt sensor 112, and the acceleration information indicating the acceleration detected by the acceleration sensor 113 are used for video display start determination processing. It is sent to the display start determination processing unit 120.

Further, the tilt information generated by the tilt sensor 112 and the acceleration information generated by the acceleration sensor 113 are output to the communication unit 230 for use in the virtual reality video generation processing.

The voice input unit 210 collects voices uttered by a user wearing the apparatus and inputs user voice data, and collects voices from the user's external environment and inputs ambient voice data. A surrounding microphone 212. The sound collected by each microphone is converted from analog electric signals to digital signal sound data by AD conversion processing, and is output to the communication unit 230.

The surrounding situation detection unit 310 detects the surrounding situation of the user and generates ambient situation information indicating the surrounding situation of the user. In the virtual reality presentation device 300, the surrounding situation detection unit 310 includes a line-of-sight camera 311 and a surrounding camera 312.

The line-of-sight camera 311 is a camera installed in front of the virtual reality presentation device 300, and acquires a front scene image (line-of-sight scene image) by capturing a scene in front of the user corresponding to the user's line of sight. The forward scenery image acquired by the line-of-sight camera 311 is output to the communication unit 230.

The peripheral camera 312 is a camera installed on the side part or the rear part of the virtual reality presentation device 300, and acquires a peripheral landscape image by capturing a scene in a range around the user that cannot be captured by the visual line camera 311. The peripheral scenery image acquired by the peripheral camera 312 is output to the communication unit 230.

The communication unit 230 transmits the tilt information and acceleration information output from the tilt sensor 112 and the acceleration sensor 113, respectively, to the external virtual reality service providing server as user status information. In addition, the communication unit 230 transmits the user voice data and the surrounding voice data respectively input from the user microphone 211 and the surrounding microphone 212 as input voice data to the virtual reality service providing server. In addition, the communication unit 230 transmits the visual line scenery image and the peripheral scenery image acquired by the visual line camera 311 and the peripheral camera 312 to the virtual reality service providing server as the surrounding state information.

  In addition, the communication unit 230 receives video encoded data for virtual reality presentation that is generated and transmitted sequentially from the virtual reality service providing server, and outputs the video encoded data to the video data temporary storage unit 240. Further, the communication unit 230 outputs the virtual reality service provision start notification information generated and transmitted by the virtual reality service providing server to the device-side virtual reality presentation control unit 220.

FIG. 10 is a diagram illustrating an example of virtual reality service provision start notification information received by the virtual reality presentation device 300. The virtual reality service provision start notification information includes a transmission source address that is the address of the virtual reality service providing server that is the transmission source, a transmission destination address that is the address of the virtual reality presentation device 300 that is the transmission destination, and the information includes the virtual reality service Information type for identifying the provision start notification information, virtual reality identification for distinguishing communication occurring between the virtual reality presentation device 300 and the virtual reality service providing server accompanying presentation of the virtual reality from other communications A number and virtual reality presentation start condition information that is a condition for starting presentation of virtual reality are included.

The communication relating to the presentation of the virtual reality that occurs between the virtual reality presentation device 300 and the virtual reality service providing server is assigned a virtual reality identification number, so that communication with other virtual reality presentation devices or It is distinguished from communication related to presentation of other virtual reality in the device 300.

The virtual reality presentation start condition is a condition for starting presentation of virtual reality, and in the present embodiment, the video display start condition is a condition for specifically starting video display for virtual reality. Is included at least.

  The device-side virtual reality presentation control unit 220 performs virtual reality presentation processing on the video display start determination processing unit 120 and the video display start condition setting unit 320 based on the virtual reality service provision start notification information received by the communication unit 230. Instruct the start of.

  The video display start condition setting unit 320 sets a video display start condition as a virtual reality presentation start condition based on an instruction from the device-side virtual reality presentation control unit 220. The video display start condition setting unit 320 is a video display start that is a reference in the determination process in the video display start determination processing unit 120 according to the virtual reality presentation start condition included in the virtual reality service provision start notification information received by the communication unit 230. Set conditions.

The video display start determination processing unit 120 determines whether the user state detected by the user state detection unit 110 satisfies the video display start condition set by the video display start condition setting unit 320.

For example, it is assumed that the video display start condition setting unit 320 sets the condition “the user has closed his eyes for 5 seconds or more” as the video display start condition X1. In this case, the video display start determination processing unit 120 inputs a video in a range including the user's eyes photographed by the internal camera 111, and determines, for example, whether the user's eyes are closed based on a matching process or the like. For example, the image data of the user's eyes in an open state or a closed state is stored in advance, and the image captured by the internal camera 111 is compared with the image data of the user's eyes in a closed state, When a correlation equal to or higher than the correlation value is detected, it can be determined that the user's eyes are closed.

When it is determined that the user's eyes are closed, the video display start determination processing unit 120 operates an internal timer to check whether the eyes are closed for 5 seconds set as a video display start condition. judge. If the user opens his eyes before the elapse of 5 seconds, the timer is reset, and a determination process is performed to determine whether the state where the eyes are closed for 5 seconds is maintained again.

The video display start determination processing unit 120 instructs the display control unit 130 to start displaying video when it is determined that the fifth second is closed as a result of the determination process.

  The display control unit 130 performs control to display a virtual reality video on the display unit 140. The display control unit 130 includes a video data reading unit 131 and a video decoding processing unit 132.

When the video display start determination processing unit 120 determines that the video display start condition is satisfied, the video data reading unit 131 reads the encoded video data for virtual reality presentation stored in the video data temporary storage unit 240. Read out and output to the video decoding processing unit 132. In the third embodiment, the video for virtual reality presentation will be specifically described as a video of a virtual reality object.

The video decoding processing unit 132 performs control to decode the video encoded data read by the video data reading unit 131 and display the video after decoding processing on the display unit 140. The display unit 140 displays a video of a virtual reality object, which is a video for virtual reality presentation, according to control from the display control unit 130.

Next, a virtual reality presentation system according to the third embodiment will be described. FIG. 11 is a block diagram illustrating a configuration of a virtual reality presentation system 3000 according to the third embodiment. The virtual reality presentation system 3000 includes a virtual reality presentation device 300, a virtual reality service providing server 3100, a voice analysis server 2200, a virtual reality content storage database 2300, and a contractor information storage database 3200.

The virtual reality service providing server 3100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, and an image analysis processing unit (video analysis processing unit) 3110. A virtual reality object reading unit 3120, a virtual space setting unit 3130, and a virtual reality object placement unit 3140.

The communication unit 2110 communicates with the virtual reality presentation device 300 via the communication network 10 such as the Internet. The communication unit 2110 receives user status information, audio data, and ambient status information transmitted from the virtual reality presentation device 300, and transmits the user status information to the display video generation processing unit 2140 and the audio data to the audio context acquisition unit 2120. The surrounding state information is output to the image analysis processing unit 3110, respectively.

The communication unit 2110 also receives the virtual reality service provision start notification information generated by the server-side virtual reality presentation control unit 2130 and the video encoded data generated by the display video generation processing unit 2140, respectively. Is transmitted via the communication network 10.

The image analysis processing unit 3110 analyzes the video included in the surrounding situation information input by the communication unit 2110 and extracts predetermined parameters. In the virtual reality service providing server 3100 according to the third embodiment, the image analysis processing unit 3110 includes an object position detection unit 3111 and an optical component extraction unit 3112.

The object position detection unit 3111 specifies the position of the object shown in the front scenery image or the surrounding scenery image included in the surrounding situation information. The object position detection unit 3111 outputs real object position information indicating the position coordinates of each identified object to the virtual reality object placement unit 3140 and the drawing processing unit 2143 of the display video generation processing unit 2140.

For example, the object position detection unit 3111 inputs a front view image acquired by a plurality of line-of-sight cameras 311 arranged at locations corresponding to the positions of the left eye and the right eye, and a peripheral view image taken by a plurality of peripheral cameras 312. In each image, boundary determination processing is performed by calculating a difference between a color difference signal and a luminance signal between adjacent pixels and blocks. Based on the boundary determination process, the object position specifying unit 3111 specifies each object, wall, floor, sky, and the like that appear in the image, and labels each specified object. The object position specifying unit 3111 obtains the distance to the object by comparing the pixel positions of the objects with the same label in the images acquired by the cameras, and exists around the user. The positional relationship of an object etc. is specified.

  Preferably, the object position specifying unit 3111 generates model data indicating the physical environment around the user by modeling each object whose size and distance to the user are specified based on the input image. That is, the object position specifying unit 3111 models the physical environment around the user in the real world with 3D polygon data. The object position specifying unit 3111 outputs the generated model data of the real world object to the virtual reality object arrangement position determining unit 3140 and the drawing processing unit 2143 of the display video generation processing unit 2140, respectively.

  The optical component extraction unit 3112 extracts an optical component from the input forward scenery image. For example, the optical component extraction unit 3112 obtains information indicating the overall brightness (illuminance) around the user by calculating the average value of the luminance signal in each pixel of the front scene image. In addition, the optical component extraction unit 3112 obtains information indicating the overall color around the user by obtaining an average value of the color difference signals of each pixel. These pieces of information correspond to the brightness (illuminance) and color of the infinity light source controlled by the light source control unit 2142 described later.

  In addition, the optical component extraction unit 3112 compares the brightness of each pixel or block of the front scene image, and identifies the shadowed portion, so that the position of the light source around the user and the brightness of the light source Get information about.

  The optical component extraction unit 3112 outputs to the light source control unit 2142 information related to the optical component extracted from the front scenery image input as the ambient state information. The optical component extraction unit 3112 performs similar processing on the surrounding scene image input as the surrounding state information, extracts information on the optical component related to the position and intensity of the light source in the virtual space, and controls the light source. It may be configured to output to the unit 2142.

The voice context acquisition unit 2120 acquires a context from the voice data received by the communication unit 2110. The voice context acquisition unit 2120 transfers the voice data to the voice analysis server 2200, and acquires the context of the voice data by receiving the context extracted based on the voice analysis processing in the voice analysis server 2200.

The server-side virtual reality presentation control unit 2130 starts the provision of the virtual reality service to the user when the context acquired by the audio context acquisition unit 2120 requests to start providing the virtual reality service. I do.

When receiving a virtual reality service provision start request, the server-side virtual reality presentation control unit 2130 refers to the contractor information stored in the contractor information storage database 3200 and can provide virtual services that can be provided to the user. The real service is specified, virtual reality service provision start notification information is generated, and output to the communication unit 2110. The server-side virtual reality presentation control unit 2130 generates virtual reality service provision start notification information including a virtual reality presentation start condition corresponding to the virtual reality service to be provided.

A user who desires to use the virtual reality service has contracted with a virtual reality service provider that provides the virtual reality service, and the contractor information storage database 3200 stores information on the contract contents of the user as contract information. . The contractor information stores information such as the user's personal information and the types of available virtual reality services.

Based on the contractor information, the server-side virtual reality presentation control unit 2130 determines whether the user can use the virtual reality service requested in the context acquired by the voice context acquisition unit 2120. As a result of the determination, if the requested virtual reality service is a virtual reality service included in the contract, the server-side virtual reality presentation control unit 2130 determines that the virtual reality service can be provided, Transition to real service provision start processing.

The server-side virtual reality presentation control unit 2130 issues an instruction to read virtual reality content used for the virtual reality service provided to the virtual reality object reading unit 3120. Further, the server-side virtual reality presentation control unit 2130 issues a new virtual space setting instruction to the virtual space setting unit 3130.

The virtual reality object reading unit 3120 reads out virtual reality object data related to the provided virtual reality service from the virtual reality content stored in the virtual reality content storage unit 2300, and sets the virtual reality object data to the virtual reality object arrangement. To the unit 3140.

The virtual reality content storage database 3200 stores virtual reality content. Here, the virtual reality content is a general term for data used to create a virtual reality world, and includes at least model data that defines the appearance of a virtual reality object. The model data includes polygon data that defines the appearance shape, texture data that is pasted on the surface of the appearance shape, and the like.

The model data of the virtual reality object includes polygon data that defines the shape of the virtual reality object and texture data that is attached to the surface of the shape that is defined by the polygon data.

The virtual space setting unit 3130 sets a new virtual space for the virtual reality service based on the virtual space setting instruction from the server side virtual reality presentation control unit 2130. Specifically, the virtual space setting unit 3130 sets a coordinate space in which a virtual reality object is arranged as a virtual space. In terms of internal processing, the virtual space setting unit 3130 performs processing for securing a virtual space memory area.

Here, it is preferable that the virtual space setting unit 3130 sets a virtual space having a size corresponding to the virtual reality service. When placing a small number of virtual reality objects in the room, there is no problem even if the virtual space is small. On the other hand, in order to realize a large virtual reality world outdoors, virtual reality objects should be placed far away from the user. This is because it is preferable to set a large virtual space as much as possible.

Therefore, the virtual space setting unit 3130 newly sets a virtual space having a size corresponding to the provided virtual reality service in accordance with an instruction from the server-side virtual reality presentation control unit 2130. The virtual space setting unit 3130 may set an orthogonal coordinate space of vertical Y, horizontal X, and height Z as the virtual space, or polar coordinates based on the origin (0, 0, 0), the distance R, the angle θ, and the angle φ. A space may be set.

  The virtual reality object placement unit 3140 places the virtual reality object for the virtual reality service read by the virtual reality object reading unit 3120 in the virtual space set by the virtual space setting unit 3130. The virtual reality object placement unit 3140 may place the read virtual reality object at a predetermined position in the virtual space, or place it at a position that does not overlap the real world object specified by the object position specification unit 3111. Also good. For example, model data of an object in the real world generated by the object position specifying unit 3111 is arranged in advance in a virtual space so that there is no area overlapping the model data of the object in the real world or less than a predetermined overlap amount. The virtual reality object is arranged in the virtual space.

The display video generation processing unit 2140 generates a video of the virtual reality object placed in the virtual space by the virtual reality object placement unit 3140. The display video generation processing unit 2140 includes a viewpoint control unit 2141, a light source control unit 2142, a drawing processing unit 2143, and a video encoding processing unit 2144.

  The viewpoint control unit 2141 controls the position and direction of the viewpoint when drawing the virtual reality object based on the user state information indicating the user state. Specifically, based on the tilt information indicating the tilt of the user's head included in the user state information and the acceleration information indicating the acceleration of the user's head, a virtual space corresponding to the position of the left eye and the position of the right eye of the user The viewpoint position at is calculated. Further, the line-of-sight direction in the virtual space corresponding to the direction of the user's line of sight is calculated based on the acceleration information. Here, the line-of-sight direction corresponds to the direction of the user's head, not the direction of the line of sight when the user moves only the eyeball.

  It is more preferable that the user status information includes azimuth information indicating the direction of the user, movement speed information indicating the movement speed of the user, and the like. The viewpoint control unit 2141 calculates the posture of the user's head based on each piece of information included in the user state information, and specifies the positions of the user's left eye and right eye and the direction in which the user's head is facing. The viewpoint control unit 2141 calculates the position coordinates of the viewpoint and the direction of the line of sight in the virtual space corresponding to the position of the user's eyes and the direction of the head.

  The light source control unit 2142 calculates the position coordinates of the light source and the illuminance (brightness) of the light source in the virtual space based on the optical component extracted by the optical component extraction unit 3112 of the image analysis processing unit 3110. For example, the light source control unit 2142 calculates the intensity of the light source at infinity by calculating the average value of the luminance of each pixel of the image extracted by the optical component extraction unit 3112, and the luminance between the pixels of the image. Based on the difference, the position coordinates and illuminance of other light sources are calculated. Further, the light source control unit 2142 may calculate the color of the light source based on the color difference signal of the pixel in the image subjected to the analysis process.

  The drawing processing unit 2143 generates a two-dimensional image by rendering the virtual reality object in real time. As an example, the rendering processing unit 2143 generates a virtual reality object image by performing rendering by combining a part or a plurality of processes such as a projection process, a clipping process, a hidden surface process, a shading process, and a texture mapping process shown below. To do.

  The drawing processing unit 2143 sets an image plane at a predetermined position in the virtual space based on the viewpoint position and the line-of-sight direction obtained by the viewpoint control unit 2141, and projects the virtual reality object. Get the projected image of the object. The drawing processing unit 2143 performs clipping processing of the image plane on which the virtual reality object is projected, and selects a range displayed on the display unit 140 in the virtual reality presentation device 300.

  The drawing processing unit 2143 performs hidden surface processing based on the model of the real world object specified by the object position specifying unit 3111. The drawing processing unit 2143 is configured to display the real world located between the viewpoint position calculated by the viewpoint control unit 2141 and the virtual reality object from among the pixels constituting the virtual reality object image projected onto the image plane. Hidden surface processing is performed by deleting pixel data that overlaps the projected image of the object model.

  The drawing processing unit 2143 performs shading processing based on the position and intensity of the light source calculated by the light source control unit 2142, and adjusts the brightness of the surface of the virtual reality object, the degree of light rise, and the light transmission state. The drawing processing unit 2143 performs texture mapping processing using texture data attached to the surface of the virtual reality object, appropriately performs correction processing such as anti-aliasing processing, and finally is performed by the display unit 140 of the virtual reality presentation device 300. Generate video for display.

  Note that the drawing processing unit 2143 generates two images, a left-eye image and a right-eye image, based on the viewpoint corresponding to the left eye and the viewpoint corresponding to the right eye controlled by the viewpoint control unit 2141. The drawing processing unit 2143 outputs the generated video to the video encoding processing unit 2144.

  The video encoding processing unit 2144 performs compression encoding processing on videos sequentially generated by the real-time rendering processing in the drawing processing unit 2143 based on a predetermined encoding method. The video encoding processing unit 2144 outputs the encoded video data after the encoding process to the communication unit 2110. The communication unit 2110 transmits the video encoded data output from the video encoding processing unit 2144 to the virtual reality presentation device 300, and the decoding control is performed by the display control unit 130 of the virtual reality presentation device 300, and the post-decoding processing is performed. The video is displayed on the display unit 140.

As described above, according to the virtual reality presentation system according to the third embodiment, conditions according to the provided virtual reality service are set, and presentation of virtual reality is started when the conditions are satisfied. Therefore, more appropriate control is performed so as to eliminate discontinuity at the timing when the world sensed by the user switches from the real world to the virtual reality world, and the user can naturally enter the virtual reality world. It becomes possible.

<Embodiment 4>
A human being is a living organism that detects events that occur in the outside world using the five senses, that is, taste, touch, vision, hearing, and smell, and processes the detected information in the brain to perform the next action. Making a user misidentify a virtual world that is different from the real world is nothing other than deceiving the five senses that are the interface between the user and the outside world. Therefore, the more you deceive your senses from the five senses, the more you will perceive the virtual world as the real world, and the virtual reality will deepen.

Of the five senses, the amount of information is the largest, and the sense that acts as the main sensor is vision. Therefore, the virtual reality presentation device of the present invention has started in that humans can easily enter virtual reality if they can deceive vision. However, if the virtual reality to be presented depends only on the video, there may be a deviation from the reality. For example, when the video of a barking dog is displayed on the display panel as being a real dog, the fact that the dog's cry is not heard is deviated from an event that can occur in the real world. Wakes up the suspicion that the current dog is not a real dog, but a created video.

In order to sublimate the virtual reality presentation device to a higher degree, there is no inconsistency between the reality and the virtual reality. The virtual reality presentation device according to the fourth embodiment reduces the boundary between virtual reality and reality by deceiving a plurality of senses from the senses that humans are born with, and allows the user to deepen into the virtual reality world. It is characterized by being able to enter.

FIG. 12 is a block diagram illustrating a functional configuration of the virtual reality presentation device 400 according to the fourth embodiment. The virtual reality presentation device 400 includes a user state detection unit 110, a video display start determination processing unit 120, a display control unit 130, a display unit 140, a voice input unit 210, a device-side virtual reality presentation control unit 220, Communication unit 230, ambient condition detection unit 310, virtual reality presentation data storage unit 410, virtual reality presentation start condition setting unit 420, audio reproduction start determination processing unit 430, audio reproduction processing unit 440, audio output Part 450.

In the virtual reality presentation device 400, the user state detection unit 110 includes an internal camera 111, a tilt sensor 112, and an acceleration sensor 113. The voice input unit 210 includes a user microphone 211 and a surrounding microphone 212. The ambient condition detection unit 310 includes a line-of-sight camera 311 and a peripheral camera 312. Since these configurations have already been described, description thereof will be omitted.

The virtual reality presentation data storage unit 410 stores virtual reality presentation data received by the communication unit 230. The virtual reality presentation data includes virtual reality presentation video data and virtual reality presentation audio data. The video data for presentation of virtual reality is, for example, video data of a virtual reality object that appears in the virtual reality world. The audio data for virtual reality presentation is audio data generated by the virtual reality object, background music data for creating a virtual reality world, and the like. These video data and audio data are encoded, and are temporarily stored in the virtual reality presentation data storage unit 410 in the form of video encoded data and audio encoded data.

The virtual reality presentation start condition setting unit 420 sets a condition for starting presentation of virtual reality. The virtual reality presentation start condition included in the virtual reality service provision start notification information received by the communication unit 230 is sent by the device-side virtual reality presentation control unit 220 to the virtual reality presentation start condition setting unit 420 to set the virtual reality presentation start condition setting. Unit 420 sets a virtual reality presentation start condition in accordance with the virtual reality presentation start condition included in the virtual reality service provision start notification information. The virtual reality presentation start condition is specifically a video display start condition that is a condition for starting video display for virtual reality presentation or a voice playback start that is a condition for starting audio playback for virtual reality presentation. Conditions are included as needed.

The video display start determination processing unit 120 receives virtual reality presentation video display determination instructions from the device-side virtual reality presentation control unit 220 based on the user status detected by the user status detection unit 110 based on the virtual status. It is determined whether the video display start condition set by the presentation start condition setting unit 420 is satisfied.

Note that the video display start determination processing unit 120 may be configured to perform the determination based on the surrounding state detected by the surrounding state detection unit 310. For example, when the condition that “no other people are around the user” is set as the video display start condition, the video display start determination processing unit 120 is configured to use the line-of-sight camera 311 or the peripheral camera that is the ambient state detection unit 310. It is determined whether or not a person is captured in the video imaged at 312. When the determination is cleared, the video display start determination processing unit 120 instructs the display control unit 130 to start displaying video for virtual reality presentation.

The sound reproduction start determination processing unit 430 receives the sound reproduction start condition set by the virtual reality presentation start condition setting unit 420 when receiving a sound reproduction determination instruction for virtual reality presentation from the device-side virtual reality presentation control unit 220. It is determined whether or not The audio reproduction start determination processing unit 430 may be linked with the video display start determination processing unit 120 or may be determined according to other conditions. The sound reproduction start determination processing unit 430 instructs the sound reproduction control unit 440 to start sound reproduction when it is determined to start sound reproduction for virtual reality.

  The audio reproduction control unit 440 performs audio reproduction control for virtual reality presentation. The audio reproduction control unit 440 includes an audio data reading unit 441 and an audio decoding processing unit 442.

The audio data reading unit 441 reads out the audio encoded data for virtual reality presentation from the virtual reality presentation data storage unit 410 based on the audio reproduction start instruction from the audio reproduction start determination processing unit 430 to generate an audio decoding processing unit. Output to 442.

  The audio decoding processing unit 442 generates audio data of each channel by decoding the audio encoded data read by the audio data reading unit 441, and outputs the decoded audio data to the audio output unit 450.

The sound output unit 450 outputs sound for virtual reality presentation. The audio output unit 450 includes a right speaker 451a and a left speaker 451b, and each speaker performs DA conversion processing on the audio data of each channel decoded by the audio decoding processing unit 442 to convert it into an analog audio signal. Furthermore, it converts into air vibration and emits sound to the outside.

FIG. 13 shows an external perspective view of the virtual reality presentation device 400 from diagonally forward and diagonally rear. As shown in FIG. 13, the virtual reality presentation device 400 takes the form of a head mounted display (HMD) device that a user wears on the head. The virtual reality presentation device 400 includes at least a frame 401 and a front panel 402.

  A frame 401 is a housing of the virtual reality presentation device 400, and includes an information processing processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read only memory), a wireless communication processing processor, A battery etc. are arranged. In addition, a dedicated video processor or the like that draws video as necessary is also arranged. The frame 401 is roughly divided into a right frame 401a, a left frame 401b, and a center frame 401c.

  The light frames 401a and 401b are case members that are connected to the left and right ears of the user, respectively, and speakers 451a and 451b that convert analog electrical signals into air vibrations and output sound are arranged inside one end, respectively. The end is connected to the center frame 401c.

In addition, a user microphone 211 that collects the sound uttered by the user is disposed below the front end inside the right frame 401a, and the surrounding microphone that collects the surrounding sound outside one end of the right frame 401a and the left frame 401b. 212a and 212b are arranged. The sound collected by the user microphone 211 and the surrounding microphones 212a and 212b is converted into a digital sound signal by AD conversion and output to the information processing processor.

  At the ends of the light frames 401a and 401b, inclination sensors 112a and 112b for detecting inclination and acceleration sensors 113a and 113b for detecting acceleration are arranged, respectively. In addition, peripheral cameras 312a and 312b that photograph surrounding scenery are arranged outside the right frame 401a and the left frame 401b, respectively.

  Both ends of the center frame 401c are connected to the right frame 401a and the left frame 401b, respectively, and the front panel 402 is connected below the center frame 401c.

  Near the center and near both ends of the center frame 401c, tilt sensors 1112c, 112d, and 112e that detect tilt and acceleration sensors 113c, 113d, and 113e that detect acceleration are arranged, respectively.

On the user side of the center frame 401c, internal cameras 111a and 111b for photographing a part or all of the facial expression of the user's face are arranged at the positions of the left and right eyes of the user. The internal cameras 111a and 111b capture at least a region including the position of the user's eyes and generate a user facial expression video as user state information.

Line-of-sight cameras 311a and 311b for photographing a scene corresponding to the user's field of view are arranged in front of the center frame 401c and at the positions of the left and right eyes of the user. The line-of-sight cameras 311a and 311b are arranged on the front side of the center frame 401c at positions separated by a distance corresponding to the distance between the left and right eyes of the user.

  The front panel 402 is disposed at a position in front of the user's eyes with the virtual reality presentation device 400 worn on the head. In the virtual reality presentation device 400, the front panel 402 is a transmissive display, and a right display panel 141a and a left display panel 141b, which are display areas for displaying images respectively, are arranged in front of the right eye and in front of the left eye.

  FIG. 14 is a cross-sectional view of the vicinity of the light display panel 141a of the virtual reality presentation device 400. A front panel 402 is disposed below the center frame 401c, and a light source 142 is disposed below the center frame 401c.

Visible light emitted from the light source 142 is irradiated to the front panel 402 side. The front panel 402 has a configuration in which the light guide plate 143 and the LCD 144 are combined. The visible light emitted from the light source 142 is reflected by the light guide plate 143 to the user side and transmitted through the display-controlled LCD 144 to present virtual reality. The video for is displayed.

The virtual reality presentation device 400 may be configured as shown in FIG. In the virtual reality presentation device 400 illustrated in FIG. 15, the LCD 144 is disposed between the light source 142 and the light guide plate 143. Therefore, since the light from the outside is irradiated to the eyes of the user without passing through the LCD 144, it is possible to prevent the brightness of the external scenery from becoming dark.

Moreover, as shown in FIG. 16, LCD145 which controls the irradiation amount of the external light which is the light from the outside may be arrange | positioned. Even if the image of the virtual reality object is displayed on the LCD 144, the light from the outside world is mixed and the image of the virtual reality object is seen through, so that the external light at the position of the virtual reality object is blocked on the LCD 145. It is possible to prevent the real object from being seen through.

Note that the position of the light guide plate 143 is better when the light guide plate 143 is arranged at a position where external light incident on the eyes of the user does not pass as shown in FIG. In the configuration of FIG. 17, the contrast of external light can be further improved.

Further, as shown in FIG. 18, an LCD 145 for controlling external light may be arranged. The external light passes through the LCD 145 and reaches the user's eyes, while the light emitted from the light source 142 passes through the LCD 144 and is reflected by the light guide plate 143 to reach the user's eyes.

Any LCD 145 may be used as long as it can control the amount of light transmitted from the outside by voltage control. For example, it is possible to use a display that changes the crystal direction by applying a voltage to increase irregular reflection.

Further, as shown in FIG. 19, a projection mechanism can be used as the display unit 140. A projection mechanism is arranged as a display unit 140 inside a center frame 401 c located at the top of the front panel 402.

  After the white light emitted from the light source 142 disposed inside the center frame 401c is collected by the lens group 146a, the polarization component is adjusted in one direction by the polarizing converter (PLC) 146b. The light is reflected in the vertical direction by the first reflecting mirror 146c. In the white light reflected by the first reflecting mirror 146c, only the red light component (R component) is reflected in the horizontal direction by the R reflecting dichroic mirror 146d, and the light of other components is transmitted through the R reflecting dichroic mirror 146d. .

  From the light transmitted through the R reflecting dichroic mirror 146d, only the green light component (G component) is reflected in the horizontal direction by the G reflecting dichroic mirror 146e, and the remaining blue light component (B component) passes through the G reflecting dichroic mirror 146e. To Penetrate.

  The B component light transmitted through the G reflecting dichroic mirror 146e is reflected in the horizontal direction by the second reflecting mirror 146f, further reflected vertically upward by the third reflecting mirror 146g, and then transmitted through the first liquid crystal panel 146h and then cross dichroic. It enters the prism 146j.

  On the other hand, the R component light reflected by the R reflecting dichroic mirror 146d is reflected in the vertical direction by the fourth reflecting mirror 146k, passes through the second liquid crystal panel 146m, and then enters the cross dichroic prism 146j. The G component light reflected by the G reflecting dichroic mirror 146e is transmitted through the third liquid crystal panel 146n and then incident on the cross dichroic prism 146j.

  The cross dichroic prism 146j combines the three light components R, G, and B that have been separated and passed through the liquid crystal panel, and reflects them in the horizontal direction. The combined light output from the cross dichroic prism 146j is reflected in the vertical direction by the galvano mirror 146p and is irradiated to the front panel 402 from the irradiation window 146q. The galvanometer mirror 146p reflects the combined light output from the cross dichroic prism 146j while scanning in the horizontal direction (y direction).

  A light guide plate 143 is disposed below the irradiation window 146q in the front panel 402, and the light reflected in the user direction by the light guide plate 143 is refracted by the front panel 402 to become parallel light. Is incident on the inside. Since light is manipulated in the horizontal direction by the galvanometer mirror 146p, the light reflected by the light guide plate 143 is scanned in the vertical direction (z direction).

  Note that each member in FIG. 19 has a configuration that is long and thin in the x direction, that is, in the direction perpendicular to the paper surface, and collectively displays a line in the x direction in the image for the right eye. The row of images is scanned in the y direction by the galvano mirror 146p and sequentially displayed, so that a two-dimensional image in the xz plane is displayed on the front panel 402. Similarly, the same projection mechanism is arranged at the upper part of the left eye, and a left-eye image is displayed in front of the left eye of the front panel 402. In this way, the display unit is formed by the right-eye projection mechanism and the left-eye projection mechanism. Note that the left-eye projection mechanism and the right-eye projection mechanism may be shared.

Further, as shown in FIG. 20, the virtual reality presentation device 400 may be a non-transparent HMD device. In this case, an image obtained by synthesizing the image for virtual reality presentation with the image captured by the visual line camera 311 is displayed on the display panel 140.

In this case, at the display position of the virtual reality object, two videos are combined and displayed on the display panel with the video captured by the line-of-sight camera as the lower layer and the video of the virtual reality object as the upper layer.

For example, at predetermined coordinates (x, y) at which the virtual reality object is displayed, the pixel data of the video of the virtual reality object generated by the video generation processing unit 2140 is (Y1, U1, V1). It is assumed that the pixel data of the video imaged by the visual line camera 311 is (Y2, U2, V2). When the placement position of the virtual reality object is closer to the user than the object captured in the video captured by the line-of-sight camera 311, the pixel data represented by the coordinates on the display panels 141a and 141b is (Y1, U1, V1). On the other hand, when the object shown in the video imaged by the line-of-sight camera 311 is closer to the user side than the arrangement position of the virtual reality object, the pixel data represented by the coordinates on the display panels 141a and 141b is (Y2, U2 , V2).

As described above, the virtual reality presentation device 400 according to the fourth embodiment includes the video display unit and the audio reproduction unit as the unit for presenting the virtual reality. Since the virtual reality world is presented as an illusion of the user's vision and hearing, the user can enter the virtual reality world more deeply.

Note that when an object or person suddenly appears in the field of view, the user feels a great sense of discomfort, but when a sound is suddenly heard from the front, the user does not feel that something discomfort. Therefore, the sound reproduction start determination processing unit 430 may be configured to determine the sound reproduction start by a determination process that is not based on the state of the user.

  FIG. 21 is a block diagram illustrating a configuration of a virtual reality presentation system 4000 according to the fourth embodiment. The virtual reality presentation system 4000 includes the virtual reality presentation device 400, the virtual reality service providing server 4100, the voice analysis server 2200, the virtual reality content storage DB 2300, and the contractor information storage DB 3200.

The virtual reality content storage database 2300 stores audio data for virtual reality presentation in addition to the model data of the virtual reality object as virtual reality content. The sound data for virtual reality presentation includes sound generated by the virtual reality object associated with each virtual reality object, sound effects and background music for creating a virtual reality world, and the like.

The virtual reality service providing server 4100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, an image analysis processing unit 3110, and a virtual space setting unit 3130. A virtual reality object placement unit 3140, a virtual reality content reading unit 4110, and a playback sound generation processing unit 4120.

The playback sound generation processing unit 4120 generates sound data for virtual reality presentation that is played back by the sound playback processing unit 440 of the virtual reality presentation device 400. The reproduction sound generation processing unit 4120 includes an auditory point control unit 4121, a sound distribution processing unit 4122, and a sound coding processing unit 4123.

  The auditory point control unit 4121 controls the auditory point in the virtual space corresponding to the positions of the left and right ears of the user based on the user state information received by the communication unit 2110.

The sound distribution processing unit 4122 generates sound data for each channel by distributing sound data for virtual reality presentation to a plurality of channels. The audio distribution processing unit 4122 includes a virtual reality object arrangement position information acquisition unit 4122a, a relative positional relationship calculation unit 4122b, and a channel-specific audio data generation processing unit 4122c.

The virtual reality object arrangement position information acquisition unit 4122a, when the audio data for virtual reality read by the virtual reality content reading unit 4110 is a sound emitted from the virtual reality object arranged in the virtual space, The position coordinates are acquired from the virtual reality object placement unit 3140 using the position where the real object is placed as a sound generation source.

The audio data read by the virtual reality content reading unit 4110 is given an identification number for identifying the corresponding virtual reality object. The virtual reality object arrangement position information acquisition unit 4122a is based on the identification number of the virtual reality object associated with the audio data read by the virtual reality content reading unit 4110, and is identified by the identification number. Is obtained from the virtual reality object placement unit 3130. The virtual reality object arrangement position information acquisition unit 4122a outputs position information indicating the position coordinates of the virtual reality object acquired from the virtual reality object arrangement unit 3140 to the relative position relationship calculation unit 4122b.

The relative positional relationship calculation unit 4122b includes the position coordinates of the auditory point corresponding to the position of the user's ear obtained by the auditory point control unit 4121 and the arrangement of the virtual reality object acquired by the virtual reality object arrangement position information acquisition unit 4122a. A relative positional relationship is calculated based on the position coordinates indicating the position. Specifically, the relative positional relationship calculation unit 4122b calculates a vector P from the position coordinates of the virtual reality object that is the sound generation source to the position coordinates of the auditory point that is the sound collection point. The relative positional relationship calculation unit 4122b outputs the vector P, which is the calculated relative positional relationship, to the channel-specific audio data generation processing unit 4122c.

The channel-specific audio data generation processing unit 4122c uses the virtual reality presentation audio data read by the virtual reality content reading unit 4110 based on the vector information that is the relative position calculated by the relative positional relationship calculation unit 4122b. By distributing to a plurality of audio channels, audio data for each channel is generated.

The audio reproduction processing unit 440 of the virtual reality presentation device 400 has a configuration capable of reproducing audio using a plurality of channels such as 5.1Ch, 6.1Ch, 7.1Ch, and the channel-specific audio data generation processing unit 4122c includes: The audio data read by the virtual reality content reading unit 4110 is generated for each channel according to the number of Ch supported by the virtual reality presentation device 400. The channel-specific sound data generation processing unit 4122c sets the sound level of each channel of the read sound data so that the sound associated with the virtual reality object is emitted from the position of the virtual reality object. By adjusting, the sound has directivity and the sound data of each channel is generated.

The speech encoding processing unit 4123 encodes the channel-specific speech data distributed to each channel by the speech distribution processing unit 4122 using a predetermined speech encoding method such as CS-ACELP (Conjugate-Structure Algebraic Code Excited Linear Prediction). And converted into encoded audio data. The audio data encoded by the audio encoding processing unit 4123 is transmitted from the communication unit 2110 to the virtual reality presentation device 400.

As described above, according to the virtual reality presentation system according to the fourth embodiment, virtual reality content is reproduced so as to cause an illusion to the user's vision and auditory sense, thereby allowing the user to naturally experience the virtual reality world. It becomes possible to get in.

In the above description, the content for hearing and vision is reproduced as a virtual reality, but a configuration including means for stimulating the sense of smell and touch may be used. For example, an odor storage unit that seals and stores a plurality of odor source substances in the virtual reality presentation device 400 and an odor by opening some of the odor storage units based on control from the virtual reality management control unit 460 The structure provided with the odor mixing part which mixes, and the odor discharge | release part which discharge | releases the mixed odor may be sufficient.

<Embodiment 5>
In the virtual reality presentation systems according to the first to fourth embodiments, the discontinuity at the time of transition from the real world to the virtual reality world can be reduced by controlling the start timing of the virtual reality presentation. Can enter the virtual reality world without perceiving the switch from the real world to the virtual reality world.

Here, when the presentation of the virtual reality world ends and returns to the real world, the user has perceived the virtual reality world from the discontinuity when changing to the virtual reality world and the real world. You will recognize that. Here, depending on the type of virtual reality service provided to the user, the discontinuity when returning from the virtual reality world to the real world is suppressed, and it is recognized that the user was temporarily in the virtual reality world. It may be preferable to control so that it cannot be performed. For example, in the case of a virtual reality service that presents the virtual reality world multiple times in a time series, it is preferable to transition from the virtual reality world to the real world continuously because of the story nature. . The virtual reality presentation system according to the fifth embodiment is characterized in that the virtual reality presentation end timing is controlled.

FIG. 22 is a block diagram illustrating a configuration of a virtual reality presentation device 500 according to the fifth embodiment. The virtual reality presentation device 500 includes a user state detection unit 110, a video display start determination processing unit 120, a display control unit 130, a display unit 140, a voice input unit 210, a device-side virtual reality presentation control unit 220, Communication unit 230, ambient condition detection unit 310, virtual reality presentation data storage unit 410, virtual reality presentation start condition setting unit 420, audio reproduction start determination processing unit 430, audio reproduction processing unit 440, audio output Unit 450, video display end determination processing unit 510, virtual reality presentation end condition setting unit 520, and audio reproduction end determination processing unit 530.

The communication unit 230 receives the virtual reality service provision end notification information for notifying the end of provision of the virtual reality service in addition to the virtual reality service provision start notification information for notifying the start of provision of the virtual reality service, and presents the virtual reality service provision end notification information Output to the controller 220.

Upon receiving the virtual reality service provision end notification information, the device-side virtual reality presentation control unit 220 instructs the virtual reality presentation end condition setting unit 520 to set the virtual reality presentation end condition. Specifically, the virtual reality presentation end condition is set based on the virtual reality presentation end condition included in the virtual reality service provision end notification information. In addition, it instructs the video display end determination processing unit 510 and the audio reproduction end determination processing unit 530 to start the corresponding determination processing.

The virtual reality presentation end condition setting unit 520 sets a virtual reality presentation end condition based on an instruction from the device-side virtual reality presentation control unit 220. The virtual reality presentation end condition includes a video display end condition for ending the video display and an audio playback end condition for ending the audio playback.

The video display end determination processing unit 510 starts a determination process for determining whether to end the display of the virtual reality video currently displayed on the display unit 140 in accordance with an instruction from the device-side virtual reality presentation control unit 220. The video display end determination processing unit 510 determines whether the user state detected by the user state detection unit 110 and the surrounding state detected by the surrounding state detection unit 310 satisfy the virtual reality presentation end condition. As a result of the determination process, when the virtual reality presentation end condition is satisfied, the video display end determination processing unit 510 instructs the display control unit 130 to end the display of the virtual reality video. The display control unit 130 performs control to end the display of the virtual reality presentation video on the display unit 140 based on the instruction.

The audio reproduction end determination processing unit 530 starts a determination process as to whether or not to end the reproduction of the virtual reality sound currently output from the audio output unit 140 in accordance with an instruction from the device-side virtual reality presentation control unit 220. . The audio reproduction end determination processing unit 530 performs a determination process as to whether the user state detected by the user state detection unit 110 and the surrounding state detected by the surrounding state detection unit 310 satisfy the virtual reality presentation end condition. If the virtual reality presentation end condition is satisfied as a result of the determination process, the audio reproduction end determination processing unit 530 instructs the audio reproduction control unit 440 to end the display of the virtual reality video. The sound reproduction control unit 440 performs control to end the sound reproduction processing for virtual reality presentation based on the instruction.

In addition, it is good also as a structure by which control regarding the said audio | voice reproduction | regeneration is not performed. In this case, the audio reproduction start determination processing unit 430 and the audio reproduction end determination processing unit 530 are excluded from the configuration.

FIG. 23 is a block diagram illustrating a configuration of a virtual reality presentation system 5000 according to the fifth embodiment. The virtual reality presentation system 5000 includes the virtual reality presentation device 500, the virtual reality service providing server 5100, the save data storage database 5200, the voice analysis server 2200, the virtual reality content storage DB 2300, and the contractor information storage DB 3200. .

The virtual reality content storage DB 2300 stores virtual reality content used for each of a plurality of types of virtual reality services. As shown in FIG. 24, the virtual reality content includes a virtual reality control program executed for presentation of virtual reality in addition to model data and audio data of the virtual reality object.

Virtual reality control programs are roughly classified into a virtual reality main control program, a virtual reality sub-control program, a virtual reality object personality control program, and a virtual reality object behavior control program. FIG. 25 shows the hierarchical structure of the virtual reality control program.

The virtual reality object behavior control program is a program for controlling the behavior of the virtual reality object arranged in the virtual space, and is read by the virtual reality object personality control program. For example, when the virtual reality object is a human object, the virtual reality object behavior control program A performs a control such that the joint point of the right knee is slightly bent and the texture data of the mouth is gradually changed to the smile texture data. It is a program. On the other hand, the virtual reality object behavior control program N performs control such that the hip joint point is bent so as to slide, and the right arm and the right elbow joint point are rotated by a predetermined amount to project the right arm forward. By executing the behavior control program, the appearance shape of the object arranged in the virtual space changes as the frame progresses.

The virtual reality object personality control program is a program read by the virtual reality sub-control program, and controls the correspondence between the input and output for the virtual reality object arranged in the virtual space.

The virtual reality sub-control program is a program that is appropriately called by the virtual reality main control program, and corresponds to the provided virtual reality service. The virtual reality sub-control program reads the virtual reality object used for the virtual reality service being provided in the virtual space, or directly reads out the virtual reality object behavior control program according to a preset order, Control is performed to perform a predetermined behavior.

The virtual reality main control program is a program corresponding to each virtual reality service, and is a program executed first when the provision of the virtual reality service is started. By executing the virtual reality main control program, the virtual space setting and environment setting for the virtual reality service corresponding to the program are performed.

The virtual reality service providing server 5100 includes a communication unit 2110, a voice context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, an image analysis processing unit 3110, and a virtual space setting unit 3130. A virtual reality object placement unit 3140, a virtual reality content reading unit 4110, a reproduction sound generation processing unit 4120, and a virtual reality object behavior control unit 5110.

The server-side virtual reality presentation control unit 2130 includes a virtual reality service provision availability determination unit 2131, a virtual reality service determination unit 2132, a virtual reality service control unit 2133, and a notification information generation processing unit 2134.

When the voice context acquired by the voice context acquisition unit 2120 requests to start providing the virtual reality service, the virtual reality service provision availability determination unit 2131 stores the contractor stored in the contractor information storage DB 3200. By referring to the information, it is determined whether the requested virtual reality service can be provided. When it is determined that the virtual reality service can be provided, the virtual reality service provision availability determination unit 2131 issues a virtual reality service determination instruction to be provided to the virtual reality service determination processing unit 2132, and a notification information generation processing unit An instruction to generate virtual reality service provision start notification information is issued to 2134.

The virtual reality service determination processing unit 2132 determines a virtual reality service to be provided to the user based on an instruction from the virtual reality service provision availability determination unit 2131. Here, the virtual reality service determination processing unit 2132 determines from which position the virtual reality service is provided based on the save data of the virtual reality service of the user stored in the save data storage database 5200. . For example, when the virtual reality service is a virtual reality service having a story property, save data indicating a portion that has already been provided is stored in the save data storage database 5200, and it is necessary to restart from the position where the previous provision ended. Because there is. The virtual reality service determination processing unit 2132 outputs information indicating a restart position in the virtual reality service indicated by the determined virtual reality service and the save data to the virtual reality service control unit 2133.

The virtual reality service control unit 2133 receives the output from the virtual reality service determination processing unit 2132 and executes various controls for providing the virtual reality service. The virtual reality service control unit 2133 outputs, to the virtual reality content reading unit 4110, a virtual reality content reading unit 4110 that is a virtual reality service determined by the virtual reality service determination processing unit 2132 and that is related to the resume position indicated by the save data. . In addition, the virtual reality service control unit 2133 outputs a virtual space setting instruction corresponding to the virtual reality service to the virtual space setting unit 3130.

In addition, the virtual reality service control unit 2133 is based on the audio context acquired by the audio context acquisition unit 2120 and the image analysis result obtained by the image analysis processing unit 3310, the virtual reality main control program related to the virtual reality service, The virtual reality service is continuously provided by executing the virtual reality service sub-control program.

Further, the virtual reality service control unit 2133 requests the termination of the virtual reality service currently provided by the user, the voice context acquired by the voice context acquisition unit 2120, the image analysis result obtained by the image analysis processing unit 3310, or the like. If there are other termination conditions, the virtual reality service provision end notification information generation instruction is issued to the notification information generation processing unit 2134.

The notification information generation processing unit 2134 generates virtual reality service provision start information and virtual reality service provision end information, respectively, based on an instruction from the virtual reality service provision availability determination unit 2131 and an instruction from the virtual reality service control unit 2133. And output to the communication unit 2110. The communication unit 2110 transmits the output notification information to the virtual reality presentation device 500.

The virtual reality object behavior control unit 5110 controls the behavior of the virtual reality object arranged in the virtual space. The virtual reality object behavior control unit 5110 executes the virtual reality object behavior control program read from the virtual reality content storage database 2300 by the virtual reality content reading unit 4110, so that the virtual reality object arranged in the virtual space is displayed. Control behavior. By the virtual reality object behavior control unit 5110, the video generation processing unit 2140 generates a two-dimensional video of the virtual reality object whose appearance shape changes in the virtual space, and finally the display unit 140 of the virtual reality presentation device 500. Is displayed.

As described above, according to the virtual reality presentation system according to the fifth embodiment, the virtual reality presentation end timing is controlled based on the state of the user, so that the user can change from the virtual reality world to the real world. Since it is possible not to recognize the transition, it is possible to prevent the user from realizing that the virtual reality world was the virtual reality world.

Note that the virtual reality presentation start condition and the virtual reality presentation end condition may be provided with the same condition. For example, it may be determined that the presentation of virtual reality ends when the user's eyes are closed for 3 seconds or more, or the presentation of virtual reality is ended when the direction in which the user is facing changes by 90 degrees or more. May be. Further, when the angle at which the user is facing changes by 45 degrees or more, it may be determined that the presentation of the virtual reality is ended.

<Embodiment 6>
In the virtual reality presentation systems according to the first to fifth embodiments, presentation of video or audio that has already been generated and is ready to be presented on the virtual reality presentation device side is started or assimilation is determined. However, in this configuration, video and audio are generated without being satisfied with the presentation start determination condition even though they are sequentially generated, which causes a problem that resource consumption increases. The configuration according to the sixth embodiment allows the user to naturally enter the virtual reality world by controlling discontinuity in switching from the real world sensed by the user to the virtual reality world while suppressing resource consumption. Making it possible.

FIG. 26 is a block diagram illustrating a configuration of a virtual reality presentation system 6000 according to the sixth embodiment. The virtual reality presentation system 6000 includes a virtual reality presentation device 600, a voice analysis server 2200, a contractor information storage DB 3200, a virtual reality content storage DB 2300, and a virtual reality service providing server 6100.

The virtual reality presentation device 600 includes a user state detection unit 110, a voice input unit 210, a display control unit 130, a display unit 140, a device-side virtual reality presentation control unit 220, a communication unit 230, and an ambient condition detection unit. 310, a virtual reality presentation data storage unit 410, an audio reproduction control unit 440, and an audio output unit 450.

The user status information that is the detection result in the user status detection unit 110, the user voice data that is input in the voice input unit 210, the ambient status information that is the detection result in the ambient status detection unit 310, etc. It is transmitted to the providing server 6100.

The virtual reality presentation video data and audio data generated by the virtual reality service providing server 6100 are temporarily stored in the virtual reality presentation data storage unit 410 and sequentially received by the communication unit 230. It is overwritten by video data or audio data.

The device-side virtual reality presentation control unit 220 issues a virtual reality presentation start instruction to the display control unit 130 and the audio reproduction control unit 440 based on the virtual reality service provision start notification information received by the communication unit 230.

Upon receiving the virtual reality presentation start instruction, the display control unit 130 sequentially reads and encodes the virtual reality presentation video data from the virtual reality presentation video data storage area in the virtual reality presentation data storage unit 410. If it is, the decoding process is performed and then the video after the decoding process is displayed on the display unit 140. The display unit 140 displays the virtual reality presentation video decoded by the display control unit 130.

Upon receiving the virtual reality presentation start instruction, the audio reproduction control unit 440 sequentially reads out and encodes the virtual reality presentation audio data from the virtual reality presentation audio data storage area in the virtual reality data storage unit 410. If so, the decoding process is performed and then the audio after the decoding process is played back. The audio output unit 450 outputs audio by converting the audio data reproduced by the audio reproduction control unit 440 into an analog electric signal, further converting it into air vibrations and emitting the sound to the outside.

  The virtual reality service providing server 6100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, an image analysis processing unit 3110, and a virtual space setting unit 3130. A virtual reality object placement unit 3140, a virtual reality content reading unit 4110, a playback sound generation processing unit 4120, a virtual reality object behavior control unit 5110, and a virtual reality presentation start determination processing unit 6110.

  The server-side virtual reality presentation control unit 2130 performs processing for starting provision of a virtual reality service based on the audio context acquired by the audio context acquisition unit 2120. The server-side virtual reality presentation control unit 2130 generates virtual reality service provision start notification information and outputs the virtual reality service provision start notification information to the communication unit 2110. Further, the server-side virtual reality presentation control unit 2130 instructs the virtual space setting unit 3130 to set the virtual space for the virtual reality service, and the virtual reality content related to the virtual reality service provided to the virtual reality content reading unit 4110 is displayed. A read instruction is issued, and an instruction to determine whether to start presenting virtual reality is issued to the virtual reality presentation start determination processing unit 6110.

Upon receiving an instruction from the server-side virtual reality presentation control unit 2130, the virtual reality presentation start determination processing unit 6110 performs virtual reality on the virtual reality presentation device 600 based on the user state information and the surrounding state information received by the communication unit 2110. It is determined whether or not it is possible to start presenting. That is, the virtual reality presentation start determination processing unit 6110 compares the virtual reality presentation start condition, which is a condition for starting the presentation of virtual reality, with the user state indicated by the user state information and the like at the current timing. Even if the virtual reality is presented to the user, it is determined whether or not the discontinuity in the transition from the real world to the virtual reality world sensed by the user can be suppressed to a predetermined reference value or less.

For example, when the condition that “the user's eyes are closed for 4 seconds or more” is set as the virtual reality presentation start condition, the virtual reality presentation start determination processing unit 6110 performs the internal camera included in the input user state information. Based on the video imaged at 111, it is determined whether the user's eyes are continuously closed for 4 seconds or more. The virtual reality presentation start determination processing unit 6110 instructs the virtual reality object placement unit 3140 to place a virtual reality object when the user state satisfies the determination condition. The virtual reality presentation start condition includes a condition that the ambient brightness (illuminance) detected by the ambient condition detection unit 310 is equal to or less than a predetermined reference value, or the user's head has rotated 45 degrees or more. Or the like may be set.

The virtual reality presentation start determination processing unit 6110 issues a virtual reality object placement instruction to the virtual reality object placement unit 3140 when it is determined that presentation of virtual reality is started as a result of the determination.

The virtual reality object placement unit 3140 sets the virtual reality object read by the virtual reality content reading unit 4110 based on the placement instruction from the virtual reality presentation start determination processing unit 6110 (generated) ) Place in virtual space. That is, even if the virtual reality object used for the virtual reality service is read from the virtual reality content reading unit 4110, the virtual reality object placement unit 3140 determines that the virtual reality presentation start determination processing unit 6110 starts the virtual reality presentation. The virtual reality object is not placed in the virtual space until it is done.

  The display video generation processing unit 2140 generates a virtual reality presentation video by drawing a virtual reality object arranged in the virtual space from the viewpoint position and the line-of-sight direction indicated by the user state information. Here, since the virtual reality object is not arranged in the virtual space until the virtual reality presentation start determination processing unit 6110 determines to start presenting virtual reality, the video generated by the display video generation processing unit 2140 is empty. It is an image of. That is, the video of the virtual reality object is not displayed on the display unit 140 of the virtual reality presentation device 600.

As described above, by providing the virtual reality service providing server with a function for determining the start of virtual reality presentation, video data of a virtual reality object that is not actually displayed is transmitted to the virtual reality presentation device. Absent. Therefore, unnecessary power consumption of wireless resources between the virtual reality presentation device and the virtual reality service providing server can be suppressed, and battery power consumption can be suppressed by suppressing unnecessary processing in the virtual reality presentation device.

Note that the virtual reality presentation system 6000 according to the sixth embodiment may have the configuration of FIG. In the virtual reality presentation system 6000 illustrated in FIG. 27, the virtual reality presentation start determination processing unit 6110 performs a determination process on whether to start presenting virtual reality based on a determination instruction from the server-side virtual reality presentation control unit 2130. Do.

When it is determined by the virtual reality presentation start determination processing unit 6110 to start presenting virtual reality, that is, when it is determined that the user state or the like that satisfies the virtual reality presentation condition is satisfied, the virtual reality presentation start determination The processing unit 6110 instructs the server-side virtual reality presentation control unit 2130 to start presentation of virtual reality. Based on the instruction from the virtual reality presentation start determination processing unit 6110, the server-side virtual reality presentation control unit 1230 instructs the virtual reality content reading unit 4110 to read the virtual reality content related to the provided virtual reality service.

Based on the read instruction, the virtual reality content read unit 4110 reads the virtual reality object and the virtual reality presentation sound data from the virtual reality content storage DB 2300. The virtual reality content reading unit 4110 outputs the read virtual reality object to the virtual reality object placement unit 3140, and outputs the virtual reality presentation audio data to the reproduction audio generation processing unit 4120. The virtual reality object placement unit 3140 places the virtual reality object read by the virtual reality content reading unit 4110 in the virtual space set by the virtual space setting unit 3130. The display video generation processing unit 2140 draws the video of the virtual reality object arranged in the virtual space based on the user state, and generates video data for virtual reality presentation. The reproduction audio generation processing unit 4120 generates audio data to be reproduced by the virtual reality presentation device 600 based on the virtual reality presentation audio data read by the virtual reality content reading unit 4110.

As described above, based on the result of the virtual reality presentation start determination, the virtual reality object is read from the database and arranged in the virtual space, and the display video generation processing unit 2140 until the determination is cleared. Since the empty video is generated, the video of the virtual reality object is not displayed on the display unit 140 of the virtual reality presentation device 600. Therefore, the same effect can be produced.

The virtual reality presentation system 6000 according to the sixth embodiment may have the configuration shown in FIG. In the virtual reality presentation system 6000 illustrated in FIG. 28, the virtual reality presentation start determination processing unit 6110 performs a determination process on whether to start presenting virtual reality based on a determination instruction from the server-side virtual reality presentation control unit 2130. Do. When the virtual reality presentation start determination processing unit 6110 determines to start the presentation of virtual reality, the virtual reality presentation start determination processing unit 6110 instructs the display video generation processing unit 2140 and the reproduction sound generation processing unit 4120 to start the presentation of virtual reality.

Based on the virtual reality presentation start instruction from the virtual reality presentation start determination processing unit 6110, the display video generation processing unit 2140 draws a virtual reality object arranged in the virtual space and generates a video for virtual reality presentation. . The reproduction sound generation processing unit 4120 generates sound for virtual reality presentation based on the virtual reality presentation start instruction from the virtual reality presentation start determination processing unit 6110.

As described above, when the state of the user who receives the virtual reality presentation satisfies the virtual reality presentation condition that is a condition for receiving the virtual reality presentation, the virtual reality presentation video and audio are generated. Unnecessary processing in the virtual reality service providing server can be reduced, and resource consumption can be suppressed.

<Embodiment 7>
The virtual reality presentation system according to the sixth embodiment is characterized in that the user can enter the virtual reality world without a sense of incongruity by controlling the conditions for starting the presentation of the virtual reality. Here, depending on the virtual reality service provided to the user, not only a single virtual reality object is displayed, but a plurality of virtual reality objects are sequentially read and arranged in the virtual space as time progresses. May take the form.

In this case, when a new virtual reality object suddenly appears in the user's field of view, the user will perceive that the virtual reality object has instantaneously moved and appeared in front of him. However, since such instantaneous movement is a phenomenon that is physically unlikely in the real world, the object (person) that appears in front of the eyes is not a real world object, but a virtual reality object. It becomes impossible to enter into the virtual reality world.

FIG. 29 shows the relationship between the virtual reality presentation video before the virtual reality presentation, after the virtual reality presentation is started, and after the virtual reality object is added, and the virtual reality world sensed by the user. The scenery at the user's line of sight before presenting virtual reality is the scenery of the room where two sofas are placed (a). Here, the user requests the provision of a virtual reality service called “Easy safari experience in the room”, and the video of the lion, which is a virtual reality object, is generated by the virtual reality service providing server, and the video display start condition (virtual reality presentation) When the condition (condition) is satisfied, the image of the lion is displayed on the display panel as the display unit (b).

Here, it is assumed that a virtual reality object of a dog, which is a second animal, is added as time passes. Here, it is assumed that one video data is generated by drawing the virtual reality object of the lion already arranged in the virtual space and the virtual reality object of the newly added dog (c). In this case, the user is perceived as if the lion stays there continuously, while the dog is perceived as if it has moved in a moment, and the dog is not a real dog, but a virtual reality. It will be clearly recognized that the video.

In the virtual reality presentation system according to the seventh embodiment, when displaying a video of a new virtual reality object, the display is performed while suppressing discontinuity between the virtual reality world and the real world. To do.

FIG. 29 is a block diagram illustrating a configuration of a virtual reality presentation system 7000 according to the seventh embodiment. The virtual reality presentation system 7000 includes a virtual reality presentation device 700, a virtual reality service providing server 7100, a voice analysis server 2200, a virtual reality content storage DB 2300, and a contractor information storage DB 3200.

The virtual reality service providing server 7100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, an image analysis processing unit 3110, and a virtual space setting unit 3130. A virtual reality object placement unit 3140, a virtual reality content reading unit 4110, a reproduction sound generation processing unit 4120, and a virtual reality object behavior control unit 5110. The virtual space setting unit 3130 includes a virtual space generation unit 3131 and a virtual space update management unit 3132.

The virtual space generation unit 3131 generates a virtual space for the virtual reality service when the server-side virtual reality presentation control unit 2120 determines to start a new virtual reality service. Here, the virtual space generation unit 3131 generates a plurality of layers in the virtual space.

The virtual space update management unit 3132 arranges in the virtual space according to the new virtual reality object placement process performed by the virtual reality object placement unit 3140 and the virtual reality object behavior control process performed by the virtual reality object behavior control unit 5110. The position and arrangement relationship of the virtual reality object being updated are updated.

The virtual space management unit 3132 manages the virtual reality object arranged in the virtual space by dividing it into a plurality of layers. For example, it is assumed that three virtual reality objects identified by the identification numbers of VO001, VO002, and VO003 are arranged in the virtual space. For example, the virtual space management unit 3132 associates VO001 and VO002 with the virtual space layer L01, and associates VO003 with the layer 002.

The virtual reality object placement unit 3140 places the virtual reality object newly read by the virtual reality content reading unit 4110 in the virtual space in association with the layer for the new virtual reality object set in the virtual space.

The display video generation processing unit 2140 separates virtual reality objects arranged in the virtual space for each layer and performs drawing processing to generate video data for each layer. The display video generation processing unit 2140 encodes the generated video data for each layer and outputs the encoded video data to the communication unit 2110. The communication unit 2110 transmits the video encoded data for each layer generated by the display video generation processing unit 2140 to the virtual reality presentation device 700.

FIG. 31 is a block diagram showing a configuration of the virtual reality presentation device 700. The virtual reality presentation device 700 includes a user state detection unit 110, a voice input unit 210, a display control unit 130, a display unit 140, a device-side virtual reality presentation control unit 220, a communication unit 230, and an ambient condition detection unit. 310, a virtual reality presentation data storage unit 410, an audio reproduction control unit 440, an audio output unit 450, an image display start condition setting unit 710, and an image display determination processing unit 720.

The video display condition setting unit 710 sets conditions for displaying video for each layer in the virtual space. For example, the video display start condition setting unit 710 sets a display start condition and a display end condition for a layer that includes a virtual reality object newly arranged in the virtual space.

The video display determination processing unit 720 determines whether to display the video of each layer based on the user status detected by the user status detection unit 110 and the ambient status detected by the ambient status detection unit 310. The video display determination processing unit 720 compares and determines whether the video display conditions set for each layer satisfy parameters such as the user state and the surrounding situation, and outputs the determination result to the display control unit 130.

The display control unit 130 includes a video data reading unit 131, a video decoding processing unit 132, and a video composition processing unit 133. The video data reading unit 131 reads from the virtual reality presentation data storage unit 410 video encoded data of a layer determined to be displayed based on the determination result in the video display determination processing unit 720.

The video decoding processing unit 132 performs a decoding process on the video encoded data of each layer read by the video data reading unit 131, and acquires video data of each layer after decoding.

The video composition processing unit 133 synthesizes the decoded video of each layer, acquires a display video, and outputs it to the display unit 140. The display unit 140 displays the video synthesized by the video synthesis processing unit 133.

As described above, in the virtual reality presentation system according to the seventh embodiment, the virtual reality object is arranged in the virtual space of any layer from among the virtual spaces of the plurality of layers generated by the virtual space generation processing unit 3131. A virtual reality presentation video is generated for each layer. Furthermore, a process for determining whether or not to display the virtual reality presentation video of the layer in which the virtual reality object is newly arranged is performed, and when the determination is cleared, the virtual reality presentation video of another layer is combined. Displayed. With this configuration, when a new virtual reality object is presented to the user in addition to the video of the virtual reality object already displayed, the display unit 140 is displayed when a predetermined condition is satisfied according to the user's state or the like. Therefore, the user is likely to have an illusion that the added virtual reality object is a real object (person), and the discontinuity between the real world and the virtual reality world can be suppressed.

FIG. 32 shows a virtual reality presentation image and a virtual reality world sensed by the user in the real world where two sofas are placed. A lion's virtual reality object is arranged in the layer 01 of the virtual space, and an image of the layer 01 in which the virtual reality object is drawn is generated and displayed on the display unit 140 (b).

Here, when a virtual reality object of a dog is newly arranged in the virtual space, the virtual reality object of the dog is arranged in a layer 02 different from the layer 01 in which the virtual reality object has already been arranged. An actual presentation video is generated (c). When the display start condition for the layer 02 is satisfied, a video in which the layer 01 and the layer 02 are combined is displayed on the display unit 140.

In this way, since the video is generated and displayed for each layer, it is easy to control the presentation of virtual reality.

33 and 34 are more preferable. In the virtual reality service providing server 7100 according to FIG. 33, the virtual reality update management unit 3132 generates layer order information indicating the order of each layer. The virtual reality update management unit 3132 includes a viewpoint position corresponding to the position of the user's eyes in the virtual space specified based on the user state information received by the communication unit 2110, and a virtual reality object placement unit 3140 arranged by the virtual reality update management unit 3132. Based on the position in the virtual space of the virtual reality object controlled by the real object behavior control unit 5110, the order of the layers including each virtual reality object is obtained to generate layer order information. The layer order information is transmitted from the communication unit 2110 to the virtual reality presentation device 700.

In the virtual reality presentation device 700, the display control unit 130 includes a layer order management unit 134. The layer order information received by the communication unit 230 is output to the layer order management unit 134. The layer order management unit 134 manages the layer order based on the layer order information received by the communication unit 230. The video composition processing unit 133 performs control to synthesize videos of a plurality of layers based on the order of the layers managed by the layer order management unit 134 and display them on the display unit 140.

In this way, layer order information indicating the layer order is transmitted to the virtual reality presentation device 700, and in accordance with the layer order indicated by the layer order information, the images of the respective layers are combined to generate a composite video for virtual reality presentation. By generating, it is possible to appropriately present the video of the virtual reality object to the user.

Note that it is more preferable to have a function of moving the virtual reality object between the layers. The virtual space update management unit 3132 performs a process of moving virtual reality objects respectively arranged in a plurality of layers to other layers. By performing this process, you can rearrange virtual reality objects that are arranged separately on multiple layers into a small number of layers, or select some virtual reality objects from the layer on which multiple virtual reality objects are arranged. It is good also as a structure which extracts and arrange | positions to another layer.

Since the display video generation processing unit 2140 generates a virtual reality presentation video by performing rendering processing for each layer, resources required for the rendering processing increase as the number of layers increases. Therefore, by performing a process of collecting virtual reality objects that can be grouped into the same layer, unnecessary layers can be deleted and the number of layers can be reduced.

In this case, the determination result in the video display determination processing unit 720 may be transmitted to the virtual reality service providing server 7100. The virtual space update management unit 3132 performs update processing for moving the virtual reality object to another layer based on the determination result. For example, when the video display determination processing unit 720 determines that the video display determination processing unit 720 starts to display the video of the virtual reality object newly arranged in the virtual space, the virtual space update management unit 3132 sets the layer for the new virtual reality object. A process of moving the arranged virtual reality object to the layer where the existing virtual reality object is arranged is performed. With this configuration, the virtual reality objects whose display has been started are drawn together with the virtual reality objects so far.

If the virtual space update management unit 3132 is configured to transfer the virtual reality object related to the end of the presentation of the virtual reality to the end layer, only the virtual reality object can be ended according to the state of the user. Is more preferable. Also in such a case, the virtual space update management unit 3132 generates layer order information and transmits it to the virtual reality presentation device 700. In addition, the virtual space update management unit 3132 transmits a display end condition regarding the layer, and the video display condition setting unit 710 sets the display end condition. The video display determination processing unit 720 ends the display of the layer by comparing the video display end condition of the end layer set in the video display condition setting unit 710 with the detected user state and the like. Judge whether to do. As a result of the determination, when it is determined that the user state or the like satisfies the video display end condition, that is, the display of the layer is to be ended, the video composition processing unit 133 terminates the process from among a plurality of layers to be combined. The determined layer is not synthesized, and control is performed to synthesize videos of other layers and display them on the display unit 140. In addition, the video display determination processing unit 720 generates display end notification information indicating that the display of the layer has ended. The communication unit 230 transmits the display end notification information to the virtual reality service providing server 7100.

In the virtual reality service providing server 7100 that has received the display end notification information, the virtual space update management unit 3132 displays a virtual reality object arranged in the layer for which the display indicated by the display end notification information is ended from the virtual space. Process to delete. By this processing, the erasure of the virtual reality object arranged in the virtual space is completed.

  In the above description, the configuration in which the virtual reality object placed in the virtual space is associated with any one of the three layers of the normal layer, the new placement layer, and the display end layer has been described. Is not to be done. That is, in the above description, the virtual reality object placement unit 3140 places the virtual reality object newly read by the virtual reality content reading unit 4110 on the new placement layer, and the virtual reality presentation device 700 side displays the video of the layer. When the display is started after being synthesized with the video for the normal layer, the virtual space update management unit 3132 performs a process of transferring the virtual reality object arranged in the normal layer to the normal layer. If there is a virtual reality object related to the end of display, the virtual space update management unit 3132 performs a process of moving the virtual reality object related to the end arranged in the normal layer to the end layer. When display end notification information is received from the virtual reality presentation device 700 in this state, the virtual reality object related to the end arranged in the end layer is deleted from the virtual space.

On the other hand, it is good also as a structure which arrange | positions each newly added virtual reality object in a respectively different layer, and is good also as a structure which manages a layer according to the distance from a user. For example, a virtual reality object at a position where the distance L from the user is 0 m <L ≦ 5 m is associated with the layer 01, and a virtual reality object at a position of 5 m <L ≦ 10 m is associated with the layer 02, and 10 m <L ≦ 30 m. The virtual reality object at the position is associated with the layer 03, the virtual reality object at the position of 30m <L ≦ 100m is associated with the layer 04, and the virtual reality object at the position of 100m <L is associated with the layer 05.

Here, when the virtual reality object moves in the virtual space by the user's own movement or behavior control by the virtual reality object behavior control unit 5110, the positional relationship between the user and each virtual reality object changes. Accordingly, the virtual space update management unit 3132 performs a process of changing the association between each layer and each virtual reality object at a predetermined cycle.

FIG. 35 is an example of a management file for managing the association between each layer and the virtual reality object. The virtual space update management unit 3132 updates the virtual reality object associated with the layer by moving the virtual reality object associated with each layer between the layers in the management file. The display video generation processing unit 2140 generates a video for each layer in which only the virtual reality object included in the layer is drawn for each layer.

Although the case where a plurality of layers are set in one virtual space has been described in the above description, the present embodiment is not limited to this. The virtual space generation unit 3131 generates a virtual space for each layer, and the virtual reality object placement unit 3140 places a virtual reality object in the virtual space of a new layer. That is, each layer represents an independent virtual space, and therefore the virtual space generation unit 3131 generates a new virtual space when a virtual reality object is newly read and arranged. Therefore, the virtual space generation unit 3131 generates a plurality of virtual spaces and identifies each virtual space by a layer. The display image generation processing unit 2140 generates an image of the virtual reality object for each layer by independently drawing the virtual space of each layer that is currently generated.

Since virtual reality objects are arranged using a plurality of virtual spaces as layers, and drawing processing is performed for each virtual space, a plurality of layers can be generated simultaneously by preparing a plurality of drawing processing units.

<Embodiment 8>
In the seventh embodiment, the virtual reality service providing server takes a form in which a virtual reality presentation video is generated based on the state of the user, transmitted to the virtual reality presentation device, and displayed. Usually, a user state is generated by a virtual reality presentation device, transmitted to a virtual reality service providing server via a wireless network, and a virtual reality presentation image is generated according to a viewpoint position and a line-of-sight direction controlled based on the user state. Is done. Therefore, a time lag occurs after the virtual reality presentation device detects the user state and displays the virtual reality presentation video generated based on the user state on the display unit.

If there is no significant change in the user status during this time lag, there is no problem, but if the user tilts or moves greatly, there is a difference between the user status used for video generation and the user status at the time of display. Will occur. Therefore, when the generated video for presenting virtual reality is displayed as it is, the fusion with the real world may not function well. The virtual reality presentation system according to the eighth embodiment is characterized in that control is performed to suppress the divergence between the virtual reality video and the real world as the user state changes.

FIG. 35 is a block diagram illustrating a configuration of a virtual reality presentation device 800 according to the eighth embodiment. The virtual reality presentation device 800 includes a user state detection unit 110, a voice input unit 210, a display control unit 130, a display unit 140, a device-side virtual reality presentation control unit 220, a communication unit 230, and an ambient condition detection unit. 310, virtual reality presentation data storage unit 410, audio reproduction control unit 440, audio output unit 450, video display start condition setting unit 710, video display determination processing unit 720, user state storage unit 810, Is provided. The display control unit 130 newly includes a video correction processing unit 135.

The user state detection unit 110 generates user state information including detection timing information indicating the detection timing of the detected user state. The user status storage unit 810 stores and manages the user status detected by the user status detection unit 110 in time series.

The communication unit 230 transmits user state information including the detected time information to the virtual reality service providing server. The communication unit 230 receives video data and audio data generated based on the user state information. The video data and audio data include use detection timing identification information indicating video data or audio data generated based on a user state detected at which detection timing.

The video correction processing unit 135 performs correction processing on the video decoded by the video decoding processing unit 132. Specifically, the user state used for the video generation is specified from the use detection timing identification information, and the difference from the user state at the latest detection timing is calculated. The video correction processing unit 135 performs video correction processing on the decoded video corresponding to the change in the user state used for the video generation and the latest user state.

For example, when the user is moving in the right direction, the entire landscape flows relatively to the left side. Therefore, unless the video displayed on the display unit 140 moves to the left side in accordance with the flow of the real scenery, the real world and the virtual reality presentation video cannot be harmonized. The video correction processing unit 135 performs processing for correcting the display position of the decoded video in accordance with the change in the user state.

The video correction processing unit 135 calculates the change amount of the display area in the display unit 140 based on the difference between the user state used for video generation and the latest user state. Specifically, the video correction processing unit 135 calculates a correction vector indicating the shift of the display area in the display unit 140 based on the difference between the user states. The video correction processing unit 135 generates a corrected video in which the display position is changed by adding the correction vector to each pixel coordinate of the decoded video.

Note that how much the display position of the virtual reality object moves in conjunction with the movement of the user differs depending on the relative positional relationship between the user position and the virtual reality object. For example, a virtual reality object arranged at a position far from the user does not change its display position greatly in response to the movement of the user in order to fuse it with a real scene, whereas a virtual reality object near the user Therefore, the display position is relatively changed. Accordingly, the video correction processing unit 135 inputs virtual reality object distance information that is information indicating the distance to the virtual reality object included in each layer, and calculates a display position change vector based on the virtual reality object distance information. It is preferable to adopt a configuration to do so. The virtual reality object distance information is transmitted from the virtual reality service providing server together with the video data.

Since video correction is performed on video for each layer, layer distance information indicating the distance from the current position of the user to the layer may be input. When multiple virtual reality objects are included in one layer, display position correction cannot be performed for each virtual reality object, and correction is performed in units of layers, so individual distances to virtual reality objects This is because it may be more appropriate to set the distance to the layer. The distance to the layer may be, for example, the average distance of the virtual reality objects included in the layer, or a representative virtual reality object is set from a plurality of virtual reality objects included in the layer, and the representative virtual reality object is set. The distance to the layer may be the distance to the layer.

Note that instead of the distance to the layer, information related to the representative position coordinates of the layer may be input, and the display position may be changed by calculating the correction amount associated with the change in the user state, or the distance to the virtual reality object Alternatively, the display position may be changed by inputting information related to the position coordinates of the virtual reality object and calculating a correction amount associated with a change in the user state.

In addition to generating an image of the range displayed on the display unit 140 by sliding the entire display according to the correction vector for which the display position has been calculated, more advanced correction may be performed. For example, a configuration may be used in which a correction process for changing the aspect ratio of the virtual reality object or enlarging / reducing the size is performed in accordance with a change in the user state.

FIG. 37 is a block diagram illustrating a configuration of a virtual reality presentation system 8000 according to the eighth embodiment. The virtual reality presentation system 8000 includes a virtual reality presentation device 800, a virtual reality service providing server 8100, a voice analysis server 2200, a virtual reality content storage DB 2300, and a contractor information storage DB 3200.

The virtual reality service providing server 8100 includes a communication unit 2110, an audio context acquisition unit 2120, a server-side virtual reality presentation control unit 2130, a display video generation processing unit 2140, an image analysis processing unit 3110, and a virtual space setting unit 3130. A virtual reality object placement unit 3140, a virtual reality content reading unit 4110, a reproduction sound generation processing unit 4120, a virtual reality object behavior control unit 5110, and an accompanying information generation processing unit 8110.

The display video generation processing unit 2140 generates a virtual reality presentation video based on the user state information received by the communication unit 2110. The display video generation processing unit 2140 specifies the viewpoint position and the line-of-sight direction based on the user state indicated by the user state information, and the virtual reality object arranged in the virtual space based on the specified viewpoint position and line-of-sight direction By drawing for each layer, a virtual reality presentation video is generated for each layer.

Here, the display video generation processing unit 2140 generates a video larger than the display area of the display unit 140 included in the virtual reality presentation device 800. For example, when the display unit 140 is a display panel having a display area of 1920 pixels × 1080 pixels, the virtual reality presentation device 800 generates an image of 2110 pixels × 1260 pixels. Specifically, the display video generation processing unit 2140 performs a clipping process of cutting out a video having a predetermined size larger than the display area on the display unit 140 from the video projected on the image plane. By generating a large image in this way, it is possible to deal with a video slide process along a correction vector generated in the video correction process in the virtual reality presentation device 800.

The display video generation processing unit 2140 generates video data by adding detection timing identification information included in the user state information used for video generation. That is, the display video generation processing unit 2140 generates a virtual reality presentation video in a state where the user status information used for video generation can be determined.

The reproduction sound generation processing unit 4120 generates sound for virtual reality presentation based on the user state information received by the communication unit 2110. Here, the reproduction audio generation processing unit 4120 generates the audio data by adding the detection timing identification information included in the user state information used for the audio generation. That is, the reproduction sound generation processing unit 4120 generates sound for virtual reality presentation in a state where the user state information used for sound generation can be determined.

The accompanying information generation processing unit 8110 generates various accompanying information used for video correction processing in the virtual reality presentation device 8100. The accompanying information generation processing unit 8110 includes, for each layer video generated by the display video generation processing unit 2140, layer order information indicating the order of the layers, layer information indicating the distance to each layer and representative position coordinates, and each layer The virtual reality object information indicating the distance and position coordinates to the virtual reality object included in is generated as the accompanying information. The communication unit 2110 transmits the generated accompanying information to the virtual reality presentation device 800.

In the virtual reality presentation device 800, the communication unit 230 receives the accompanying information. When the layer order information is included in the accompanying information, the communication unit 230 outputs the layer order information to the layer order management unit 134. If the accompanying information includes layer information or virtual reality object information, the communication unit 230 outputs these information to the video correction processing unit 135.

As described above, in the virtual reality presentation system according to the eighth embodiment, the virtual reality presentation video is generated on the virtual reality service providing server side, and predetermined correction processing is performed on the virtual reality presentation apparatus side. Presented to the user.

In general, the user state detection rate is different from the video generation rate and the video display rate. In addition, a time lag associated with video generation processing and communication processing occurs from when the user state is detected to when the video based on the user state is displayed. If the state of the user changes during the time lag, there is a possibility that the video generated based on the past user state cannot be merged with the actual scenery. For this reason, after the generated video is received, correction processing corresponding to the change in the user state is performed, and display control is performed to maintain the fusion with the actual scenery, and then the display is displayed on the display unit. With this configuration, for example, even if the user's head moves, the virtual reality object displayed on the display unit is displayed so as to maintain the same position in the virtual reality world. You can continue to sense the real world without doubt.

In the above description, the case where the correction process is performed on the video has been described, but the same process may be performed on the sound. That is, an audio correction processing unit is separately provided in the audio reproduction control unit 440, and a process of correcting audio distributed to each channel is performed in accordance with the user state change. However, the human position resolution with respect to hearing is not so high, and there is little problem with the change in the user state from when the user state is detected until the sound is played back. It is also possible to adopt a configuration without this. When the user status is updated and transmitted from the communication unit 230 to the virtual reality service providing server 8100, the reproduction audio generation processing unit 4120 of the virtual reality service providing server 8100 converts the audio data of each channel based on the latest user status. Generated. Therefore, it is possible to sufficiently follow the positional resolution in hearing. On the other hand, since the visual position resolution is much finer, even if there is a slight change in the user state, the vibration of the virtual reality object is perceived by the user, so that correction processing is performed at a higher rate.

<Ninth Embodiment>
While we act based on objective information in the real world perceived by the five senses, we sometimes distort and recognize the real world by feelings such as hope and desire. For example, even if it is impossible in reality, I hope in my heart that it will be realized by wishing.

The virtual reality world is characterized by reproducing events that cannot occur in the real world as if they had occurred in the real world. Therefore, by presenting a virtual reality world that corresponds to human behavior such as prayers and wishes, users will feel that their prayers and wishes have been communicated, and naturally enter the virtual reality world. be able to.

For example, suppose a user has a desire to meet a grandfather who has already died. While all of us humanly recognize the cruel fact that the dead will not come back to life, we expect them to be revived somewhere in our hearts. In such a case, the user closes his eyes and asks, wishes and prays in his heart, "I want to meet my grandfather again." And when the grandfather appears in front of the user, the user feels that the prayer has passed, so even if the grandfather's figure is created virtually, it is the real thing that appeared in the real world It becomes easy to make an illusion. The virtual reality presentation system according to the ninth embodiment is characterized in that the virtual reality world is presented to the user by using such a gap between human hearts.

The virtual reality presentation device according to the ninth embodiment will be described with reference to FIG. The user state detection unit 110 includes an internal camera 111 that captures at least a part or all of the user's face, and based on the video captured by the internal camera 111, the video display determination processing unit 120 is for virtual reality presentation. Judgment processing of whether or not to start the display of the video.

Here, the video display determination processing unit 120 performs the determination process based on whether or not the user has closed eyes deeply. That is, the video display start condition setting unit 420 sets, as the video display start condition, that the user closes his eyes deeper than a predetermined reference for a predetermined time.

Normally when people pray or pray, they meditate deeply and wrinkle between the eyebrows to focus their consciousness in front of the head. Therefore, the video display determination processing unit 120 is based not only on whether the eyes are open or closed based on the video of the user's face taken by the internal camera 111, but the user meditates deeply beyond a predetermined reference. It is determined whether or not. For example, the video display determination processing unit 120 performs a matching process between the image of the user's eye image captured by the internal camera 111 and the image of the eye in a state where the eyes are deeply meditated, and a correlation greater than or equal to a predetermined correlation value is obtained It is determined whether it is obtained. As a result of the determination, when a correlation equal to or greater than a predetermined reference value is obtained, it is determined that the user is deeply meditating.

As a result of the determination, when it is determined that the user has deeply closed the eyes for a predetermined period or longer, the display control unit 130 performs control to start displaying video for presenting virtual reality, and the display unit 140 Displays a video for presenting reality.

Similarly, the audio reproduction determination processing unit 430 performs similar determination processing, and the audio reproduction control unit 440 performs control for starting reproduction of audio for virtual reality presentation based on the determination result. Audio for virtual reality presentation is output.

As described above, in the virtual reality presentation device according to the ninth embodiment, the video display start determination processing unit 120 determines whether the user has closed eyes deeply based on the video captured by the user state detection unit 110. Then, the display control unit 130 starts displaying the virtual reality presentation video on the display unit 140 when it is determined that the user has closed eyes deeply.

With this configuration, it is possible to present the virtual reality world in the form that the user's prayers and wishes have been communicated. It is preferable that the virtual reality service providing server selects an appropriate virtual reality content based on the voice context input by the voice input unit 210 and presents the virtual reality world to the user.

For example, it is assumed that a voice saying “I want to meet my grandfather” is input from the voice input unit 210. In this case, based on the contractor information of the user stored in the contractor information storage DB, a virtual reality representing the appearance of the grandfather of the user stored in the virtual reality content storage DB in advance as a virtual reality object. Read the object and place it in the virtual space. The virtual reality content storage DB may store the grandfather's real voice data in advance. The virtual reality presentation video data representing the appearance of the grandfather is drawn to generate virtual reality presentation video data, and the audio data of the virtual reality presentation is obtained by reading the grandfather audio data and distributing it to each channel. Generated. These data are transmitted to the virtual reality presentation device, and the presentation to the user is started when the determination criterion is cleared according to the determination processing in the video display start determination processing unit 120 and the audio reproduction start determination processing unit 430.

In the above configuration, the case where the internal camera 111 serving as the user state detection unit 110 detects the user state by capturing an area including at least the eyes of the user has been described, but the present invention is not limited to this. For example, as in a virtual reality presentation device 900 shown in FIG. 38, a configuration including an electroencephalogram sensor 114 that detects a user's brain wave and a biological sensor 115 that detects a user's pulse rate, blood flow, blood pressure, sweating, and the like. good.

Usually, when a user prays, he closes his eyes and concentrates his consciousness on the front of the head. This action increases the blood flow in the frontal lobe and emits alpha waves. The electroencephalogram sensor-114 determines that the user is praying or wishing by detecting the alpha wave. Further, the video display start determination may be performed by combining the detection results from these sensors and the closing state of the eyes by the internal camera 111.

Note that a laser interferometer that measures the height of the peripheral portion of the user's eyes can be used as detection means for detecting whether the user has closed his eyes deeply. When the eyes are closed deeply, a deep eyelid is formed around the eyes. Therefore, it is possible to detect whether the user has closed the eyes deeply by detecting the eyelid. Moreover, it is good also as a structure which detects whether the user has closed eyes deeply by detecting the number and depth of the eyelid around eyes. In the image of the area including the user's eyes photographed by the internal camera 111, the wrinkle portion is detected by matching processing, and the number of the wrinkles, the length of the wrinkles, and the portion between the wrinkle portion and the adjacent portion The depth of the ridge is detected from the difference in brightness. The video display start determination processing unit 120 compares the number of wrinkles, the length of the wrinkles, the depth of the wrinkles, and the like with the number of wrinkles, the length of the wrinkles, and the depth of the wrinkles, so that the user state Determines whether the conditions for starting the presentation of virtual reality are satisfied. It is good also as the said structure.

<Embodiment 10>
We humans live in this real world by sensing various events that occur in the real world using the sensory organs we have born and acting in the real world. That is, humans can be said to be organisms that input information from outside using the five senses, process the input information in the brain, and output the processing results in the form of actions.

This means that by deceiving human vision and hearing, the virtual reality world created artificially can be illusioned as if it were the real world. The virtual reality presentation device is such a user interface device.

  The virtual reality world, which is an artificial world presented to the user by the virtual reality presentation device, is partially in common with the real world. Accordingly, it is possible to newly introduce a parameter representing the degree of deviation from the real world with respect to the virtual reality world. In this specification, the parameter is referred to as virtual reality. The virtual reality is represented by a numerical value of 0 to 100. The virtual reality 0 indicates that nothing is presented to the user and the world sensed by the user is the real world itself. Reference numeral 100 denotes an ideal state where the world perceived by the user is completely separated from the real world. That is, the virtual reality degree 100 indicates that the virtual reality presentation device completely blocks information from the real world, and all the information sensed by the user is based on artificially created information.

FIG. 68 is a diagram for explaining the virtual reality degree. Assume that the user is in a room with two sofas. When the virtual reality presentation device displays nothing, the scenery sensed by the user is the scenery of the real world A (a). Here, when the virtual reality presentation device displays the image of the dog (b), the scenery sensed by the user is the scenery where the dog is sleeping on the sofa, and the world B of the virtual reality world B different from the real world A Will be watching. When the virtual reality presentation device displays an image of a lion (c), the view sensed by the user is a view of the lion sleeping on the sofa, and sees a world of a virtual reality world C different from the real world A. Will be.

The virtual reality world B is an artificial world different from the real world, but in the real world, an event that a dog is sleeping on a sofa is relatively likely to occur. On the other hand, the phenomenon that the lion is sleeping on the sofa at home is relatively unlikely, and the virtual reality world C is higher in virtual reality than the virtual reality world B.

The real world where we live is subject to physical and social constraints. For example, it is difficult to realize interplanetary travel due to the physical restriction that a substance with mass cannot exceed the speed of light, and the physical body said that the law of conservation of energy governs the sky in the living body freely. I can't fly.

  Even if it is a physically possible event, human behavior is socially restricted by ethical norms and laws set to give peace and order to the real human society. . Thus, while human beings are in a position to enjoy freedom, in the real world they only enjoy conditional freedom subject to various restrictions.

The virtual reality presenting apparatus can find its existence value by removing the restriction that we regard as common sense in this way and letting the user present a world that cannot be experienced in the real world. Therefore, an indicator that the existence value is so high that the user can be presented with a virtual reality world with a high degree of virtual reality greatly deviating from the real world.

On the other hand, if the user cannot imagine that the virtual reality world presented by the virtual reality presentation device is the real world, the virtual reality world with a high degree of virtual reality is presented to the user. However, it does not contribute to the construction of a virtual reality world. Therefore, when the virtual reality presentation device presents the virtual reality world to the user, a new parameter indicating the degree of illusion that the virtual reality world sensed by itself is the real world has been introduced. It becomes possible to do. In the present specification, the degree of the illusion is referred to as the illusion degree.

FIG. 69 is a diagram illustrating a temporal change in the illusion degree of the user when the virtual reality presentation device presents the virtual reality world to the user. A threshold T0 can be set for the illusion degree. If the illusion degree exceeds the threshold Th0, the user indicates that the world that he / she currently senses is an illusion. If the threshold value is less than the threshold Th0, the user recognizes that the world that the user is currently sensing is an artificial world that is not a real world.

In FIG. 69, the virtual reality presentation device displays a dog image until the timing of T1, and the user senses the virtual reality world where the dog is sitting on the sofa. Since the illusion degree of the user exceeds the threshold Th0, the user has an illusion that the scenery he / she sees is the real world. Here, it is assumed that the virtual reality presentation device switches to an elephant image at the timing of T1. The user perceives the virtual reality world where the elephant is standing on the sofa, but since the illusion is below the threshold Th0, the user is not an elephant in the real world but the video It is recognized as an elephant.

As described above, generally, the higher the virtual reality degree, the lower the illusion degree of the user. However, even in a virtual reality world with a high virtual reality, it is possible to present the virtual illusion in stages without reducing the illusion of the user.

For example, when a virtual reality service called “world tour with dinosaurs” is provided to a user, if a video of a large dinosaur is suddenly displayed in front of the user, the user immediately recognizes that it is a fake video. On the other hand, a dinosaur image is displayed so that it can be seen in a distant place, and it is displayed to gradually approach the user, or a lizard image is displayed at first, and slowly over time. It is possible to maintain a high illusion degree by continuously changing the illusion.

As described above, in order for a user to enter the virtual reality world, it is necessary to present the virtual reality world to the user while maintaining a high illusion degree. It wasn't done, and it was only unilaterally displaying artificial images. Therefore, while the user can enjoy the image represented in the real world, he / she only enjoys it after recognizing it as a product, and the presented virtual reality world is realized in the real world. I had an illusion that I couldn't have the experience of entering the virtual reality world.

In the present embodiment, in view of the above-described problems, the presentation of the virtual reality world is controlled so as to make an illusion that the virtual reality world is a real world. Hereinafter, it will be described in detail with reference to the drawings. In the following description, parts denoted by the same reference numerals have substantially the same functions. For the sake of clarification of the invention, explanations of overlapping parts are omitted as appropriate.

FIG. 40 is a block diagram showing the configuration of the virtual reality presentation system 1000 according to the tenth embodiment of the present invention. The virtual reality presentation system 1000 includes a user state detection unit 1010, a presentable virtual reality calculation unit 1020, a video generation processing unit 1030, and a video display unit 1040.

The user state detection unit 1010 detects the state of the user who is receiving the virtual reality presentation from the virtual reality presentation system 1000. The user state detection unit 1010 includes, for example, an acceleration sensor that detects the user's acceleration, an inclination sensor that detects the user's inclination, and a GPS receiver that detects the user's position.

The presentable virtual reality calculation unit 1020 calculates the virtual reality of virtual reality that can be presented to the user. Here, virtual reality (VR) refers to a world with reality created artificially by combining computer graphics (CG) and sound effects. The virtual reality degree indicates the degree of divergence between the real world where the user exists and the virtual reality world presented by the virtual reality presentation system 1000. In the tenth embodiment, the virtual reality degree is indicated by any value from 0 to 100. .

The video generation processing unit 1030 generates a video for virtual reality presentation so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020 based on the user status detected by the user status detection unit 1010. To do.

The virtual reality presentation video is, for example, a video including a virtual reality object. The virtual reality object refers to a virtual person or a building represented in the virtual reality world by being arranged in a virtual space that is a virtual space artificially set. By setting the virtual space so as to correspond to the real space around the user, the virtual reality world is presented to the user.

Each type of virtual reality object is composed of model data including 3D polygon data defining the appearance shape, texture data pasted on the surface of the shape formed by the polygon data, and the like. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint point data and the vertex of the polygon data are associated, and changing the position and angle of the joint point changes the position of the vertex of the associated polygon data, and the appearance of the virtual reality object The shape changes. A plurality of types of texture data are also prepared, and each texture data is identified by an identification number. An image of the virtual reality object is generated by pasting the selected texture onto the polygon surface.

The video generation processing unit 1030 generates a video for virtual reality presentation based on the user state detected by the user state detection unit 1010 so as to match the real world scene.

The video display unit 1040 displays the virtual reality presentation video generated by the video generation processing unit 1030.

As described above, according to the virtual reality presentation system according to the tenth embodiment, by presenting the virtual reality world while controlling the degree of deviation from the real world in the virtual reality world presented to the user, It is characterized by maintaining the illusion of the user that the scenery he / she sees is a real world scenery. Accordingly, the user can enter the virtual reality world and experience a world physically and socially impossible in the real world.

It should be noted that the virtual reality presentation system 1000 according to the tenth embodiment is more favorable when configured as shown in FIG. The virtual reality presentation system 1000b illustrated in FIG. 41 further includes a virtual reality content storage unit 1050 and a presenting virtual reality calculation unit 1060.

The virtual reality content storage unit 1050 stores virtual reality content. The virtual reality content is a generic term for data used to create a virtual reality world, and includes at least model data that defines the appearance of a virtual reality object. The model data includes polygon data that defines the appearance shape, texture data that is pasted on the surface of the appearance shape, and the like. Virtual reality content includes background music data used to create a virtual reality world, audio data generated by a virtual reality object, a virtual reality object behavior control program that controls the behavior of a virtual reality object, Virtual reality control programs that control temporal changes (story characteristics) are included.

The presenting virtual reality calculation unit 1060 calculates the virtual reality related to the virtual reality world currently being presented. For example, the presenting virtual reality calculation unit 1060 displays the types of virtual reality objects displayed as the virtual reality presentation video generated by the video generation processing unit 1030, the number of virtual reality objects included in the video, the virtual reality Based on the information amount of the video for presentation, the virtual reality degree of the virtual reality world currently being presented is calculated.

For example, the presenting virtual reality calculation unit 1060 obtains the sum of the virtual reality assigned to each virtual reality object displayed as the virtual reality presentation video generated by the video generation processing unit 1030. Calculate the virtual reality of the virtual reality world being presented.

Further, for example, the presenting virtual reality calculation unit 1060 indicates the number of pixels of the virtual reality presentation video generated by the video generation processing unit 1030 with respect to the display area (total number of display pixels) included in the video display unit 140. The virtual reality of the virtual reality world currently being presented is calculated according to the ratio.

The presentable virtual reality calculation unit 1020 adds, subtracts, multiplies, and divides a predetermined value on the basis of the virtual reality related to the world of the virtual reality being presented calculated by the presenting virtual reality calculation unit 1060 The virtual reality degree related to the virtual reality world that can be presented is calculated based on a predetermined calculation formula. The video generation processing unit 1030 displays the virtual reality presentation video so that the virtual reality degree of the virtual reality presentation video to be generated is equal to or less than the virtual reality degree of the calculated virtual reality world that can be presented. Generate.

The human brain is insensitive to gradual changes. When two images with different parts are switched, humans can recognize the difference appropriately. However, when two images are gradually changed over a period of several tens of seconds, There is a feature that the difference cannot be recognized.

This is because the brains of living organisms have evolved with the characteristic of making situation judgments in order to survive the survival competition. A human being is a living organism that detects events that occur in the outside world using the five senses, that is, taste, touch, vision, hearing, and smell, and processes the detected information in the brain to perform the next action. The human brain is configured to judge the situation from the difference between the situation that is currently observed and sensed and the situation that was sensed in the past. Therefore, as the time passes, the details of the surrounding situation that have been sensed become out of memory, so even if there is a big difference from the initial situation by slowly changing it, the human brain will appropriately It has the feature that it cannot be captured.

Needless to say, human beings are the final winners of the 3.8 billion year long survival competition since the earth was born. The most important thing in the survival competition is to appropriately identify how to act next, and we human beings judge this situation far more than we think in life on earth. A creature that has specialized in ability.

In the situation judgment, the latter has overwhelmingly important information in the situation 10 seconds ago and the situation 1 second ago. On the other hand, it is also important to properly grasp the changes in the situation over time. Therefore, the human brain forgets the details of the sensed information with time, and compresses the information into a form called a concept to cope with time-series changes. Accordingly, by slowing down the time series change in the virtual reality world, it is possible to enter the virtual reality world without causing a psychological obstacle to the currently sensed virtual reality world. This is in common with saying that a person who is dreaming about an unrealistic dream cannot recognize that the dream world is not a real world but a dream world. is there. Therefore, the present invention can be realized as a virtual reality world in which a wonderful dream world experience is superimposed on a more realistic real world.

The virtual reality presentation system 1000b uses such characteristics inherent in the human brain. In other words, even when presenting users with a virtual reality world that has a high degree of virtual reality and is far from the real world, the virtual reality world should be shaped by gradually changing from the real world. The user is characterized by preventing the user from recognizing that the world that the user senses is moving from the real world to the virtual reality world.

In other words, a user's ability to make judgments by taking the opposite of the principle that creates a high judgment ability of the human brain that has evolved so that it can respond appropriately to the situation changes that can occur in the real world through a long process of evolution. Present the virtual reality world so that does not function properly.

The virtual reality is obtained as a virtual reality that can be presented at present by giving a predetermined difference from the virtual reality that is being presented with reference to the virtual reality that is related to the virtual reality that is currently being presented. Then, by generating the virtual reality presentation video so as to be within the obtained presentable virtual reality level, the world sensed by the user gradually shifts from the real world to the virtual reality world.

  By performing control based on the virtual reality level, discontinuity in the transition from the real world to the virtual reality world, even in a virtual reality world that seems impossible at first glance Can be suppressed. Therefore, by entering the blind spot of the human brain processing mechanism specializing in situation determination in the real world, there is an effect that prevents the user from recognizing that the virtual reality world is virtual under the obvious consciousness. be able to.

Further, the virtual reality presentation system 1000 according to the tenth embodiment is more favorable when configured as shown in FIG. The virtual reality presentation system 1000c illustrated in FIG. 42 further includes an audio generation processing unit 1070 and an audio output unit 1080.

The sound generation processing unit 1070 reads the virtual reality presentation sound data from the virtual reality content storage unit 1050 and generates the virtual reality presentation sound output from the sound output unit 1080. More preferably, the voice generation processing unit 1070 is for virtual reality presentation so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020 based on the user status detected by the user status detection unit 1010. Generate the voice of

For example, the audio generation processing unit 1070 distributes the audio data for virtual reality presentation read from the virtual reality content storage unit 1050 to the audio data of a plurality of channels based on the user status detected by the user status detection unit 1010. To do.

The sound output unit 1080 outputs the sound generated by the sound generation processing unit 1070. The audio output unit 1080 is specifically a speaker having a plurality of channels. The digital audio signal of each channel generated by the audio generation processing unit 1070 is converted to an analog audio signal by performing DA conversion processing, and is subjected to air vibration. Convert and emit to the outside.

  The virtual reality presentation sound is output in a form linked to the virtual reality presentation video, so that the user has an illusion of both visual and auditory perception of the outside world, making it easy for the user It is possible to enter the virtual reality world.

  The presentable virtual reality calculation unit 1020 and the presenting virtual reality calculation unit 1060 include a virtual reality presentation video generated by the video generation processing unit 1030 and a virtual reality presentation audio generated by the audio generation processing unit 1070. Virtual reality may be assigned to each of these, or virtual reality may be assigned together.

For example, the presentable virtual reality calculation unit 1020 assigns the virtual reality 70 to the virtual reality related to the presentable virtual reality presentation video, and the virtual reality related to the presentable virtual reality presentation sound. Is assigned a virtual reality degree of 20. The video generation processing unit 1030 generates a virtual reality presentation video so that the virtual reality related to the generated video is 70 or less, and the audio generation processing unit 1070 has a virtual reality related to the generated audio of 20 or less. The voice for virtual reality presentation is generated so that

Further, for example, the presentable virtual reality calculation unit 1020 assigns the virtual reality 80 collectively as the virtual reality related to the presentable virtual reality world. The video generation processing unit 1030 and the audio generation processing unit 1070 are configured so that the virtual reality in the virtual reality world formed by the virtual reality presentation video and audio generated by these two generation units is 80 or less. Realistic video and audio are generated respectively.

FIG. 43 is a block diagram showing a configuration of a virtual reality presentation system in a case where the virtual reality degrees are assigned collectively. The virtual reality presentation system 1000d includes a user state detection unit 1010, a presentable virtual reality calculation unit 1020, a presenting virtual reality calculation unit 1060, a virtual reality presentation data generation processing unit 1090, and a virtual reality presentation unit 1100. With.

The virtual reality presentation data generation processing unit 1090 includes a video generation processing unit 1030 and an audio generation processing unit 1070. The virtual reality presentation data generation processing unit 1090 includes the presentable virtual reality world in which the virtual reality related to the virtual reality world presented by the virtual reality presentation unit 1100 is calculated by the presentable virtual reality calculation unit 1020. The virtual reality content stored in the virtual reality content storage unit 1050 is read so as to be within the virtual reality level according to the above, and video and audio for virtual reality presentation are generated.

In this way, the presentable virtual reality calculation unit 1020 can present the virtual reality that can be currently assigned to the virtual reality presentation data generation processing unit 1090 that generates video data and audio data for virtual reality presentation. Calculated as a virtual reality degree related to the virtual reality world.

Based on the virtual reality presentation data generated by the virtual reality presentation data generation processing unit 1090, the presenting virtual reality calculation unit 1060 calculates the virtual reality related to the virtual reality currently being presented.

The virtual reality presentation unit 1100 presents the virtual reality world to the user by outputting various data generated by the virtual reality presentation data generation processing unit 1090 in a predetermined output format. The virtual reality presentation unit 1100 includes a video display unit 1040 and an audio output unit 1080. However, when configured to make the user's haptic illusion, a separate mechanism such as a vibration output unit may be provided.

  In addition, since video and audio are often linked in the virtual reality world to be presented, the audio output unit 1080 is generated in consideration of virtual reality without considering virtual reality. A configuration may be adopted in which audio associated with the virtual reality presentation video is output.

Note that all the functions constituting the virtual reality presentation system 1000 described above may be provided in a virtual reality presentation device worn by the user, or some functions may be provided by an information processing server provided outside. May be. In that case, the virtual reality presentation device and the information processing server each include a communication unit, and transmit and receive video data, user state information, and the like.
User status detection means for detecting the user status;

<Embodiment 11>
The present invention introduces a first parameter called virtual reality for the virtual reality world presented to the user, and presents the virtual reality world so that the temporal change of the virtual reality falls within a predetermined value. By controlling this, the user is prevented from recognizing that the world he / she senses is not real but virtual.

Here, the issue of how rapidly the virtual reality changes with time is a new problem. When considering the evaluation index for the virtual reality presentation system, it is better to be able to present the virtual reality world that is more distant from the reality such as future city and fantasy city without realizing that it is virtual It becomes a reality presentation system.

Therefore, if possible, it is preferable that a virtual reality world with a high virtual reality degree can be presented to the user as soon as possible, starting from a real world having a virtual reality degree of zero. On the other hand, if the increase ratio of the virtual reality level is too large, the user is more likely to recognize that the perceived world is not the real world but the virtual reality world. That is, the virtual reality world in which the user has an illusion is broken.

The virtual reality presentation system according to the eleventh embodiment is characterized in that an optimal virtual reality world can be presented to the user by appropriately managing the temporal change of the virtual reality.

FIG. 44 is a block diagram showing a configuration of the virtual reality presentation system 2000 according to the eleventh embodiment. The virtual reality presentation system 2000 includes a user state detection unit 1010, a video generation processing unit 1020, a presentable virtual reality calculation unit 1030, a video display unit 1040, a virtual reality content storage unit 1050, and a virtual reality being presented. A calculation unit 1060 and an illusion degree calculation unit 2010 are provided.

The illusion degree calculation unit 2010 calculates the current illusion degree of the user based on the user state detected by the user state detection unit 1010. Here, the illusion degree is a second parameter indicating the degree of illusion that the virtual reality world sensed by the user is the real world.

The illusion degree is indicated by any value from 0 to 100. For example, when the user who sees the view of FIG. 40B believes that the dog displayed as sitting on the sofa is actually a dog sitting on the sofa, the illusion degree is 100.

On the other hand, the user who has seen the scenery in FIG. 40 (c) has a suspicion that the scenery he / she sees is virtual, from a scene far from the daily life that a lion is sitting on the sofa. Therefore, the user's sense of illusion is not entirely illusion that the world he / she perceives is a real world, but a virtual reality world somewhere in his / her heart. Is less than 100, for example, about 40.

Here, the illusion degree of the user does not become zero. This is the key to realizing the virtual reality world. This point will be described in detail.

  We human adults have somehow incredible experiences that do not fit in the past rule of thumb in life. "That's a lie?", "That stupid!", "I can't!" "Seriously?", "I can't believe ...", "Please give me ...", "Worst ..." It ’s no exaggeration to say that there is no adult who has never spoken a word in life. What does this mean?

We humans have built a database of past experience knowledge cultivated in their own lives, and perform matching processing to compare the external situation detected through vision and hearing with previous experience knowledge. I know what I am currently in.

In this world where the social order is established, the situation perceived by humans is a repetition of similar situations, and such situations are imprinted in the brain as empirical knowledge. However, you may be in a situation where you have never experienced it before. In such a case, the human will utter the above words when the situation that he / she currently senses cannot be compared with the experience knowledge he / she has built in the brain.

In this way, human adults know, as a rule of thumb, that in life, events that are instinctively inconsistent with their common sense and experience knowledge may occur in themselves. In such a case, human beings say that if they stop thinking because they do not agree with their own knowledge and experience, they will lose their survival competition. I know instinctively.

For example, when walking on a mountain and encountering a huge bear, a human asks, "Is it a lie? This is because he has no such experience and is outside the scope of his assumptions. However, in order to survive in such a critical situation, the human brain recalls various workarounds and attempts to take the action that seems to be the best in order to escape from the irregular situation.

Thus, human adults know empirically that extraordinary events can occur. Saying that you are in a situation that does not fit your own knowledge means that you do not know the proper way to deal with the current situation, and therefore the level of survival is rising. The human beings who are the best in survival competition essentially have thought circuits that act in the worst case for this critical situation.

This is the reason why the illusion degree of the user experiencing the virtual reality world does not fall to zero. In other words, the virtual reality world is a world that deviates from the real world, that is, an extraordinary world. When we sense such a world, even if we think that our current situation is “impossible,” the human instinct that tries to increase the survival probability in the worst case is “unbelievable. But it may be reality. " Therefore, the virtual reality world is established.

In the above description, the reason why the illusion degree of a human adult does not fall to 0 has been described, but there is also a reason why the illusion degree does not fall to 0 for a human child. Human children instinctively understand that they are weak in the survival competition. In addition, since there is no accumulation of experience, there is a tendency to capture what is perceived through vision and hearing. This is why children are obedient to novelty.

Since we have not built an empirical knowledge database, we instinctively understand that it is not possible to increase the survival probability only by matching processing, and the next action can be taken as it is without processing / distorting the information obtained through the five senses. Judge the situation to take. Therefore, when presenting the virtual reality world by the virtual reality presentation system, the child accepts the world that the child sees and hears as it is, trusting his senses. Therefore, the illusion degree is higher than that of an adult. This is why we human beings are more interested in games, theme parks, and animations in childhood than in adulthood. This is to accept the exciting world presented by games and theme parks.

Therefore, an adult who has seen the scenery in FIG. 40 (c) has some doubts about the extraordinary event that a lion is in the room, but he / she is unaware of it. The psychology of risk hedging works against the worst case of being killed, and this lion cannot be completely ignored. In other words, the virtual reality world is partly illusioned to be the real world, and the illusion degree is not zero. The child understands the current scenery that he sees as it is and has the illusion that there is a lion in front of him.

In the virtual reality presentation system 2000 according to the eleventh embodiment, the illusion degree calculation unit 2010 calculates the illusion degree based on the user state. Since there is a difference in the user's state between when the user has an illusion of virtual reality and when it is not, the illusion degree of the user is estimated based on the state of the user, and the value of the parameter called illusion degree Can be calculated.

In the virtual reality presentation system 2000 according to the eleventh embodiment, the user state detection unit 1010 includes a tilt sensor 1011, an acceleration sensor 1012, an orientation sensor 1013, a position sensor 1014, an internal camera 1015, an electroencephalogram sensor 1016, A blood pressure sensor 1017, a pulse sensor 1018, and a sweat sensor 1019 are provided.

The tilt sensor 1011 detects the tilt of the head of the user wearing the virtual reality presentation device constituting the virtual reality presentation system 2000 and acquires tilt information. The acceleration sensor 1012 detects acceleration related to the user's head and acquires acceleration information. The direction sensor 1013 detects the direction in which the user is facing and acquires direction information. The direction sensor 1013 can use an electronic compass, for example.

The position sensor 1014 detects the current position of the user. As the position sensor 1014, for example, a GPS receiver can be used.

  The internal camera 1015 captures part or all of the user's face to acquire a user facial expression image.

The electroencephalogram sensor 1016 detects the user's electroencephalogram. The electroencephalogram sensor 1016 detects the type of electroencephalogram emitted from the user's brain, the intensity of the electroencephalogram, the electroencephalogram emission position, and the like. The blood pressure sensor 1017 measures the blood pressure of the user. The pulse sensor 1018 measures the user's heart rate. The sweat sensor 1019 detects the amount of sweat of the user.

Since the electroencephalogram sensor 1016, blood pressure sensor 1017, pulse sensor 1018, and sweat sensor 1019 are a group of sensors that detect a user's biological reaction, they may be referred to as a biological reaction sensor in the following description. The biological reaction sensor may include other sensors such as a body temperature sensor that measures a user's body temperature and a respiration sensor that detects respiration frequency.

The image generation processing unit 1030 combines information such as inclination information acquired by the inclination sensor 1011, acceleration information acquired by the acceleration sensor 1012, direction information acquired by the direction sensor 1013, and the like, A change in the line-of-sight direction is specified, and a virtual reality presentation video corresponding to the specified eye position and line-of-sight direction is generated. The video generation processing unit 1030 generates a virtual reality presentation video so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020. The video display unit 1040 displays the generated video for presenting virtual reality.

The illusion degree calculation unit 2010 calculates the current illusion degree of the user based on the facial expression of the user photographed by the internal camera 1015 and the biological reaction of the user detected by the biological reaction sensor.

For example, the illusion degree calculation unit 2010 specifies whether the user's line-of-sight direction specified based on the image captured by the internal camera 1015 is toward the virtual reality presentation video displayed on the video display unit 1040. To do. For example, when a virtual reality object is displayed on the video display unit 1040 as a virtual reality presentation video, the user's line-of-sight direction specified based on the video shot by the internal camera 1015 captures the virtual reality object. Calculate the percentage that you are doing. The illusion degree is calculated assuming that the higher the ratio of capturing the virtual reality object, the higher the illusion degree of the user.

Further, for example, after the virtual reality presentation video is displayed, when the α wave detected by the electroencephalogram sensor 1016 tends to increase, the illusion degree is calculated to be high, and the β wave tends to increase. The illusion degree is calculated assuming that the illusion degree is low.

Further, for example, after a virtual reality presentation image is displayed, an increase in blood pressure detected by the blood pressure sensor 1017, an increase in heart rate detected by the pulse sensor 1018, and an increase in the amount of sweat detected by the sweat sensor 1019. Are continuously observed, the user is excited about the virtual reality world and the illusion degree is high, and the illusion degree is calculated.

Note that the above-described calculation method is an example, and it is only necessary to appropriately set a calculation formula that leads to the relationship between the user state and the illusion degree. For example, the illusion degree calculation unit 2010 may be configured to calculate the illusion degree by substituting detection results of some of the sensors included in the user state detection unit 1010 into a predetermined calculation formula.

In addition, the illusion degree may be calculated so that the illusion degree decreases as the user's movement or the change in acceleration or inclination related to the user increases. The video generation processing unit 1030 controls the display position so that the position of the virtual reality object does not move in the virtual reality world in conjunction with the movement of the user. However, if the user's movement is large, the control may be shifted, and the virtual reality object may appear to slide to the user's eyes. Even if such an object that is originally stationary is a little, it slides out of an event that can occur in the real world, and therefore it raises the suspicion that it is a created image. Therefore, when the user is moving or when rapid acceleration is applied, the illusion degree calculation unit 2010 may be configured to calculate the illusion degree low.

In addition, it is not necessary to provide all the above-described sensors as the user state detection unit 1010, and some sensors may be provided, and other sensors such as a body temperature sensor and a vibration sensor may be used as the user state detection unit 1010. It may be provided.

Each sensor described above may be provided in a virtual reality presentation device worn by the user, or may be provided on a wall surface around the user who is receiving the virtual reality presentation.

The presentable virtual reality calculation unit 1020 has the illusion calculated by the illusion degree calculation unit 2010 with reference to the virtual reality related to the present virtual reality world calculated by the presenting virtual reality calculation unit 1060. The virtual reality degree related to the virtual reality world that can be presented at present can be calculated by adding, subtracting, multiplying, or dividing the value determined based on the degree.

For example, X represents the virtual reality related to the currently presented virtual reality calculated by the presenting virtual reality calculation unit 1060, Y represents the illusion degree of the current user calculated by the illusion degree calculation unit 2010, When the virtual reality degree related to the currently presentable virtual reality world calculated by the possible virtual reality degree calculation unit 1020 is Z, the presentable virtual reality degree calculation unit 1020 calculates Z using the following Equation 1. To do.
(Formula 1) Z = X + (Y-50) / 10

The video generation processing unit 1030 generates a virtual reality presentation video so as to be within the virtual reality Z related to the presentable virtual reality world calculated by the presentable virtual reality calculation unit 1020. The video display unit 1040 displays the generated video.

As described above, according to the virtual reality presentation system according to the eleventh embodiment, the virtual reality presented to the user according to the state of the user so that the user's consciousness does not fall off from the virtual reality world. The world is properly controlled. Therefore, the user can enjoy the virtual reality world for a long time, and can present the user with an attractive virtual reality world that is more distant from the real world.

Note that the calculation of the virtual reality related to the virtual reality world being presented and the virtual reality related to the available virtual reality world may be calculated at a predetermined periodic interval instead of being configured in real time. . When the virtual reality that can be presented is calculated, the video generation processing unit 1030 calculates the latest virtual reality that has been calculated until the virtual reality related to the virtual reality that can be newly presented at the next calculation timing. It is good also as a structure which produces | generates an image | video using the virtual reality degree which concerns on the virtual reality world which can be shown.

<Twelfth embodiment>
We humans capture the scenes of the outside world with one eye to acquire a 2D image, and process the 2D images acquired with both parallax eyes in the brain to create a 3D space. To grasp. The virtual reality presentation device needs to display an image as if a virtual object or person exists in the three-dimensional space that is the real world.

FIG. 45 is a block diagram illustrating a configuration of a virtual reality presentation system 3000 according to the twelfth embodiment. The virtual reality presentation system 3000 includes a user state detection unit 1010, a presentable virtual reality calculation unit 1020, a video generation processing unit 1030, a video display unit 1040, a virtual reality content storage unit 1050, and a virtual reality being presented. A calculation unit 1060, a virtual reality presentation control unit 3010, a virtual space generation processing unit 3020, a virtual reality object arrangement processing unit 3030, and a viewpoint control unit 3040 are provided. The video generation processing unit 1030 includes a drawing processing unit 1031.

The virtual reality presentation control unit 3010 performs processing for starting presentation of virtual reality to the user and control for ending presentation of virtual reality. When the virtual reality presentation control unit 3010 determines to start presentation of virtual reality, the virtual reality presentation control unit 3010 instructs the virtual reality generation processing unit 3020 to generate a new virtual space for the virtual reality to be presented. In addition, the virtual reality presentation control unit 3010 issues an instruction to arrange a virtual reality object related to the virtual reality to be presented.

When receiving a virtual space generation instruction from the virtual reality presentation control unit 3010, the virtual space generation processing unit 3020 generates a virtual space that is an information space for virtual reality presentation. Specifically, the virtual space generation processing unit 3020 generates a coordinate space having a predetermined area as a virtual space.

When the virtual reality object placement processing unit 3030 receives a virtual reality object placement instruction from the virtual reality presentation control unit 3010, the virtual reality object placement processing unit 3030 reads the virtual reality object from the virtual reality content storage unit 1050 and generates the virtual space generation processing unit 3020. Place in the virtual space. The virtual reality object placement processing unit 3030 reads the virtual reality object from the virtual reality content storage unit 1050 and places it in the virtual space so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020.

A virtual reality degree is assigned to each virtual reality object stored in the virtual reality content storage unit 1050. FIG. 46 is an example of a management table showing the correspondence between the virtual reality object and the virtual reality level. Each virtual reality object is identified by an identification number assigned uniquely, and a virtual reality degree is assigned to each virtual reality object. For example, since the dinosaur Tyrannosaurus does not exist in the real world, a relatively high virtual reality is given to the virtual reality object that models the Tyrannosaurus. On the other hand, Shiba Inu is also present in the real world, so it has a relatively low virtual reality.

For example, assume that the presentable virtual reality degree calculated by the presentable virtual reality degree calculation unit 3030 is 70. The virtual reality object arrangement processing unit 3030 reads out the virtual reality objects from the virtual reality objects stored in the virtual reality content storage unit 1050 so that the total is 70 or less and arranges them in the virtual space.

  Based on the user state detected by the user state detection unit 1010, the viewpoint control unit 3040 includes viewpoint position coordinates corresponding to the user's eye position and line-of-sight direction in the virtual space generated by the virtual space generation processing unit 3020. Control gaze direction. The viewpoint control unit 3040 calculates viewpoint position coordinates and a line-of-sight direction corresponding to the user's eye position and line-of-sight direction from the change in the inclination of the user's head and the change in acceleration detected by the user state detection unit 1010, respectively. Update.

The drawing processing unit 1031 generates a video of the virtual reality object arranged in the virtual space as a virtual reality presentation video. Specifically, the rendering processing unit 1031 uses, as an image plane, a virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction placed in the virtual space controlled by the viewpoint control unit 3040. A two-dimensional image is acquired by a projection process for projection. The drawing processing unit 1031 performs a clipping process for cutting the projected video in accordance with the display area in the video display unit 1040, and performs a shading process for correcting the luminance of the virtual reality object or a texture for pasting texture data on the surface of the virtual reality object. A video for virtual reality presentation is generated by performing various processes such as a mapping process and a hidden surface process for hiding a part of the virtual reality object.

The video display unit 1040 displays a video of the virtual reality object drawn by the drawing processing unit 1031.

  As described above, according to the virtual reality presentation system according to the twelfth embodiment, a virtual reality object is selected and arranged in the virtual space so as to be within the virtual reality that can be presented, and the arranged virtual reality object Is displayed as a virtual reality presentation video. Therefore, the difference between the virtual reality world sensed by the user and the real world can be controlled by selecting the virtual reality object associated with the virtual reality degree.

In the above description, it is preferable that the virtual reality degree related to the presentable virtual reality world calculated by the virtual reality degree calculation unit 1020 is changed according to the state of the user. For example, when the user is exercising intensely, the display control of the virtual reality presentation video displayed on the video display unit 1040 cannot follow, and the virtual world that is artificially created by the user and the real world that the user senses. There is a high possibility that it will not be consistent with. That is, the viewpoint control unit 3040 sequentially updates the viewpoint position coordinates and the line-of-sight direction based on the user state change detected by the user state detection unit 1010, and the virtual reality object based on the updated viewpoint position coordinates and the line-of-sight direction Is drawn and a video is generated, but the user's state changes greatly during the processing time required to generate the video, so that a video that is not based on the current user state may be displayed.

  Therefore, as in the virtual reality presentation system 3000b illustrated in FIG. 47, the presentable virtual reality calculation unit 1020 calculates the presentable virtual reality based on the user state detected by the user state detection unit 1010. A configuration is preferable. For example, the presentable virtual reality calculation unit 1020 calculates the presentable virtual reality based on the acceleration applied to the user's head detected by the user state detection unit 1010, the user's moving speed, and the like. For example, it is calculated that a relatively low virtual reality can be presented as the acceleration applied to the user's head increases.

Also, for example, the presentable virtual reality calculation unit 1020 calculates that a relatively low virtual reality can be presented as the brain actively functions based on the strength of the user's brain waves. This is because it is possible to present a higher virtual reality world because the judgment of recognizing the difference between the virtual reality world and the real world is lower when the brain activity is stagnant. is there. For example, the presentable virtual reality calculation unit 1020 calculates that a relatively high virtual reality can be presented when a θ wave or a δ wave is detected in a brain wave at a predetermined threshold or more.

Note that the calculation formula for calculating the virtual reality that can be presented can be selected and used as appropriate. Since there is a difference for each user, the calculation formula (calculation method) may be set for each user.

In addition, it is even better to calculate a virtual reality that can be presented in combination with the illusion degree described in the eleventh embodiment.

Note that what kind of virtual reality the user wants to experience differs depending on the user and the mood of the user at that time. Therefore, the configuration of the virtual reality presentation system 3000c shown in FIG. 48 is even better. The virtual reality presentation system shown in FIG. 48 includes a virtual reality service request input unit 3050 and a contractor information storage unit 3060.

The virtual reality service request input unit 3050 inputs information for specifying the requested virtual reality service. For example, the user inputs a virtual reality service that the user wants to receive from the virtual reality service request input unit 3050. Here, the virtual reality service is a service related to the presentation of virtual reality and is a service provided to the user by the virtual reality presentation system 3000c.

The contractor information storage unit 3060 stores contractor information that summarizes information of each user. Each user has a contract with a virtual reality service provider in advance to receive provision of the virtual reality service, and the contents of the contract are registered in the contractor information.

FIG. 49 shows a part of the contractor information. A virtual reality service that can be used by each user is registered in the contractor information.

When a request for a virtual reality service is input from the virtual reality service request input unit 3050, the virtual reality presentation control unit 3010 refers to the contractor information stored in the contractor information storage unit 3060 and is requested. It is determined whether or not the virtual reality service can be provided to the user. As a result of the determination, when it is determined that the requested virtual reality service is provided to the user, the virtual reality presentation control unit 3010 corresponds to the virtual space generation unit 3020 and the virtual reality object placement unit 3030, respectively. An instruction to start the process is issued.

  FIG. 50 is an example of a management file for managing virtual reality objects stored in the virtual reality content storage unit 1050 in the virtual reality presentation system 3000c. Each virtual reality service stores a virtual reality object to be used, and a virtual reality degree is associated with the virtual reality object.

The virtual reality object placement unit 3030 reads the virtual reality object associated with the virtual reality service instructed from the virtual reality presentation control unit 3010 within the range of virtual reality that can be presented, and places the virtual reality object in the virtual space.

Thus, the virtual reality service requested by the user can be presented to the user in the virtual reality world while controlling the virtual reality.

<Embodiment 13>
The virtual reality presentation system is intended to enrich the user's life by presenting the user with a virtual reality world that is an extension of the physical world with physical and social constraints. Here, the user senses the virtual reality world by viewing both of the virtual reality presentation images displayed together with the real world, and therefore, depending on the compatibility between the real world scenery and the virtual reality presentation images. It is preferable that the virtual reality changes.

For example, when a virtual reality object that is a dinosaur is placed in a virtual space and a video of the dinosaur is displayed, if the dinosaur appears in front of the user when at home, it is far from the real scene. Therefore, the virtual reality becomes high. On the other hand, when the user is at the zoo, even if a dinosaur is seen next to the lion, it is a relatively “possible sight”, so the virtual reality is relatively low.

As described above, even when the virtual reality presentation system displays the same video, the virtual reality degree of the virtual reality world felt by the user varies depending on the surrounding situation. Therefore, when presenting the virtual reality world while controlling the virtual reality, it is preferable that the virtual reality presentation video and music are determined according to the surrounding environment of the user.

FIG. 51 is a block diagram illustrating a configuration of a virtual reality presentation system 4000 according to the thirteenth embodiment. The virtual reality presentation system 4000 includes a user state detection unit 1010, a presentable virtual reality calculation unit 1020, a video generation processing unit 1030, a video display unit 1040, a virtual reality content storage unit 1050, and a virtual reality being presented. A calculation unit 1060, a virtual reality presentation control unit 3010, a virtual space generation processing unit 3020, a virtual reality object placement processing unit 3030, a viewpoint control unit 3040, an ambient condition detection unit 4010, an ambient condition identification unit 4020, Is provided.

The surrounding situation detection unit 4010 detects the surrounding situation of the user. The ambient condition detection unit 4010 includes, for example, a camera that captures scenery around the user, and detects the environment around the user by capturing the user's surroundings with the camera.

The surrounding situation specifying unit 4020 specifies the surrounding situation of the user based on the detection result in the surrounding situation detection unit 4010. Specifically, the surrounding situation specifying unit 4020 specifies the surrounding situation of the user as one of a plurality of previously registered classifications. “Home”, “Outdoor”, “Park”, “Station”, “Toilet”, “One person”, “Road”, “Zoo”, “Theme park”, “Hamabe”, “Yamanaka”, “Pool”, “Urban”, “ “Countryside” is registered in advance, and the surrounding situation specifying unit 4020 obtains a correlation value between the detection result in the surrounding situation detection unit 4010 and these registered situations, so that the correlation value is the highest. Identifies the current user's surroundings.

The virtual reality content storage unit 1050 stores virtual reality objects in association with virtual reality for each situation. FIG. 52 is an example of a management file indicating the association between the virtual reality object and the virtual reality for each situation. The virtual reality object is associated with the virtual reality level for each situation.

For example, the virtual reality object identified by V0003 is a virtual reality object that models a blue dragon that is a fantasy creature. Because it is an imaginary creature that resembles a dinosaur, virtual reality is low because it is `` possible '' if you are outdoors, rather than being at home, especially if you are in a park or zoo For this reason, the virtual reality is set lower.

The virtual reality object placement unit 3030 reads the virtual reality object from the virtual reality content storage unit 1050 and places it in the virtual space so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020. Here, the virtual reality object placement unit 3030 is a virtual reality degree associated with the virtual reality object to be placed, and the virtual reality degree associated with the situation around the user identified by the ambient situation identification unit 4020. A virtual reality object is selected so that the total does not exceed the presentable virtual reality.

As described above, according to the virtual reality presentation system according to the thirteenth embodiment, an appropriate virtual reality object is selected according to the situation around the user and placed in the virtual space. A virtual reality object is sensed. Accordingly, the user can easily enter the virtual reality world.

Note that the virtual reality presentation system according to the thirteenth embodiment is more favorable when configured as shown in FIG. The virtual reality presentation system 4000b includes a user state detection unit 1010, a presentable virtual reality calculation unit 1020, a video generation processing unit 1030, a video display unit 1040, a virtual reality content storage unit 1050, and a virtual reality being presented. Calculation unit 1060, audio generation processing unit 1070, audio output unit 1080, virtual reality presentation control unit 3010, virtual space generation processing unit 3020, virtual reality object placement processing unit 3030, viewpoint control unit 3040, and surroundings A situation detection unit 4010, an ambient situation identification unit 4020, an image analysis processing unit 4030, a light source control unit 4040, and a model data arrangement processing unit 4050 are provided.

  The ambient condition detection unit 4010 includes a front camera 4011 that captures the front of the user and acquires a forward scenery image, and a peripheral camera 4012 that captures the periphery of the user such as both sides and the rear of the user and acquires a peripheral scenery image. Is done.

The image analysis processing unit 4030 analyzes images captured by the front camera 4011 and the peripheral camera 4012 included in the ambient condition detection unit 4010 and extracts predetermined parameters. Specifically, the image analysis processing unit 4030 includes an object identification processing unit 4031, an optical component extraction processing unit 4032, an object identification processing unit 4033, and a model data generation processing unit 4034.

The object identification processing unit 4031 identifies an object that appears in the front scene image acquired by the front camera 4011 or the peripheral scene image acquired by the peripheral camera 4012. Specifically, the object specification processing unit 4031 performs boundary specification processing for specifying the boundary of each object appearing in the image by comparing adjacent pixels of these images, and a plurality of objects included in the image Is identified.

The optical component extraction processing unit 4032 extracts information related to the brightness (illuminance) of the user and the position of the light source based on the front scene image acquired by the front camera 4011 and the peripheral scene image acquired by the peripheral camera 4012. .

  The object identification processing unit 4033 identifies what the object specified by the object specification processing unit 4031 is. For example, the object identification processing unit 4033 performs a matching process between the image of each object stored in the database and each object specified by the object specifying processing unit 4031, and is captured by the user state detecting unit 4010. Identify what the object is in the image.

  The model data generation processing unit 4034 generates model data that defines the appearance shape of each object specified by the object specification processing unit 4031. The model data generation processing unit 4034 generates the model data by modeling each object existing around the user with a 3D polygon based on the position difference of the same object included in images captured by a plurality of cameras. To do.

The surrounding situation specifying unit 4020 specifies what kind of situation the user is surrounding based on the objects existing around the user identified by the object identification processing unit 4033. The ambient situation specifying unit 4020 stores, for example, a management file that summarizes the correspondence between the ambient situation as shown in FIG. 54 and the objects related to the situation. The ambient situation identification unit 4020 refers to the management file and identifies the situation that includes the largest number of objects identified by the object identification processing unit 4033 as the situation around the user.

The light source control unit 4040 controls the position and illuminance of the light source in the virtual space generated by the virtual space generation processing unit 3020 based on the information on the position and illuminance of the light source extracted by the optical component extraction unit 4032.

The model data arrangement processing unit 4050 arranges model data such as an object or a person existing around the user generated by the model data generation processing unit 4034 in the virtual space generated by the virtual space generation unit 3020. Here, it is more preferable that the model data arrangement processing unit 4050 arranges the model data in a layer different from the virtual space in which the virtual reality object arrangement processing unit 3030 arranges the virtual reality object.

That is, the virtual space generation processing unit 3020 generates a virtual space of a plurality of layers of a first layer for virtual reality object placement and a second layer for placement of model data of a real object. The virtual reality object arrangement processing unit 3030 arranges the virtual reality object read from the virtual reality content storage unit 1050 in the first layer of the virtual space, and the model data arrangement processing unit 4050 is generated by the model data generation processing unit 4034. The model data of the object in the real world is arranged in the second layer. As described above, the hidden surface processing in the subsequent drawing processing can be simplified by arranging the layers in different layers.

The drawing processing unit 1031 generates a virtual reality presentation video by drawing a virtual reality object arranged in the first layer of the virtual space. Specifically, the rendering processing unit 1031 projects the virtual reality arranged in the first layer of the virtual space on the image plane based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 3040. Obtain a two-dimensional image of the object.

Similarly, the drawing processing unit 1031 projects the model data of the real object arranged in the second layer of the virtual space onto the image plane based on the viewpoint position and the line-of-sight direction used for the first layer projection processing. Thus, a projection image of a real object is generated. The drawing processing unit 1031 performs hidden surface processing that is a projection image of the above-described real object in the two-dimensional image of the virtual reality object and is hidden by the projection image of the object positioned on the user side with respect to the virtual reality object.

Subsequently, the drawing processing unit 1031 performs a clipping process that cuts the two-dimensional image obtained by the projection process into a predetermined size in accordance with the resolution corresponding to the video display unit 1040.

Subsequently, the drawing processing unit 1031 performs a texture mapping process for pasting the texture to the two-dimensional image of the virtual reality object.

Subsequently, the drawing processing unit 1031 performs a shading process for adjusting the luminance of the surface of the two-dimensional image of the virtual reality object based on the position and illuminance of the light source controlled by the light source control unit 3040.

The drawing processing unit 1031 generates an image for virtual reality presentation by performing anti-aliasing processing, display position correction processing based on a change in user state during the drawing processing, and the like as necessary. The video display unit 1040 displays the virtual reality presentation video obtained through the above processing.

The audio generation processing unit 1070 reads the virtual reality presentation audio data corresponding to the virtual reality service determined by the virtual reality presentation control unit 3010 from the virtual reality content storage unit 1050 and distributes the audio data to each audio channel. By performing amplification processing for amplifying the level, audio data for virtual reality presentation output from each audio channel is generated.

In performing the distribution process, the sound generation processing unit 1070 converts a plurality of sound data based on a relative vector between the position coordinates of the virtual reality object arranged in the virtual space and the viewpoint position coordinates controlled by the viewpoint control unit 3040. It is preferable to distribute the sound channels and reproduce the three-dimensional sound.

The audio output unit 1080 outputs audio by converting the audio data of each audio channel generated by the audio generation processing unit 1070 into an analog electric signal, and then converting it into air vibration and releasing the sound to the outside.

Note that even if the same virtual reality object is arranged near the user, the impression received by the user differs depending on whether it is arranged near the user. For example, when a virtual reality object of a dragon that cannot be in the real world is arranged at a position 3 m ahead of the user, the virtual reality world has a high virtual reality, but when a dragon is arranged 50 m ahead of the user The virtual reality is low because it is not visible to the user.

Therefore, the virtual reality object placement unit 3030 controls the distance from the user so as to be within the virtual reality calculated by the presentable virtual reality calculation unit 1020, and reads the virtual reality object read from the virtual reality content storage unit 1050. Place in the virtual space.

For example, the virtual reality service determined by the virtual reality presentation control unit 3010 to start providing is a virtual reality service “Let's hunt the dragon”, and the presentable virtual reality degree set by the presentable virtual reality calculation unit 1020 Assume that the initial value is 50. Here, the virtual reality when the dragon, which is a virtual reality object, is placed 3 m ahead of the user is 70, the virtual reality is 50 when the user is placed 10 meters away, and the virtual reality 40 is 40 meters when placed 20 meters away. Suppose that

In this case, the virtual reality object placement processing unit 3030 reads out the model data of the dragon, which is a virtual reality object related to the virtual reality service instructed from the virtual reality presentation control unit 3010, from the virtual reality content storage unit 1050 and stores it in the virtual space. Deploy. Here, the virtual reality object arrangement processing unit 3030 has a virtual reality degree of 10 m or more from the user position so that the virtual reality degree when the read virtual reality object is arranged in the virtual space is less than 50, which is the virtual reality degree that can be presented. Place a virtual reality object at a distance. As described above, it is more preferable that the virtual reality object arrangement processing unit 3030 is configured to determine the arrangement position of the virtual reality object in the virtual space so as to satisfy the presentable virtual reality.

As the user position, the viewpoint position updated by the viewpoint control unit 3040 based on the user state detected by the user state detection unit 1010 can be used as it is as the user position.

It is further preferable that the virtual reality content storage unit 1050 is configured to store the virtual reality object in association with the virtual reality level set for each virtual reality object. The virtual reality object arrangement processing unit 3030 reads from the virtual reality content storage unit 1050 a virtual reality object associated with a virtual reality level lower than the virtual reality level calculated by the presentable virtual reality level calculation unit 1020. The virtual space generation processing unit 3020 arranges the virtual space.

The virtual reality object placement processing unit 3030 obtains the surplus virtual obtained by subtracting the virtual reality calculated by the presenting virtual reality calculation unit 1060 from the virtual reality calculated by the presentable virtual reality calculation unit 1020. It is preferable that a virtual reality object associated with a virtual reality level lower than the reality level is read from the virtual reality content storage unit 1050 and arranged in the virtual space generated by the virtual space generation processing unit 3020.

The virtual reality object placement processing unit 3030 also displays the virtual reality being placed in the virtual space when the virtual reality in the virtual reality world that can be presented is lower than the virtual reality in the virtual reality world that is currently being presented. By deleting the real object from the virtual space, the virtual reality level of the virtual reality world to be presented falls within the virtual reality level of the virtual reality world that can be presented. The presentable virtual reality calculation unit 1020 calculates the presentable virtual reality, and the presenting virtual reality calculation unit 1060 calculates the total virtual reality of the virtual reality objects currently arranged in the virtual space. . When the value obtained by subtracting the presenting virtual reality from the presentable virtual reality is negative, the virtual reality object placement processing unit 3030 reduces the virtual space so that the presenting virtual reality is less than the presentable virtual reality. The virtual reality object to be deleted is selected from the virtual reality objects arranged in the virtual space object and deleted from the virtual space.

<Embodiment 14>
The user wears the virtual reality presentation device and experiences the virtual reality world. The virtual reality presentation device worn directly by the user is preferably lightweight and compact in order to improve wearability and portability, but various controls and processes are required to present the virtual reality world. . The virtual reality presentation system according to the fourteenth embodiment is a lightweight virtual reality presentation device that is actually worn by the user by sharing the processing necessary for presenting the virtual reality world with a plurality of devices and systems.・ It is aimed at miniaturization.

FIG. 55 is a block diagram illustrating a configuration of the virtual reality presentation system according to the fourteenth embodiment. The virtual reality presentation system includes a virtual reality presentation device 500 and a virtual reality service providing system 5000.

The virtual reality service providing system 5000 is a system that provides a user with a virtual reality service, which is a service that presents the virtual reality world, and includes one or a plurality of information processing servers and databases.

  The virtual reality service providing system 5000 includes a communication unit 5010, a virtual reality service provision control unit 5020, a virtual reality object behavior control unit 5030, a layer order update processing unit 5040, a layer information generation processing unit 5050, a presentable virtual Reality calculation unit 1020, video generation processing unit 1030, virtual reality content storage unit 1050, presenting virtual reality calculation unit 1060, audio generation processing unit 1070, virtual space generation processing unit 3020, virtual reality object An arrangement processing unit 3030, a viewpoint control unit 3040, an ambient condition specifying unit 4020, an image analysis processing unit 4030, a light source control unit 4040, and a model data arrangement processing unit 4050 are provided.

The communication unit 5010 communicates with the virtual reality presentation device 500 via a communication network such as the Internet. The communication unit 5010 includes at least a receiving unit 5011 that receives user status information indicating the user status, and a transmitting unit 5011 that transmits video data for virtual reality presentation.

The receiving unit 5011 receives user state information indicating a user's state and a surrounding scene image obtained by photographing the user's surroundings. The user state information and the surrounding scene image each include user state detection time information (user state detection timing information) indicating the timing when the user state is detected, and an ambient scene image shooting time indicating the timing when the surrounding scene image is captured. Information (ambient status detection timing information) is included. The transmission unit 5011 transmits video encoded data for virtual reality presentation, audio encoded data for virtual reality presentation, layer information, various notification information, and the like.

The virtual reality service provision control unit 5020 controls provision of the virtual reality service. Specifically, the virtual reality service provision control unit 5020 reads out and executes a control program related to the virtual reality service to be provided from the virtual reality content stored in the virtual reality content storage unit 1050, thereby executing the virtual reality service. Start providing.

The virtual reality content storage unit 1050 stores virtual reality content that is data for realizing a virtual reality service. The virtual reality content includes a virtual reality object, a virtual reality service control program that controls the progress of the virtual reality service, a virtual reality object behavior control program that controls the behavior of the virtual reality object, and audio data for virtual reality presentation. It stores voice data for virtual reality presentation.

When the provision of the virtual reality service is started, the virtual reality service provision control unit 5020 generates virtual reality service provision start notification information for notifying the start of provision of the virtual reality service.

FIG. 56 is a diagram illustrating an example of the virtual reality service provision start notification information. The virtual reality service provision start notification information includes a transmission source address that is the address of its own server, a transmission destination address that is an address of the virtual reality presentation device 500 that is the transmission destination, and the information is virtual reality service provision start notification information. And a communication identification that is a number assigned to each packet in order to distinguish communication generated between the virtual reality presentation device 500 and the virtual reality presentation system 5000 in relation to the virtual reality service from other communication It includes a number, virtual reality service identification information for identifying a virtual reality service to be provided, virtual reality control parameter information used in control and processing necessary for providing the virtual reality service, and the like.

As virtual reality control parameter information, detected user state specifying information for specifying what kind of user state is detected as the user state, and user state transmission indicating the transmission frequency of the user state information indicating the detected user state The frequency, the transmission frequency of the surrounding scenery image that is an image of the surrounding scenery of the photographed user, the surrounding scenery image encoding format indicating the image compression encoding format of the surrounding scenery image, and the video frame rate indicating the frame rate of the video to be displayed The virtual reality presentation video data encoding format indicating the encoding format of the virtual reality presentation video data appropriately transmitted from the virtual reality service providing system 5000 is included.

In addition, the virtual reality service provision control unit 5020 controls the virtual reality service provision end notification information for notifying the end of provision of the currently provided virtual reality service or the control parameter change notification in the virtual reality service as necessary. Various information such as parameter change notification information is generated. These pieces of information generated by the virtual reality service provision control unit 5020 are transmitted from the transmission unit 5012 to the virtual reality presentation device 500.

When the provision of the virtual reality service is started, the virtual reality service provision control unit 5020 issues a virtual space generation instruction and a virtual reality object arrangement instruction to the virtual space generation processing unit 3020 and the virtual reality object arrangement processing unit 3030, respectively. .

In addition, the virtual reality service provision control unit 5020 issues an instruction to generate video data for virtual reality presentation and an instruction to generate audio data for virtual reality presentation to the video generation processing unit 1030 and the audio generation processing unit 1070, respectively.

The virtual space generation processing unit 3020 generates a virtual space used for the virtual reality service based on the virtual space generation instruction from the virtual reality service provision control unit 5020. The virtual space is a coordinate space having a predetermined area, and is an information processing space created artificially so as to correspond to a space in the real world.

Based on the virtual reality object placement instruction from the virtual reality service provision control unit 5020, the virtual reality object placement processing unit 3030 reads out the virtual reality object related to the virtual reality service from the virtual reality content storage unit 1050, and performs virtual space generation processing. Arranged in the virtual space generated by the unit 3020.

Here, the virtual reality object placement processing unit 3030 selects the virtual reality object so as to be within the virtual reality of the presently available virtual reality calculated by the presentable virtual reality calculation unit 1020 and selects the virtual space. To place.

In addition, when there are a plurality of virtual reality objects to be arranged, the virtual reality object arrangement processing unit 3030 arranges the read virtual reality objects in the virtual spaces of different layers.

That is, the virtual space generation processing unit 3020 generates a virtual space of a plurality of layers having a common origin, and the virtual reality object arrangement processing unit 3030 receives the plurality of virtual reality objects read from the virtual reality content storage unit 1050, respectively. Place in a virtual space of different layers.

FIG. 57 shows a case where the plurality of virtual reality objects are arranged in different layers of the virtual space. Three virtual reality objects, an elephant, a dog, and a lion, are arranged in three different layers of virtual space, each having a common origin.

The viewpoint control unit 3040, based on the user state information received by the receiving unit 5011, in the virtual space generated by the virtual space generation processing unit 3020, the viewpoint position coordinates and the line-of-sight direction corresponding to the user's eye position and line-of-sight direction. And control. The viewpoint control unit 3040 calculates viewpoint position coordinates corresponding to the user's eye position and line-of-sight direction from the change in the inclination and acceleration of the user's head specified based on the inclination information and acceleration information included in the user state information. And the line-of-sight direction are calculated and updated. Here, since the user state information includes user state detection time information that is a time stamp indicating the timing at which the corresponding user state is detected, the viewpoint control unit 3040 can display the current viewpoint position coordinates and the line-of-sight direction. Manages the viewpoint position coordinates and the line-of-sight direction set based on the user state detected at which time.

The virtual reality object behavior control unit 5030 controls the behavior of the virtual reality object arranged in the virtual space. Some virtual reality objects arranged in each layer of the virtual space by the virtual reality object arrangement processing unit 3030 are configured to be operable like a virtual reality object of a human being or an animal, for example. . The virtual reality object behavior control unit 5030 changes the appearance shape of the virtual reality object arranged in the virtual space by reading out and executing the behavior control program of the virtual reality object from the virtual reality content storage unit 1050. Or moving a virtual reality object in the virtual space.

The presenting virtual reality calculation unit 1060 calculates a virtual reality related to the virtual reality world currently being presented. The presenting virtual reality calculation unit 1060 is based on the type and arrangement position of the virtual reality object that the virtual reality object arrangement processing unit 3030 arranges in the virtual space, the situation around the user specified by the surrounding situation specifying unit 4020, and the like. The virtual reality degree related to the virtual reality world currently being presented is calculated. When a plurality of virtual reality objects are arranged in a virtual space of a plurality of layers, the presenting virtual reality calculation unit 1060 presents by summing the virtual reality associated with these virtual reality objects. It is preferable that the virtual reality related to the virtual reality world is calculated.

The presentable virtual reality calculation unit 1020 calculates the virtual reality related to the virtual reality world that can be presented to the user at present. The presentable virtual reality calculation unit 1020 adds, subtracts, multiplies, and divides a predetermined value on the basis of the virtual reality related to the world of the virtual reality being presented calculated by the presenting virtual reality calculation unit 1060 By doing so, it is preferable to calculate the virtual reality related to the world of virtual reality that can be presented.

The image analysis processing unit 4030 extracts predetermined information by performing predetermined analysis processing on the surrounding scenery image obtained by photographing the scenery around the user received by the receiving unit 5012. The surrounding scene image is an image taken by a plurality of cameras included in the virtual reality presentation device 500.

For example, the image analysis processing unit 4030 extracts an optical component from the surrounding scenery image, and extracts information such as the average illuminance around the user and the position and direction of the light source as optical component information.

In addition, the image analysis processing unit 4030 performs processing for comparing the positions of the objects included in the plurality of surrounding scenery images and comparison processing for comparing adjacent pixels of the same surrounding scenery image, so that each object included in the surrounding scenery image is compared. Specify the position and shape. The image analysis processing unit 4030 generates model data by modeling the external shape of the object or the like based on the position or shape of the object or the like existing in the real world around the user.

The light source control unit 4040 updates the position coordinates of the light source and the illuminance or luminance of the light source in the virtual space generated by the virtual space generation processing unit 3020 based on the optical component information extracted by the image analysis processing unit 4030. For example, the light source control unit 4040 sets the illuminance or luminance of the light source at infinity based on the average luminance in the surrounding scene image obtained by the image analysis processing unit 4030. Also, a local light source is newly arranged at a corresponding position in the virtual space based on the optical component information extracted based on the position of the shadow appearing in the surrounding scene image in the image analysis processing unit 4030. In addition, the light source control unit 4040 may set the color of the light source based on the optical component information extracted by the image analysis processing unit 4030.

The model data arrangement processing unit 4050 arranges model data such as an object existing in the real world around the user generated by the image analysis processing unit 4030 in the virtual space. The model data arrangement processing unit 4050 identifies the image of the surrounding scene image taken from which position and in which direction based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 3040, and performs image analysis. The model data of each object generated by the processing unit 4030 is arranged in the virtual space.

Here, it is preferable that the virtual space generation processing unit 3020 generates a virtual space of a model data dedicated layer in which model data of an object existing in the real world is arranged. The model data arrangement processing unit 4050 arranges the model data of the real world object generated by the image analysis processing unit 4030 in the virtual space of the model data dedicated layer.

The surrounding situation specifying unit 4020 specifies the surrounding situation of the user based on the result of the image analysis processing in the image analysis processing unit 4030. For example, the surrounding situation specifying unit 4020 displays the surrounding situation of the user as “indoor”, “outdoor”, “park”, “station”, “road”, “home”, “in the mountain”, “beach”, “ The situation around the user is specified by selecting from any of the categories registered in advance, such as “Department Store”, “Rural”, “Ground”, and “School”.

The drawing processing unit 1031 generates a virtual reality presentation video by drawing a virtual reality object arranged in the virtual space based on a virtual reality presentation video generation instruction from the virtual reality service provision control unit 5020. . Here, the drawing processing unit 1031 generates a virtual reality presentation video for each layer by performing drawing processing for each virtual reality layer.

The drawing processing unit 1031 selects one of the layers in the virtual space and the model data dedicated layer, and in the virtual space of the selected layer based on the current viewpoint position coordinates and the line-of-sight direction set by the viewpoint control unit 3040, respectively. Projection processing is performed for projecting the virtual reality object and the model of the object in the real world onto the image plane arranged forward in the line-of-sight direction from the viewpoint position coordinates. The drawing processing unit 1031 deletes the model portion of the object in the real world from the two-dimensional projection image obtained by the projection processing, thereby obtaining the two-dimensional projection image of the virtual reality object arranged in the selected layer. get.

  The drawing processing unit 1031 performs a clipping process that cuts the two-dimensional projection image obtained by the projection process into a predetermined size. Here, the drawing processing unit 1031 is characterized by cutting out in a range exceeding the resolution of the display panel of the virtual reality presentation device 500. For example, when the display panel of the virtual reality presentation device 500 has a resolution of 1080 pixels × 1920 pixels, the drawing processing unit 1031 performs a clipping process of cutting the two-dimensional projection image by 1188 × 2112, which is 10% higher. It should be noted that how much the size is to be cut out may be determined based on the magnitude of change in the user state. This is because when the user is stationary, the subsequent correction process is not necessary, and it is not necessary to cut it out into large parts.

  Next, the drawing processing unit 1031 performs a texture mapping process in which the texture mapping data of the virtual reality object is pasted on the two-dimensional projection image after the clipping process.

Next, the drawing processing unit 1031 performs a shading process for adding a shadow to the two-dimensional projection image based on the light source set by the light source control unit 4040. The drawing processing unit 1031 performs anti-aliasing processing as necessary to obtain a virtual reality presentation video for the layer.

Here, the drawing processing unit 1031 assigns information indicating the time at which the virtual reality presentation video of the layer generated by drawing is a video drawn based on the user state, and the virtual processing of the layer is performed. Reality-presenting video is generated. The viewpoint position coordinates and the line-of-sight direction used for the drawing process are updated and set by the viewpoint control unit 3040. The time when the user state used when setting the currently used viewpoint position coordinates and the line-of-sight direction is detected is set. The user status detection time information, which is the time stamp shown, is given, and the virtual reality presentation video of the layer is generated.

The drawing processing unit 1031 generates a virtual reality presentation video for each layer by drawing the virtual space of each layer generated by the virtual space generation processing unit 3020 by parallel processing.

The video encoding processing unit 1032 encodes the virtual reality presentation video of each layer generated by the drawing processing unit 1031 to generate layered virtual reality presentation encoded video data. The layer-by-layer virtual reality presentation encoded video data is transmitted from the transmission unit 5011 to the virtual reality presentation device 500.

The layer order update processing unit 5040 updates the order of each layer in the virtual space generated by the virtual space generation processing unit 3020. Since the rendering processing unit 1031 performs rendering for each layer and generates virtual reality presentation videos for each layer, it is necessary to finally synthesize a plurality of virtual reality presentation videos for each layer in the virtual reality presentation device 500. is there. Therefore, the layer order update processing unit 5040 sets the order of the layers in each virtual space and updates the order of the layers.

The layer order is determined by the relationship between the position of the user and the position of the virtual reality object. That is, the layer order is determined in order from the layer where the virtual reality object closer to the user is arranged. The position of the user changes as appropriate, and the position of the virtual reality object also changes due to behavior control performed by the virtual reality object behavior control unit 5030. Therefore, the layer order update processing unit 5040 is arranged in the user's viewpoint position coordinates controlled by the viewpoint control unit 3040 based on the user state information received by the receiving unit 5011 and in each layer of the virtual space. The layer order is updated based on the position coordinates of the virtual reality object.

For example, in the case where virtual reality objects A to C are arranged in layers 1 to 3 of the virtual space as shown in FIG. 58, in the state 1, the virtual reality objects A, B, and C are in order from the side closer to the user. . Therefore, the layer order update processing unit 5040 assumes the order of layers 1, 2, and 3. Here, when the positional relationship of the state 2 is obtained due to the user moving in the real world, the virtual reality objects are in the order of B, C, and A from the side closer to the user. Accordingly, the layer order update processing unit 5040 manages the layer order by changing the layer order to layers 2, 3, and 1.

The layer information generation processing unit 5050 generates layer information indicating the current layer order based on the update processing in the layer order update processing unit 5040. The layer information generation processing unit 5050 includes layer order information indicating the layer order, positional relationship information indicating the positional relationship between the user and each layer, the type and number of virtual reality objects included in each layer, and the importance of each layer. It is preferable to generate layer information including information such as degree. The positional relationship information is information indicating a distance and a vector from the user position to the virtual reality object included in each layer. The layer information is transmitted from the transmission unit 5012 to the virtual reality presentation device 500 together with the encoded video data for virtual reality presentation for each layer.

The voice generation processing unit 1070 reads the virtual reality presentation voice data related to the virtual reality service provided from the virtual reality content storage unit 1050 based on the virtual reality presentation voice generation instruction from the virtual reality service provision control unit 5020, Audio data for virtual reality presentation for each channel is generated. For example, the audio generation processing unit 1070 is based on the relative relationship between the viewpoint position set by the viewpoint control unit 3040 and the position of the virtual reality object arranged by the virtual reality object arrangement processing unit 3030. The virtual reality presentation audio data is generated by distributing the read virtual reality presentation audio data to a plurality of channels.

Here, the audio reproduction processing unit 1070 reads a plurality of virtual reality presentation audio data from the virtual reality content storage unit 1050, distributes each of the virtual reality presentation audio data to the audio data of each channel, and combines them, thereby synthesizing each channel virtual reality presentation audio. Generate data. Here, as a channel, it is preferable to provide a sound output unit 530 of a plurality of channels such as 5.1Ch, 6.1Ch, 7.1Ch on the virtual reality presentation device 500 side so as to realize three-dimensional sound.

The voice generation processing unit 1070 generates the channel-specific virtual reality presentation speech encoded data by encoding the generated channel-specific virtual reality presentation speech data in a predetermined encoding format. The channel-specific virtual reality presentation speech encoded data is transmitted from the transmission unit 5012 to the virtual reality presentation device 500.

The virtual reality presentation device 500 includes a user state detection unit 1010, a video display unit 1040, an audio output unit 1080, an ambient state detection unit 4010, a communication unit 510, and a virtual reality presentation control unit 520.

The user state detection unit 1010 includes an inclination sensor 1011 that detects the inclination of the user's head, an acceleration sensor 1012 that detects the acceleration of the user's head, an orientation sensor 1013 that detects the user's orientation, and a position that detects the position of the user. A sensor 1014, an internal camera 1015 for photographing a user's facial expression, and the like are included. Detection results of these sensors and the like are collected as user state information together with user state detection time information that is a time stamp indicating the detection time, and transmitted from the communication unit 510 to the virtual reality service providing system 5000. A part of the user status information is used for video correction processing in the video display unit 1040.

The ambient state detection unit 4010 includes a line-of-sight camera 4011 that captures the front of the user corresponding to the user's line of sight, and a peripheral camera 4012 that captures the user's periphery such as both sides and the back of the user, and detects the state of the user's surroundings. The line-of-sight cameras 4011 are arranged at positions of the user's left eye and right eye, respectively. Ambient scene images captured and acquired by these cameras are collected as ambient status information together with ambient status detection time information that is a time stamp indicating the shooting time, and transmitted from the communication unit 510 to the virtual reality service providing system 5000. .

The communication unit 510 communicates with the virtual reality service providing system 5000 via a communication network such as the Internet. The communication unit 510 includes a wireless transmission unit 511 that wirelessly transmits user state information and ambient state information detected by the user state detection unit 1010 and the ambient state detection unit 4010, and virtual reality transmitted from the virtual reality service providing system 5000. A wireless receiving unit 512 that wirelessly receives presentation encoded video data, layer information, virtual reality presentation audio encoded data, various types of notification information, and the like.

The video display unit 140 includes a video related data storage unit 141, a video decoding processing unit 142, a video correction processing unit 143, a video composition processing unit 144, a display area selection unit 145, and a display panel 146.

The video related data storage unit 141 temporarily stores the virtual reality presentation video encoded data and the layer information wirelessly received by the wireless receiving unit 512 as video related data.

The video decoding processing unit 142 performs a decoding process on the virtual reality presentation video encoded data for each layer stored in the video related data storage unit 141, and acquires virtual reality presentation video data.

The video correction processing unit 143 performs position correction of the decoded virtual reality presentation video data for each layer based on the user state information and the positional relationship information included in the layer information. Each virtual reality presentation video data generated by the video generation processing unit 1030 is generated based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 3040 based on the user state. Here, the user's viewpoint position and line-of-sight direction change during communication lag between the virtual reality presentation device 500 and the virtual reality service providing server 5000, and the time required for drawing processing, encoding processing, decoding processing, and the like. There is a possibility. For this reason, the video correction processing unit 143 performs a predetermined correction on the generated virtual reality presentation video to correct the video so as to match the current user state.

For example, as shown in FIG. 59, there are two virtual reality presentation videos of layer 1 and layer 2, and relative position vectors (x, x) to the virtual reality object included in each layer indicated by the positional relationship information of each layer. y, z) are (+0.3, +2.2, −0.4) and (+2.1, +21.4, +0.6), respectively. Here, when the user is moving in the −X direction, the viewpoint position and the line-of-sight direction are different between when the video is generated and at the present time. Therefore, the video correction processing unit 143 determines the position of the virtual reality object included in the video based on the user status detected between the user status detection time information given to the virtual reality presentation video and the current time. Correct the size. The video correction processing unit 143 determines how to correct the video of the virtual reality object according to the change in the viewpoint position and the line-of-sight direction, because it is determined by the relative positional relationship between the viewpoint position and the line-of-sight direction and the virtual reality object. Performs a correction process after calculating a correction amount for the virtual reality presentation video for each layer.

The video composition processing unit 144 performs a process of compositing the virtual reality presentation videos of each layer after the correction processing by superimposing them according to the layer order information included in the layer information. The video of the virtual reality object included in the rear layer is hidden by the virtual reality object included in the front layer.

The display area selection unit 145 selects an area to be displayed on the display panel 146 from the virtual reality presentation video after the synthesis process. Since each virtual reality presentation video is a video having a resolution higher than the resolution of the display panel 146 so as to be compatible with the correction process, the virtual reality presentation video is displayed on the display panel 146 from the virtual reality presentation video after the correction process and the synthesis process. Select the range to display.

The display panel 146 displays the virtual reality presentation video in the range selected by the display area selection unit 145. The display panel 146 is two panels arranged in front of the user's left eye and right eye, and the video generation processing unit 1030 generates a virtual reality presentation video for each of the two panels. That is, the viewpoint control unit 3040 sets two viewpoint position coordinates and line-of-sight directions corresponding to the left eye and right eye of the user, and the video generation processing unit 1030 draws from each of these two viewpoints and presents virtual reality for the right eye. Video for virtual reality and video for virtual reality presentation for the left eye are generated.

The audio output unit 1080 performs a decoding process on the encoded audio data for virtual reality presentation received by the wireless reception unit 512, and outputs the audio by emitting the audio data for virtual reality presentation for each channel from the speaker. Output.

The virtual reality presentation control unit 520 controls each process for virtual reality presentation on the virtual reality presentation device 500 side. Based on various notification information and parameter change information received by the wireless reception unit 512, the virtual reality presentation control unit 520 encodes the user status and the surrounding situation detection period, the frame rate of the video display, the data encoding / transmission / transmission Set the decryption format. For example, the virtual reality presentation control unit 520 sets the type of user state to be detected by the user state detection unit 1010 based on the detection user state designation information of the virtual reality control parameter information included in the virtual reality service provision start information, Based on the user state detection frequency, a user state detection frequency to be detected by the user state detection unit 1010 is set.

Further, the virtual reality presentation control unit 520 monitors the state of the virtual reality presentation device 500 such as the remaining battery level, temperature, resource consumption, wireless communication speed, and the like, and collects this information in a report to provide a virtual reality service providing server 5000. Send to. The report is sent to the virtual reality service provision control unit 5020 and used for various controls.

FIG. 60 is an external perspective view of the virtual reality presentation device 500 from obliquely forward and obliquely rearward. The virtual reality presentation device 500 is a head mount display (HMD) device that a user wears on his / her head, and is configured so that the appearance of the outside world can be observed through a transmissive display. The virtual reality presentation device 500 includes at least a frame 501 and a front panel 502.

  A frame 501 is a housing of the virtual reality presentation device 500, and includes an information processing processor such as a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read only memory), a wireless communication processing processor, A battery etc. are arranged. In addition, a dedicated video processor or the like that draws video as necessary is also arranged. The frame 501 is roughly divided into a right frame 501a, a left frame 501b, and a center frame 501c.

  The light frames 501a and 501b are case members that are connected to the left and right ears of the user, respectively, and speakers 181a and 181b that convert analog electric signals into air vibrations and output sound are arranged inside one end, respectively. The end is connected to the center frame 501c.

  At the ends of the light frames 501a and 501b, inclination sensors 1011a and 1011b for detecting inclination and acceleration sensors 1012a and 1012b for detecting acceleration are arranged, respectively. In addition, peripheral cameras 4012a and 4012b for photographing surrounding scenery are arranged outside the right frame 501a and the left frame 501b, respectively.

  Both ends of the center frame 501c are connected to the right frame 501a and the left frame 501b, respectively, and the front panel 502 is connected below the center frame 501c.

  Near the center and near both ends of the center frame 501c, tilt sensors 1011c, 1011d, and 1011e that detect tilt and acceleration sensors 1012c, 1012d, and 1012e that detect acceleration are arranged, respectively.

On the user side of the center frame 501c, internal cameras 1015a and 1015b for photographing a part or all of the facial expression of the user's face are arranged at the positions of the left and right eyes of the user. The internal cameras 1015a and 1015b capture at least a region including the position of the user's eyes and generate a user facial expression video as user state information.

Line-of-sight cameras 4011a and 4011b for photographing a scene corresponding to the user's field of view are arranged in front of the center frame 501c and at the positions of the left and right eyes of the user. The line-of-sight cameras 4011a and 4011b are arranged on the front side of the center frame 501c at positions separated by a distance corresponding to the distance between the left and right eyes of the user.

  The front panel 502 is disposed at a position in front of the user's eyes with the virtual reality presentation device 500 mounted on the head. In the virtual reality presentation device 500, the front panel 502 is a transmissive display, and a right display panel 146a and a left display panel 146b, which are display areas for displaying images, are arranged in front of the right eye and the front of the left eye, respectively.

  61 shows a cross-sectional view of the vicinity of the light display panel 146a of the virtual reality presentation device 500. FIG. A front panel 502 is disposed below the center frame 501c, and a light source 146-1 is disposed below the center frame 501c.

Visible light emitted from the light source 146-1 is irradiated to the front panel 502 side. The front panel 502 has a configuration in which the light guide plate 146-2 and the LCD 146-3 are combined, and the visible light emitted from the light source 146-1 is reflected to the user side by the light guide plate 146-2, and the display-controlled LCD 146 is controlled. The virtual reality presentation video is displayed by passing through -3.

Note that the virtual reality presentation device 500 may be configured as shown in FIG. In the virtual reality presentation device 500 shown in FIG. 62, an LCD 146-3 is disposed between the light source 146-1 and the light guide plate 146-2. Therefore, since the light from the outside is irradiated to the eyes of the user without passing through the LCD 146-3, it is possible to prevent the brightness of the external scenery from becoming dark.

In addition, as shown in FIG. 63, an LCD 146-4 for controlling the irradiation amount of external light that is light from the outside may be arranged. Even if the image of the virtual reality object is displayed on the LCD 146-3, the light from the outside world is mixed and the image of the virtual reality object is seen through, so that the external light at the position of the virtual reality object is blocked on the LCD 145. It is possible to prevent the virtual reality object from being seen through.

As shown in FIG. 64, the light guide plate 146-2 is better positioned so that external light incident on the user's eyes does not pass through. In the configuration of FIG. 64, the contrast of external light can be further improved.

Further, as shown in FIG. 65, an LCD 146-4 for controlling external light may be arranged. The external light passes through the LCD 146-4 and reaches the user's eyes, while the light emitted from the light source 146-1 passes through the LCD 146-3 and is reflected by the light guide plate 146-2 to reach the user's eyes. To do.

Any LCD 146-4 may be used as long as it can control the amount of light transmitted from the outside by voltage control. For example, it is possible to use a display that changes the crystal direction by applying a voltage to increase irregular reflection.

In addition, as shown in FIG. 66, a projection mechanism can be used as the display panel 146. A projection mechanism is arranged as a display panel 146 inside a center frame 501c located at the top of the front panel 502.

  The white light emitted from the light source 146-1 disposed above the center frame 501c is collected by the lens group 146-7a and then polarized by the polarizing converter (PLC) 146-7b in one direction. Then, the light is reflected in the vertical direction by the first reflecting mirror 146-7c. In the white light reflected by the first reflecting mirror 146-7c, only the red light component (R component) is reflected in the horizontal direction by the R reflecting dichroic mirror 146-7d, and the light of the other components is reflected by the R reflecting dichroic mirror. 146-7d is transmitted.

  From the light transmitted through the R reflecting dichroic mirror 146-7d, only the green light component (G component) is reflected in the horizontal direction by the G reflecting dichroic mirror 146-7e, and the remaining blue light component (B component) is reflected by G. It passes through the dichroic mirror 146-7e.

  The B component light that has passed through the G reflecting dichroic mirror 146-7e is reflected in the horizontal direction by the second reflecting mirror 146-7f, and further reflected vertically upward by the third reflecting mirror 146-7g. After passing through -7h, the light enters the cross dichroic prism 146-7j.

  On the other hand, the R component light reflected by the R reflecting dichroic mirror 146-7d is reflected in the vertical direction by the fourth reflecting mirror 146-7k, passes through the second liquid crystal panel 146-7m, and then cross-dichroic prism 146-7j. Is incident on. The G component light reflected by the G reflecting dichroic mirror 146-7e passes through the third liquid crystal panel 146-7n and then enters the cross dichroic prism 146-7j.

  The cross dichroic prism 146-7j combines the three light components R, G, and B that have been separated and passed through the liquid crystal panel, and reflects them in the horizontal direction. The combined light output from the cross dichroic prism 146-7j is reflected in the vertical direction by the galvanometer mirror 146-7p, and is irradiated to the front panel 402 from the irradiation window 146-7q. The galvanometer mirror 146-7p reflects the combined light output from the cross dichroic prism 146-7j while scanning in the horizontal direction (y direction).

  In the front panel 502, a light guide plate 146-2 is disposed below the irradiation window 146-7q, and the light reflected by the light guide plate 146-2 toward the user is refracted by the front panel 502 to be parallel light. And enters the user's eyes. Since the light is operated in the horizontal direction by the galvanometer mirror 146-7p, the light reflected by the light guide plate 146-2 is scanned in the vertical direction (z direction).

  Note that each member in FIG. 66 has a configuration that is long and thin in the x direction, that is, the direction perpendicular to the paper surface, and collectively displays a line in the x direction in the right-eye image. The one row of images is scanned in the y direction by the galvanometer mirror 146-7p and sequentially displayed, so that a two-dimensional image in the xz plane is displayed on the front panel 402. Similarly, the same projection mechanism is arranged at the upper part of the left eye, and a left-eye image is displayed in front of the left eye of the front panel 502. In this way, the display unit is formed by the right-eye projection mechanism and the left-eye projection mechanism. Note that the left-eye projection mechanism and the right-eye projection mechanism may be shared.

As shown in FIG. 67, the virtual reality presentation device 500 may be a non-transparent HMD device. In this case, an image obtained by synthesizing a video for virtual reality presentation with an image captured by the line-of-sight camera 4011 is displayed on the display panel 146.

In this case, at the display position of the virtual reality object, two videos are combined and displayed on the display panel with the video captured by the line-of-sight camera as the lower layer and the video of the virtual reality object as the upper layer.

For example, at predetermined coordinates (x, y) at which the virtual reality object is displayed, the pixel data of the video of the virtual reality object generated by the video generation processing unit 1030 is (Y1, U1, V1). It is assumed that the pixel data of the video imaged by the visual line camera 4011 is (Y2, U2, V2). When the arrangement position of the virtual reality object is closer to the user than the object captured in the video captured by the line-of-sight camera 4011, the pixel data represented by the coordinates on the display panels 146a and 146b is (Y1, U1, V1). On the other hand, when the object shown in the video imaged by the line-of-sight camera 4011 is closer to the user than the arrangement position of the virtual reality object, the pixel data represented by the coordinates on the display panels 146a and 146b is (Y2, U2 , V2).

As described above, in the virtual reality presentation system according to the fourteenth embodiment, a virtual reality service providing server having a high processing capability is in charge of a part of the processing that occurs along with the presentation of virtual reality, so that more advanced virtual It is possible to present the real world.

Note that the above-described sharing is an example, and the virtual reality presentation device 500 may include some functions on the virtual reality service providing system 5000 side. For example, since the human auditory sense does not have the positional resolution as much as the visual sense, audio data for virtual reality presentation is stored in advance on the virtual reality presentation device 500 side, and the audio data is read based on an instruction from the virtual reality service providing system 5000. Then, after performing distribution processing for distributing to a plurality of channels and amplification processing for adjusting the sound level based on the user state, sound is emitted from the speaker.

The layer order information described above may not be transmitted. The rendering processing unit 1031 generates video data for virtual reality presentation by layer in order according to the layer order, and is encoded by the video encoding processing unit 1032 and transmitted to the virtual reality presentation device 500. The video composition processing unit 144 can synthesize the virtual reality presentation videos for each layer in accordance with the order of transmission, without requiring layer information.

<Embodiment 15>
A human being is a living organism that senses events around itself by the five senses, that is, taste, touch, sight, hearing, and smell, and processes the sensed information in the brain to perform future actions. Therefore, by deceiving these human senses using the virtual reality presentation device and causing an illusion, the user can be guided from the real world to the virtual reality world.

However, the narrower the boundary between the artificially created virtual reality world and the real world, the more users have experienced before returning to the real world from the virtual reality world. You will not be able to recognize that the world is a virtual reality world.

For example, a virtual reality world that can interact with the idol X by placing the idol X that the user likes in the virtual space and controlling the behavior of the idol X in the virtual space while being linked to the real world is presented. Let's consider the case.

  The service provides an opportunity to contact the user by placing an idol X in the virtual reality world where the user cannot meet or speak in the real world. Therefore, the virtual reality service providing system basically controls the behavior of the idle X arranged in the virtual space so that the user is pleased. For example, the user can spend time like a dream by taking a behavior that the idol X in the virtual space confesses to the user.

As a matter of course, the idol is a virtual idol X arranged in the virtual space, and the idol X in the real world does not confess to this user. However, a user who enters the highly controlled virtual reality world does not doubt that the real idol X has confessed to him.

Later, in the real world, a friend said, “I was confessed by that idol X and I was going to get along.” In addition, there is a possibility that a situation may occur in which the user pushes against the actual office of the idol X who has confessed but never came to see him. When the user asked, “You told me you would go out at that time!”, The confession to the user was the idol X created in the virtual reality world, not the real idol X. It's clear that I don't know what you're talking about.

The virtual reality presentation device has the greatest significance in attracting users by creating an event in the virtual reality world that cannot be realized by physical or social constraints in the real world. Therefore, the feelings of users entering the virtual reality world created as a wonderful world than the real world are uplifting, and even after returning to the real world, the world that was perceived by the user is virtual There may be cases where you refuse to accept the fact that it was reality. In the above case, since the cruel reality gap that the best experience that Idol X that I had admired confessed was virtual was too large, even if a third party explains that, the user May reject psychologically.

In this way, in order to make the virtual reality world presented to the user even more wonderful, it is necessary to develop technology in a direction that eliminates the boundary with the real world, while the virtual reality world and the real It is said that the more the boundary with the world of the world disappears, the more the gap between the wonderful world of virtual reality and the real world cannot be psychologically accepted, and there is a possibility that it will hinder the life of the real world after that. There was a problem of antinomy.

In view of the above problems, the present embodiment is a virtual reality presentation system that ensures that the virtual reality world experienced in the past is accepted when the user cannot psychologically accept the fact that the virtual reality world is not the real world. And a virtual reality presentation method. Hereinafter, the present embodiment will be described in detail with reference to the drawings. In the following description, parts denoted by the same reference numerals have substantially the same functions. For the sake of clarification of the invention, explanations of overlapping parts are omitted as appropriate.

FIG. 70 is a conceptual diagram showing the relationship between persons and organizations related to the virtual reality service according to the fifteenth embodiment of the present invention. A virtual reality service use contract is made in advance between a virtual reality service provider that provides the virtual reality service and a user who uses (uses) the virtual reality service. When using a virtual reality service, a user concludes a virtual reality service usage contract with a virtual reality service provider, and pays a fee on a flat rate basis or a pay-as-you-go basis as a price associated with the use of the service.

Here, virtual reality (VR) refers to a world with reality created artificially by combining computer graphics (CG) and sound effects, and is a virtual reality service. Is a service that allows the user to experience such a virtual reality world. Humans sense events that occur around them using the five senses, and process the sensed information in the brain to grasp the situation. Therefore, a virtual reality world can be created by displaying images and outputting sounds so as to deceive human senses. Since the virtual reality world is a world that is artificially created by displaying video for virtual reality presentation or outputting audio for virtual reality presentation, there are various forms of virtual reality services.

FIG. 71 is an example of contractor information (user information) managed by the virtual reality service provider. The virtual reality service provider provides a contract that summarizes the personal information (contractor's personal information) of the user who has signed a virtual reality service use contract, the plan that the user has contracted, and the service content included in the plan. Information is stored and managed in a database. Further, the contractor information includes a user ID for uniquely identifying a user, a connection password used for user authentication, and the like.

The virtual reality service provider prepares a plurality of plans in advance, and the fee and the contents of the virtual reality service differ for each plan. For example, in FIG. 761, the user A makes a contract with a plan with the content “service that can talk to the idol X”.

The service content of this plan is not a service that says that you can talk to the idol X in the real world, but this idol X is created in the virtual reality world and created in the virtual reality world. In addition, it is a service that can talk to the idol X and the virtual reality world.

Naturally, it is difficult for the idol X in the real world to talk to the user in the real world due to time and physical constraints. However, there are many fans who want to experience the conversation with the idol X. Therefore, the virtual reality service provider creates a virtual reality world in which the idol X appears, and in the state where the user has a sense of reality in the virtual reality world, the idol X and the user in the virtual reality world Provide an opportunity to talk.

In order to provide such a service, it is necessary to create an idle X virtual reality object. The virtual reality object will be described later. Since the idol X has a portrait right, it is illegal for the virtual reality service provider to create and use the virtual reality object of the idol X without permission. Accordingly, the virtual reality service providing company concludes a person data use permission contract with the idol X directly or with the office to which the idol X belongs. After signing the contract, the virtual reality service provider uses its own or external virtual reality content production company to create data that constitutes the virtual reality object of the idle X, and stores it in the database in a usable state. Store it.

User B has a contract for a service content plan “Teacher Service in the Virtual Space World” provided by a virtual reality service provider. The service included in the plan is a service in which a virtual reality world in which a historical person appears is created, and a class is given to the user in a field in which the person left a feat in the historical person.

Because historical figures appearing in the virtual reality world take classes, users can take classes in a more uplifted state compared to teachers in the real world. It becomes possible to learn with higher learning efficiency.

In addition, the user C contracts with a service content plan provided by a virtual reality service provider, such as “create a virtual space world decorated with vases”. The service included in the plan is a service that creates a virtual reality world in which a vase is placed at a predetermined position when the user is at home. Users who do not have time to care for flowers, etc. receive the service, so users can see the scenery where the vase is placed on the shelf, so they do not have to buy flowers in the real world The experience of spending time in a colorful room is also possible.

Further, the user D is contracted with a service content plan such as “Keep pets” provided by the virtual reality service provider. The service included in the plan is a service that is said to create a virtual reality world where a pet is present when the user is at home. When a user who cannot actually keep a pet due to housing rules, care time, etc. receives this service, the user can see the pet moving around in the room, so the pet in the real world Even if you can't keep a pet, you can experience a space with pets. The pet is not a real pet, but a pet created in a virtual reality world. Therefore, the experience of keeping fictional animals such as lions, dinosaurs and other unicorns as pets becomes possible.

FIG. 72 is a block diagram illustrating a configuration of a virtual reality presentation system 1000 according to the fifteenth embodiment. The virtual reality presentation system 1000 includes a virtual reality presentation device 100, a virtual reality service providing server 200, a contractor information database 300, a virtual reality content storage unit 400, and a provided virtual reality related data storage database 500. . In the present specification, the virtual reality presentation device 100 will be described as a head mounted display (HMD) device that a user wears on the head. The virtual reality service providing server 200 is connected to a virtual reality presentation device 100 (100a to 100c) worn by a user via a communication network 10 such as the Internet.

The virtual reality presentation device 100 is a head mounted display device worn by a user on the head, and includes at least a display panel that displays a video of a virtual reality object, and a communication unit that communicates with the virtual reality service providing server 200. Prepare. The virtual reality presentation device 100 displays the video including the virtual reality object on the display panel based on the virtual reality presentation video data transmitted from the virtual reality service providing server 200, thereby allowing the user to see the virtual reality world. Present.

FIG. 73 shows the real world where the user wearing the virtual reality presentation device 100 is present, the virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100, and the virtual reality presented to the user. It shows the world, that is, the virtual reality world perceived by the user.

The user is actually standing in a room without a dog and looking at the front sofa. Here, a video of a dog that is a virtual reality object is displayed on the display panel of the virtual reality presentation device 100. The user can see a virtual reality world where a dog sitting on the sofa is a fusion of the scenery of the sofa, which is the real world, and the image of the dog displayed on the display panel.

This dog can only be seen by the user wearing the virtual reality presentation device 100. Thus, while others live in a real world without dogs, this user lives in a virtual reality world created with dogs.

Note that the virtual reality presentation device 100 has an audio reproduction function, and reproduces the dog's cry as the virtual reality object, thereby stimulating vision and hearing and presenting a virtual reality world to the user. Is preferable because it can enhance a sense of reality in the virtual reality world. In this case, in addition to the virtual reality presentation video data including the virtual reality object, virtual reality presentation audio data that is audio data for forming the virtual reality world is transmitted from the virtual reality service providing server 200.

The virtual reality service providing server 200 provides a virtual reality service by displaying a video of a virtual reality object on the display panel of the virtual reality presentation device 100. The virtual reality service providing server 200 may draw the virtual reality object to create virtual reality presentation video data, and transmit the virtual reality presentation video data to the virtual reality presentation device 100. The virtual reality object model data may be transmitted to the virtual reality presentation device 100, and the virtual reality presentation device 100 may draw the model data and display the video of the virtual reality object on the display panel.

The virtual reality content storage database 400 is a database that stores virtual reality content. Here, the virtual reality content is a general term for data used for creating a virtual reality world, and includes at least a virtual reality object (data) and a virtual reality object control program. FIG. 74 shows the structure of virtual reality content.

The virtual reality object refers to a virtual person or a building that is arranged in the virtual reality world. Virtual reality objects are classified into human objects, animal objects, non-animal objects, and other objects according to the types of objects arranged in the virtual reality world. Each virtual reality object is assigned an object ID (virtual reality object identification number) for uniquely identifying the virtual reality object.

A person object is an object such as a real person, an imaginary / imaginary person, a historical person, a creative person, etc., and is at least operable, and preferably speaks (outputs sound). It has the feature that it is possible.

The animal object is an object such as a real animal, an imaginary / fantasy animal, or a creative animal, and has a feature that it can operate. In addition, a part of the animal object can be configured to speak (output a sound). Since it is a virtual animal placed in the virtual reality world, it is possible to adopt a configuration that speaks like a human.

Non-animal objects are objects such as plants, furniture, buildings, and the like that basically do not move. However, the plant may be configured to partially move in conjunction with the wind blowing in the real world.

The other objects are, for example, video effects, sound effects, and the like, which are objects for giving reality to the virtual reality world. Other objects may be used alone to create a virtual reality world, or may be used in combination with a person object, an animal object, or the like.

Various virtual reality objects are composed of model data, basic audio data, and the like. The model data includes polygon data that defines the appearance shape of the virtual reality object, texture data that is pasted on the polygon surface, and the like.

FIG. 75 is an example of a management file for managing virtual reality objects. Each virtual reality object is assigned an identification number that uniquely identifies it, and is classified according to a plurality of types. For example, type 1 indicates a rough division of the virtual reality object, type 2 indicates whether it is a real thing (person) or an imaginary thing (person), and type 3 is included in the object This shows the gender of the person. For example, video data of a sleeping female lion is stored in the storage address xx010110101 of the management number No. 000012.

FIG. 76 shows an example of model data of an animal object having an object ID number: No000009 which is a kind of virtual reality object. The model video data includes 3D polygon data that defines the appearance shape of the virtual reality object, and texture data that is pasted on the surface of the shape formed by the polygon data. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint points are connected by a skeleton, and each joint point is connected to several polygon vertices. The position coordinates and angles of the joint points can be controlled, and the appearance shape of the virtual reality object changes by changing the joint points.

The sound basic data constituting the virtual reality object is data serving as the basis of the sound emitted by the virtual reality object. For example, when the virtual reality object is a person object created based on a real person, the data is voice quality, pitch, voice waveform, phoneme, etc. extracted from the real voice of the real person. These voice basic data are synthesized by voice synthesis processing and presented to the user, so that the user senses that the person object is actually speaking. Note that the basic voice data extraction method (recording method) can be extracted at various angles, and may be extracted in the form of voice pronunciation data and voice expression data, or other classification methods. It may be extracted with. The extraction method is determined by a speech synthesis algorithm to be performed.

The virtual reality object control program included in the virtual reality content is a program related to the above-described control of the virtual reality object, and is classified into a plurality of levels of programs as shown in FIG.

The main control program controls the overall flow of the virtual reality service. The virtual reality service is provided by reading out and executing the main control program corresponding to the user's contract contents when the provision of the virtual reality service is started, corresponding to the service contents of the virtual reality service shown in FIG. To start.

The sub-control program is appropriately read from the main control program and controls selection, arrangement position, arrangement configuration, and the like of the virtual reality object arranged in the virtual reality world.

The object behavior control program controls the behavior of the objects read by the sub-control program and arranged in the virtual reality world. For example, when the object is a person object, an object behavior control program corresponding to each operation such as raising an arm, changing a facial expression such as laughing or crying, or jumping is prepared. The sub-control program selects an appropriate object behavior control program according to the situation of the virtual reality world, and when the object behavior control program is executed, the virtual reality object placed in the virtual reality world becomes The operation corresponding to the selected object behavior control program is performed.

As an example, it is assumed that the user has made a contract of a plan called “Idle X and 1 hour talk course”. When the virtual reality service providing server 200 determines to start providing the virtual reality service to the user, the virtual reality service providing server 200 reads the main control program corresponding to the plan with which the user has a contract from the virtual reality content storage database 400 and executes it. To do. Thus, the main control program is a program that controls the overall flow of the virtual reality service to be provided.

A plurality of related sub-control programs are read out by the main control program, and a virtual reality object is selected by executing a sub-control program related to virtual reality object selection. Here, the virtual reality object of idle X is selected and read from the virtual reality content storage database 400.

The sub-control program determines an initial arrangement position and an arrangement angle in the virtual space of the idle X that is a virtual reality object read from the virtual reality content storage database 400, and arranges the idle X in the virtual space. To do. The virtual reality service providing server 200 draws an idle X that is a virtual reality object arranged in the virtual space by the sub-control program, generates two-dimensional video data, performs a predetermined encoding process, and then performs virtual processing. It transmits to the reality presentation device 100. The virtual reality presentation device 100 decodes the encoded video data and displays the video on the display panel, whereby the video of idle X is displayed on the display panel.

The sub control program appropriately reads and executes an object behavior control program for controlling the behavior of the virtual reality object based on the state of the user wearing the virtual reality presentation device 100 and the like. By executing the read object behavior control program, the virtual reality object operates, and a video of the virtual reality object whose operation is controlled by the object behavior control program is displayed on the display panel of the virtual reality presentation device 100. Is displayed.

In the sub control program, a program related to the personality of the virtual reality object is provided. The sub-control program related to the personality may be particularly referred to as a personality control program. Different personality control programs are read based on feedback from the virtual reality presentation device 100, the virtual reality object behavior control program is selected according to the personality control program, and the selected virtual reality object behavior control program is executed. As a result, the virtual reality object operates.

For example, as the personality control program for the virtual reality object of the above-mentioned idle X, “normal state 1 to 10”, “anger state 1 to 5”, “joy state 1 to 5”, “sad state 1 to 5”, “fun” It is assumed that 33 types of personality control programs of “states 1 to 5” and “tsundere states 1 to 3” are stored in the virtual reality content storage database 400 as virtual reality content.

It is assumed that the personality control program of “normal state 1” is selected as the personality control program for controlling the personality of the idol X that is the current virtual reality object. In this state, when the user calls “X, good morning!” Via the virtual reality presentation device 100, the personality control program reads out and executes the corresponding virtual reality object behavior program, so that the virtual reality world is read. The idol X in the laughing smiles and utters a voice saying "Good morning."

On the other hand, when the personality control program in the tsundere state 2 is selected, when the user calls “X, good morning!” Via the virtual reality presentation device 100, the personality control program in the tsundere state 2 A different virtual reality object behavior control program is read out, and shows a behavior that emits a voice saying “Oh, I don't greet you!

Thus, the response to the input is determined by the personality control program, and the behavior of the virtual reality object is controlled by the virtual reality object behavior control program. The user wearing the virtual reality presentation device 100 is presented with video and audio of the virtual reality object for which these controls are performed.

The presented virtual reality related data storage database 500 stores data relating to the construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user by the virtual reality presentation device 100 can be reproduced. . More specifically, the presented virtual reality related data storage database 500 stores data related to the video in a state where the video of the virtual reality object displayed on the display panel of the virtual reality presentation device 100 can be reproduced. . The presented virtual reality-related data storage database 500 stores the video displayed on the display panel as data related to the video. Data encoded by a moving image encoding method such as H.264 may be stored. The presented virtual reality related data storage database 500 may store the execution history of the virtual reality object control program as data related to the video. This is because the virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100 can be reproduced by executing the virtual reality object control program again according to the execution history.

As described above, the virtual reality presentation system 1000 according to the fifteenth embodiment is used to construct the virtual reality presentation apparatus 100 that presents the virtual reality world related to the virtual reality service to the user and the virtual reality world. A virtual reality service providing server 200 that generates at least virtual reality presentation video data and transmits it to the virtual reality presentation device 100, and the virtual reality world that can be reproduced by the virtual reality presentation device 100 in a state where the virtual reality world can be reproduced. It includes at least a presented virtual reality related data storage database 500 that stores data relating to the construction of the real world as virtual reality related data.

With this configuration, users who have returned to the real world after experiencing the virtual reality world cannot recognize the boundary between the real world and the virtual reality world, When it is not possible to recognize that it is not the real world but the virtual reality world, the user can be appropriately returned to the real world by reproducing the virtual reality world experienced by the user again.

More specifically, in the fifteenth embodiment, the virtual reality presentation device 100 takes the form of a head-mounted display device that the user wears on the head. The head-mounted display device includes at least a display panel that displays an image, and allows the user to be drawn into the virtual reality world by deceiving the user's vision. The virtual reality service providing system according to the fifteenth embodiment is displayed on a virtual reality service providing server 200 that displays a video of a virtual reality object on the display panel of the head mounted display device 100 and the display panel of the head mounted display device 100. And a provided virtual reality related data storage database 500 that stores data related to the video in a state where the video of the virtual reality object can be reproduced.

According to this configuration, after experiencing virtual reality, even if there is an error in the boundary between virtual reality and reality, the virtual reality that the user was viewing at that time based on the related data stored in the database By explicitly presenting the object video to the user again, it is possible to accept the fact that the world believed to be real was the world of virtual reality.

It is more preferable that the virtual reality presentation system 1000 includes a history information storage server 600 and a virtual reality reproduction processing server 700 as shown in FIG.

The history information storage server 600 stores the virtual reality service used by the user, that is, the history of the virtual reality service provided to the user, as virtual reality service history information. Here, the virtual reality service history information stored in the history information storage server 600 is a so-called log.

The virtual reality content storage database 400 stores virtual reality content related to a plurality of types of virtual reality services, and the virtual reality service providing server 200 stores various data related to virtual reality services provided from the virtual reality content storage database 400. Are read out and subjected to predetermined processing to sequentially generate video data and audio data for virtual reality presentation. The history information storage server 600 summarizes the correspondence between the identification information (user ID) for identifying the user and the virtual reality service identification information for identifying the virtual reality service provided to the user by the virtual reality service providing server 200 in time series. The provided virtual reality service history information is stored.

FIG. 79 shows an example of virtual reality service history information stored in the history information storage server 600. The virtual reality service history information includes user identification information for identifying a user who has been provided with the virtual reality service, virtual reality service identification information for identifying a virtual reality service provided to the user, and the virtual reality service. The date and time information indicating the date and time is grouped in a time series in a state of being associated with each other.

In the virtual reality service providing server 200, user identification information for identifying a user who provides a virtual reality picture service and virtual reality service identification information for identifying a virtual reality service provided to the user are associated with each other in time series. A virtual reality service history information update processing unit for generating and updating the recorded virtual reality service history information is provided.

Only the virtual reality service history information stored in the history information storage server 600 cannot reproduce the virtual reality world presented to the user. This is because there are a plurality of virtual reality objects related to the virtual reality service, and it is impossible to specify which type of virtual reality object is used to build the virtual reality world.

The virtual reality reproduction processing server 700 receives a history disclosure request from the normal personal computer 20 or the virtual reality presentation device 100 via the communication network 10 such as the Internet. The history disclosure request is a packet for requesting disclosure of a virtual reality service log (usage history) used so far by the user, and includes at least a user ID and a password for authentication.

When the history disclosure request is received, the virtual reality reproduction processing server 700 matches the user ID and password pair stored in the contractor information storage database 300 with the user ID and password pair included in the history disclosure request. Perform authentication processing to see if you are doing it.

When the authentication is successful, the virtual reality reproduction processing server 700 reads the virtual reality service history information from the history information storage server 600 and returns it to the transmission source terminal of the history disclosure request. When the presented virtual reality reproduction request including information designating the provided virtual reality service related to the virtual reality world requesting the reproduction is received after the reply, the virtual current reproduction processing server 700 is designated by the request. The virtual reality related data related to the provided virtual reality service is read from the presented virtual reality related data storage database 500, and the virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100 is reproduced to obtain the request source terminal. Send to.

Since the user can directly check the video that he / she saw at that time through a terminal such as the computer 20, he / she has to recognize that the world he / she has experienced is a virtual reality world. There will be no. Therefore, it is possible to recognize that it was a virtual reality more reliably than a case where a third party simply persuades the user.

Among virtual reality services, there is a type of virtual reality service that has story characteristics and allows one virtual reality service to be experienced multiple times. Therefore, it may not be sufficient to specify the virtual reality service that has been provided in order to specify the virtual reality world that requires reproduction. Must be specified.

Therefore, it is further preferable that the history information storage server 600 is configured to store virtual reality history information in which the virtual reality world presented to the user is summarized in time series. In this case, the virtual reality service providing server 200 includes a virtual reality identification number allocation unit (virtual reality identification information allocation unit) that allocates a virtual reality identification number (virtual reality identification information) that identifies the virtual reality world presented to the user. To do.

When providing a virtual reality service to a user, the virtual reality service providing server 200 assigns a virtual reality identification number that uniquely identifies the virtual reality world to the virtual reality world related to the virtual reality service. That is, each virtual reality service that can be provided by the virtual reality service providing server 200 is identified by the virtual reality service identification information uniquely assigned to the virtual reality service, and is presented to the user when the virtual reality service is provided. For each virtual reality world, a virtual reality identification number for identifying the virtual reality world is assigned by the virtual reality identification number assigning unit.

Each virtual reality service can be provided to each user who wants to use the virtual reality service, and the same virtual reality service can be used multiple times. On the other hand, even if they are the same virtual reality service, the virtual reality world related to the virtual reality service is all different. For example, when using the virtual reality service “Let's talk with idol X”, the appearance of idol X, which is a virtual reality object, is displayed in front of the user. As a result, the reaction from the idol X changes, so it may become a virtual reality world that has been associated with the confession to the idol X, or it may become a virtual reality world that quarrels with the idol X. Because.

Therefore, when it is determined that the provision of the virtual reality service to the user is started based on a virtual reality service use request from the user, the virtual reality identification number assigning unit creates the virtual reality created by the provided virtual reality service. A virtual reality identification number that uniquely identifies the virtual reality world is assigned to the world in advance.

The virtual reality related data storage database 500 stores various data used for construction of the virtual reality world as virtual reality related data, but stores the virtual reality related data in a state associated with the virtual reality identification number. Keep it. With this configuration, by designating the virtual reality identification number at a later date, the virtual reality related data associated with the virtual reality identification number can be read and the virtual reality world can be reproduced.

The history information storage server 600 stores virtual reality history information in which correspondences between identification information for identifying a user and virtual reality identification information assigned to the virtual reality world presented to the user are summarized in time series. FIG. 80 shows an example of virtual reality history information. The virtual reality history information includes a virtual reality identification number that identifies the virtual reality world presented to the user, a user ID that identifies the user who receives the presentation of the virtual reality world identified by the virtual reality identification number, A virtual reality service identification number for identifying a virtual reality service related to the virtual reality world, and chapter information indicating which chapter (chapter) of the virtual reality service the virtual reality world is related to; The position information and time information indicating the position and time (time zone) of the user who receives the presentation of the virtual reality world are collected.

In this case as well, the virtual reality world presented to the user cannot be reproduced only with the virtual reality history information stored in the history information storage server 600. In order to reproduce the virtual reality world for the user, the virtual reality identification number included in the virtual reality history information is designated and associated with the virtual reality identification number from the virtual reality related data storage database 500. It is necessary to read virtual reality related data and regenerate virtual reality presentation video and virtual reality presentation audio.

<Embodiment 16>
FIG. 81 is an external perspective view of the virtual reality presentation device 100 according to the sixteenth embodiment from obliquely forward and obliquely rearward. A virtual reality presentation device 100 that is a head-mounted display device includes at least a frame 101 and a front panel 102.

The frame 101 is a housing of the virtual reality presentation device 100, and an information processing processor, a RAM (Random Access Memory), a ROM (Read only memory), a battery, and the like are arranged therein. In addition, a dedicated video processor or the like that draws video as necessary is also arranged. The frame 101 is roughly divided into a right frame 101a, a left frame 101b, and a center frame 101c.

The light frames 101a and 101b are members connected to the left and right ears, respectively. Speakers 171a and 171b that convert an analog electric signal into air vibration and output sound are arranged inside one end, and the other end is a center frame. 101c.

In addition, peripheral microphones 132a and 132b that collect surrounding sounds are arranged outside one ends of the right frame 101a and the left frame 101b. In addition, a user microphone 131 that collects a voice uttered by the user is provided below the front end inside the light frame 101a. The sound collected by the user microphone 131 and the peripheral microphones 132a and 132b is converted into a digital signal by AD conversion and output to the information processing processor.

At the ends of the light frames 101a and 101b, inclination sensors 111a and 111b for detecting inclination and acceleration sensors 112a and 112b for detecting acceleration are arranged, respectively. In addition, peripheral cameras 122a and 122b that capture surrounding scenery are respectively arranged outside the right frame 101a and the left frame 101b.

Both ends of the center frame 101c are connected to the right frame 101a and the left frame 101b, respectively, and the front panel 102 is connected to the lower side of the center frame 101c.

Near the center and near both ends of the center frame 101c, tilt sensors 111c, 111d, and 111e that detect tilt and acceleration sensors 112c, 112d, and 112e that detect acceleration are arranged, respectively. In addition, line-of-sight cameras 121a and 122b that capture a scene corresponding to the user's field of view are disposed on the front side of the center frame 101c and on the left and right eyes of the user. The line-of-sight cameras 121a and 121b are disposed on the front side of the center frame 101c at positions separated by a distance corresponding to the distance between the left and right eyes of the user.

The front panel 102 is disposed at a position in front of the user's eyes with the virtual reality presentation device 100 mounted on the head. In the virtual reality presentation device 100 according to the sixteenth embodiment, the front panel 102 is a transmissive display, and is a light display panel (first video display unit) that is a display area that displays images in front of the right eye and in front of the left eye. 151a and a left display panel (second video display unit) 151b are arranged.

The right display panel 151a and the left display panel 151b have a structure in which a light guide plate and a liquid crystal panel (LCD) are overlapped. The liquid crystal panel has a structure in which a transparent electrode (transparent conductive film), a polarizing plate (polarizing film), a liquid crystal layer, a glass substrate, a color filter, and the like are laminated.

82 shows a cross-sectional view of the vicinity of the light display panel 151a of the virtual reality presentation device 100. FIG. A front panel 102 is disposed below the center frame 101c, and a light source 152 is disposed below the center frame 101c.

Visible light emitted from the light source 152 is irradiated to the front panel 102 side. The front panel 102 has a configuration in which the light guide plate 153 and the LCD 154 are combined. The visible light emitted from the light source 152 is reflected to the user side by the light guide plate 153 and is transmitted through the display-controlled LCD 154 to display an image. Is done.

FIG. 83 is a block diagram illustrating a functional configuration of the virtual reality presentation system 2000 according to the sixteenth embodiment. The virtual reality presentation system 2000 includes a virtual reality presentation device 100, a virtual reality service providing server 200, a contractor information storage DB 300, a virtual reality content storage DB 400, and a presented virtual reality related data storage database 500. The virtual reality presentation device 100 and the virtual reality service providing server 200 are connected to each other via a communication network such as the Internet.

The virtual reality presentation device 100 includes a user state detection unit 110, an ambient condition detection unit 120, a voice input unit 130, a display control unit 140, a display unit 150, a voice reproduction control unit 160, and a voice output unit 170. The virtual reality presentation control unit 180 and the communication unit 190 are provided.

The user state detection unit 110 detects the state of a user who is using the virtual reality presentation device 100 that is a virtual reality presentation device. As the user state detection unit 110, the virtual reality presentation device 100 includes an inclination sensor 111 that detects inclination and an acceleration sensor 112 that detects acceleration. The inclination information generated by the inclination sensor 111 and the acceleration information generated by the acceleration sensor 112 are transmitted from the communication unit 190 to the virtual reality service providing server 200 as user state information indicating the state of the user.

The surrounding state detection unit 120 detects the surrounding state of the user who is using the virtual reality presentation device 100. Specifically, the surrounding situation detection unit 120 detects the surrounding situation by acquiring an image of the external environment. As the surrounding state detection unit 120, the virtual reality presentation device 100 includes a line-of-sight camera 121 that captures a scene corresponding to the user's line of sight, and a peripheral camera 122 that captures a range of scenes that cannot be captured by the line-of-sight camera 121, such as the user's sides or the back. And comprising. The video of the scenery in front of the user captured by the line-of-sight camera 121 and the video of the surroundings of the user captured by the peripheral camera 122 are each appropriately encoded such as compression, and then transmitted from the communication unit 190 as the surrounding state information. It is transmitted to the virtual reality service providing server 200.

The voice input unit 130 collects voice, converts it into an analog electric signal, performs AD conversion processing, and converts it into a voice digital signal, thereby inputting the voice. As the voice input unit 130, the virtual reality presentation device 100 includes a user microphone 131 that collects voices uttered by a user and a peripheral microphone 132 that collects peripheral voices.

The user microphone 131 is arranged inside the virtual reality presentation device 100 (user side) so that the user's voice can be collected efficiently, while the peripheral microphone 132 efficiently collects the voice from the user's external environment. It is arranged outside the virtual reality presentation device 100 so that it can. The peripheral microphone 132 is preferably arranged in the vicinity of the position of the ear in the virtual reality presentation device 100 so that the user can collect sound that can be heard when the virtual reality presentation device 100 is not worn. The user microphone 131 and the peripheral microphone 132 collect sound and convert it into an analog electric signal, and perform AD conversion processing to convert it into a sound digital signal, thereby inputting the sound. The input speech is subjected to speech encoding as necessary, and then transmitted to the virtual reality service providing server 200 from the communication unit 190 as user speech information and ambient speech information, respectively.

The display control unit 140 inputs virtual reality presentation video data generated by the virtual reality service providing server 200 among various data received by the communication unit 190, and performs virtual processing based on the input virtual reality presentation video data. Control is performed to display the video of the real object on the display unit 150 as a virtual reality presentation video.

The display unit 150 displays the virtual reality presentation video based on the control from the display control unit 140. The virtual reality presentation video displayed on the display unit 150 is a video such as a still image or a moving image of a virtual reality object for making the user feel virtual reality. As the display unit 150, the virtual reality presentation device 100 is provided with a right display panel 151a and a left display panel 151b, and each displays a virtual reality presentation video with parallax added thereto.

The audio reproduction control unit 160 inputs virtual reality presentation audio data generated by the virtual reality service providing server 200 among various data received by the communication unit 190, and based on the input virtual reality presentation audio data. Then, the audio output unit 170 performs control to output audio generated by the virtual reality object, background music for constructing the virtual reality world, and the like.

  Here, in order to make the user recognize the sound caused by the virtual phenomenon as the sound caused by the real phenomenon, the user is recognized as if the sound is transmitted from the position of the virtual reality object that is the sound generation source. This is preferable in attracting users to the virtual reality world. Therefore, it is preferable that the audio reproduction control unit 160 reproduces the audio data in a form corresponding to the three-dimensional sound such as 5.1Ch, 6.1Ch, or 7.1Ch.

  The audio output unit 170 outputs audio according to the reproduction control from the audio reproduction control unit 160. As the audio output unit 170, the virtual reality presentation device 100 includes speakers 171 a and 171 b arranged near the rear ends of the right frame 101 a and the left frame 101 b, and DA is added to the audio data reproduced by the audio reproduction control unit 160. Conversion is performed to convert it into an analog electric signal, which is further converted into air vibration and emitted to the outside.

The virtual reality presentation control unit 180 controls processing for presenting virtual reality to the user by the virtual reality presentation device 100. The virtual reality presentation control unit 180 starts control for presenting virtual reality when the communication unit 190 receives virtual reality service provision start notification information instructing the start of the virtual reality service.

Specifically, based on the virtual reality service provision start notification information, the virtual reality presentation control unit 180 provides user status information, ambient status information, and voice status information to the user status detection unit 110, the ambient status detection unit 120, and the voice input unit 130, respectively. Instruct to start acquisition of audio information and instruct the communication unit 190 to periodically transmit the information to the virtual reality service providing server 200. Further, the virtual reality presentation control unit 180 instructs the display control unit 140 to start control of displaying video on the display unit 150, and starts playback control of audio output from the audio output unit 170 to the audio playback control unit 160. Instruct.

  In accordance with control from the virtual reality presentation control unit 180, the communication unit 190 includes user state information indicating the user state detected by the user state detection unit 110, and surroundings indicating the user's surroundings detected by the surrounding state detection unit 120 The situation information and the voice information indicating the voice input by the voice input unit 130 are transmitted to the virtual reality service providing server 200, respectively. Further, the communication unit 190 outputs the virtual reality presentation video data and the virtual reality presentation audio data received from the virtual reality service providing server 200 to the display control unit 140 and the audio reproduction processing unit 160, respectively.

Next, the virtual reality service providing server 200 will be described. The virtual reality service providing server 200 includes a communication unit 210, an audio analysis processing unit 220, an image analysis processing unit 230, a virtual reality service provision control unit 240, a virtual reality object behavior control unit 250, and a display video generation processing unit. 260 and a reproduction sound generation processing unit 270.

The communication unit 210 communicates with the virtual reality presentation device 100 worn by each user via a communication network such as the Internet. The communication unit 210 receives, from the virtual reality presentation device 100, user status information indicating the user status, ambient status information indicating the user's surroundings, voice information generated by the user and voice information indicating the surrounding sounds. The communication unit 210 outputs audio information to the audio analysis processing unit 220, outputs ambient state information to the image analysis processing unit 230, and outputs user state information to the display video reproduction processing unit 260 and the reproduction audio generation processing unit 270, respectively. To do.

  The voice analysis processing unit 220 receives the voice information received by the communication unit 210 and analyzes the voice included in the voice information to extract the context and emotional elements of the voice uttered by the user. The voice analysis processing unit 220 outputs information about the extracted voice context and emotional elements to the virtual reality service provision control unit 240 and the virtual reality object behavior control unit 250, respectively. In addition, the voice analysis processing unit 220 specifies the voice level of the surrounding voice and the voice level of the user's voice input by the voice input unit 130 of the virtual reality presentation device 100, and outputs them to the playback voice generation processing unit 270.

The image analysis processing unit 230 receives the ambient situation information received by the communication unit 210 and analyzes an image (video) included in the ambient situation information to thereby determine the positional relationship of objects around the user and the brightness of the surroundings. Extract information such as size. The image analysis processing unit 230 outputs information such as the positional relationship and brightness of the extracted objects to the virtual reality object behavior control unit 250 and the display video generation processing unit 260, respectively.

The virtual reality service provision control unit 240 performs processing for starting provision of the virtual reality service and processing for ending. The virtual reality service provision control unit 240 performs scheduling related to the virtual reality service in consideration of the resources of the server 200 and the like, and refers to the contractor information stored in the contractor information storage database 300 to the user. Decide whether to start providing virtual reality services. Further, based on the plan contracted by the user indicated by the contractor information, it is determined whether or not to end the currently provided virtual reality service.

The virtual reality service provision control unit 240 instructs the virtual reality presentation device 100 to start the virtual reality service via the communication unit 210 when determining whether to provide the virtual reality service to the user. Based on the voice information and surrounding situation information received from the virtual reality presentation device 100 based on the instruction, it is determined whether the user can enjoy the virtual reality service, and the virtual reality service is provided based on the determination result. Decide to start.

When the virtual reality service provision control unit 240 determines the start of provision of the virtual reality service, the virtual reality service provision control unit 240 reads the virtual reality content related to the virtual reality service to be provided from the virtual reality content storage database 400, and places the virtual reality object in the virtual space. Deploy.

The virtual space is a virtual space represented by, for example, XYZ orthogonal coordinates or RθZ polar coordinates with the user's position as the origin, and the positional relationship of surrounding objects grasped by the analysis processing in the image analysis processing unit 230. In addition, the virtual reality object is positioned.

  The virtual reality object behavior control unit 250 controls the behavior of the virtual reality object based on the voice context and emotion information extracted by the voice analysis processing unit 220 and the movement of the object extracted by the image analysis processing unit 230. To do. Specifically, control for deforming the shape of the virtual reality object is performed, or sound emitted from the virtual reality object is controlled.

Specifically, the virtual reality object behavior control unit 250 creates a behavior control program for controlling the behavior of the virtual reality object arranged in the virtual space based on the analysis results in the voice analysis processing unit 220 and the image analysis processing unit 230. Then, the virtual reality content storage database 400 is selected, and the selected behavior control program is executed to sequentially change the shape of the virtual reality object.

The display video generation processing unit 260 generates a video displayed on the display unit 150 of the virtual reality presentation device 100. Specifically, the display video generation processing unit 260 determines a display position on the display panel of the virtual reality object controlled by the virtual reality object behavior control unit 250 based on the user state information, and sets the display position to the determined display position. A video in which a virtual reality object is displayed is generated. The display video generation processing unit 260 also determines the brightness of the virtual reality object based on the brightness around the user extracted by the image analysis processing unit 230 or the brightness (illuminance) at the position where the virtual reality object is placed. An optical correction process for correcting (illuminance) is performed, and an image for displaying the virtual reality object after the correction process at the determined display position is generated. The display video generation processing unit 260 converts the generated video into MPEG2 or H.264. The data is output to the communication unit 210 after being encoded by a predetermined encoding method such as H.264 / AVC.

The reproduction sound generation processing unit 270 generates sound output from the sound output unit 170 of the virtual reality presentation device 100. The reproduction sound generation processing unit 270 generates sound output from the virtual reality object controlled by the virtual reality object behavior control unit 250 and output from each channel based on the positional relationship between the user and the virtual reality object. Further, if the playback sound generation processing unit 270 is configured to perform a sound level correction process for correcting the sound level of the sound to be generated in accordance with the surrounding sound level obtained by the sound analysis processing unit 220, the reality is further increased. preferable. The reproduction sound generation processing unit 270 performs the sound level correction processing on the sound emitted from the virtual reality object controlled by the virtual reality object behavior control unit 250, and outputs the sound from the sound output unit 170 of the virtual reality presentation device 100. Generate audio. The reproduction sound generation processing unit 270 encodes the generated sound using a predetermined encoding method such as MPEG3 or CELP, and then outputs the encoded sound to the communication unit 210.

The communication unit 210 inputs various control information (notification information) instructing presentation of virtual reality in the virtual reality presentation device 100 from the virtual reality service provision control unit 240, and transmits it to the virtual reality presentation device 100. In addition, the communication unit 210 transmits the virtual reality presentation video data and audio data respectively input from the display video generation processing unit 260 and the reproduction audio generation processing unit 270 to the virtual reality presentation device 100. In addition, the communication unit 210 outputs the user state information, the surrounding state information, and the voice information received from the virtual reality presentation device 100 to each corresponding unit.

The presented virtual reality related data storage database 500 associates the video data generated by the display video generation processing unit 260 and the audio data generated by the reproduction audio generation processing unit 270 with the user related to the presentation of the virtual reality service. And remember. Since the generated video data and audio data are displayed or output to the user, the user senses the virtual reality world. By accessing and reading the data, it is possible to reproduce the virtual reality world experienced at that time.

<Embodiment 17>
FIG. 84 is a block diagram illustrating a configuration of a virtual reality presentation system 3000 according to the seventeenth embodiment. The virtual reality presentation system 3000 includes a virtual reality presentation device 100, a contractor information storage DB 300, a virtual reality content storage DB 400, a presented virtual reality related data storage DB 500, and a virtual reality service providing server 800.

  The virtual reality service providing server 800 includes a communication unit 210, an audio analysis processing unit 220, an image analysis processing unit 230, a virtual reality service provision control unit 240, a virtual reality object behavior control unit 250, and a display video generation processing unit. 260, sound generation processing unit 270, virtual space generation processing unit 810, virtual reality object arrangement processing unit 820, viewpoint control unit 830, light source control unit 840, model data arrangement processing unit 850, and layer order update A processing unit 860 and a layer information generation processing unit 870 are provided.

The communication unit 210 communicates with the virtual reality presentation device 100 via a communication network such as the Internet. The communication unit 210 includes at least a receiving unit that receives user status information indicating a user status, and a transmitting unit that transmits video data for virtual reality presentation.

The communication unit 210 receives user state information indicating a user's state, a surrounding scene image obtained by photographing the user's surroundings, user voice information that is a voice uttered by the user, ambient voice information that is a voice around the user, and the like. To do. In addition, the communication unit 210 receives various types of information used for virtual reality presentation control in the virtual reality presentation device 100 as necessary. Such information includes report information related to the remaining battery capacity, resource consumption, line speed, etc. of the virtual reality presentation device 100.

The user status information and the surrounding scenery image received by the communication unit 210 include user status detection time information (user status detection timing information) indicating the timing when the user status is detected, and the timing when the surrounding scenery image is captured. A surrounding scene image photographing time information (ambient state detection timing information) is included.

The communication unit 210 transmits video encoded data for virtual reality presentation, audio encoded data for virtual reality presentation, layer information, various types of notification information, and the like.

The voice analysis processing unit 220 performs voice analysis processing on the user voice information received by the communication unit 210, thereby acquiring the emotion information indicating the context of the voice uttered by the user and emotion elements. Moreover, the voice analysis process is performed on the surrounding voice information received by the communication unit 210 as necessary, and the situation around the user is specified.

The image analysis processing unit 230 extracts predetermined information by performing an image analysis process on the surrounding scenery image obtained by photographing the scenery around the user received by the communication unit 210. The surrounding scene image is an image taken by a plurality of line-of-sight cameras 121 and peripheral cameras 122 included in the virtual reality presentation device 100.

For example, the image analysis processing unit 230 extracts an optical component from the surrounding scenery image, and extracts information such as the average illuminance around the user and the position and direction of the light source as optical component information.

In addition, the image analysis processing unit 230 compares each object position included in the surrounding scenery image by comparing the position of each object included in the plurality of surrounding scenery images and comparing the adjacent pixels of the same surrounding scenery image. Specify the position and shape. Based on the position and shape of these objects and the like that exist in the real world around the user, the image analysis processing unit 230 models the appearance shape of the objects and the like to generate model data.

Further, the image analysis processing unit 230 detects an instruction operation from the user based on the user's operation included in the series of surrounding scenery images. For example, as an operation indicating interruption of provision of the virtual reality service, a gesture such as raising the index finger and little finger in front of the line-of-sight camera 121 is assigned in advance. The image analysis processing unit 230 identifies the user's operation included in the surrounding scenery image received by the communication unit 210 by performing a matching process with a predetermined operation video, and indicates operation information indicating an instruction operation from the user To extract.

The virtual reality service provision control unit 240 controls provision of the virtual reality service. Specifically, the virtual reality service provision control unit 240 reads out and executes a control program related to the virtual reality service to be provided from the virtual reality content stored in the virtual reality content storage DB 400, thereby executing the virtual reality service. Start offering.

The virtual reality service provision control unit 240 requests that the voice context extracted by the voice analysis processing unit 220 or the operation information extracted by the image analysis processing unit 230 start provision of the virtual reality service. If there is, the virtual reality service provision start process is performed for the user.

Whether or not the virtual reality service provision control unit 240 can provide the requested virtual reality service to the user with reference to the contractor information of the user stored in the contractor information storage DB 300 That is, it is determined whether the user has a right to use the virtual reality service.

In the contractor information storage DB 300, a virtual reality service that can be used by each user is recorded as contractor information. Further, in the contractor information, a user ID for identifying each user, an authentication password, and other personal information of the user may be recorded together. The virtual reality service provision control unit 240 matches the combination of the user ID and password stored in the contractor information with the user ID and password indicated by the information extracted by the voice analysis processing unit 220 and the image analysis processing unit 230. It is even better if it is configured to determine whether or not the requested virtual reality service can be provided by performing authentication processing or the like.

As a result of the determination, if the requested virtual reality service can be provided, the virtual reality service provision control unit 240 generates virtual reality service provision start notification information for notifying the start of provision of the virtual reality service. . The communication unit 210 transmits the virtual reality service provision start notification information to the virtual reality presentation device 100.

FIG. 85 is a diagram illustrating an example of the virtual reality service provision start notification information. The virtual reality service provision start notification information includes the transmission source address that is the address of the server 800, the transmission destination address that is the address of the virtual reality presentation device 100 that is the transmission destination, and the information is the virtual reality service provision start notification information. And a number assigned to each packet in order to distinguish communication generated in relation to the virtual reality service between the virtual reality presentation device 100 and the virtual reality service providing server 800 from other communication. It includes a communication identification number, virtual reality service identification information for identifying a virtual reality service to be provided, virtual reality control parameter information used in control and processing necessary for provision of the virtual reality service, and the like.

As virtual reality control parameter information, detected user state specifying information for specifying what kind of user state is detected as the user state, and user state transmission indicating the transmission frequency of the user state information indicating the detected user state The frequency, the transmission frequency of the surrounding scenery image that is an image of the surrounding scenery of the photographed user, the surrounding scenery image encoding format indicating the image compression encoding format of the surrounding scenery image, and the video frame rate indicating the frame rate of the video to be displayed The virtual reality presentation video data encoding format indicating the encoding format of the virtual reality presentation video data appropriately transmitted from the virtual reality service providing server 800 is included. The virtual reality service provision control unit 240 sets these parameters based on report information such as the type of virtual reality service to be provided, the processing capability and resources of the virtual reality presentation device 100 worn by the user, and the communication line speed. The virtual reality service provision start notification information is generated using the determined parameter value as virtual reality control parameter information.

In addition, the virtual reality service provision control unit 240 controls the virtual reality service provision end notification information for notifying the end of provision of the currently provided virtual reality service or the control parameter change notification in the virtual reality service as necessary. Various information such as parameter change notification information is generated. These pieces of information generated by the virtual reality service provision control unit 240 are transmitted from the communication unit 210 to the virtual reality presentation device 100.

The function of the virtual reality identification number assigning unit described above may be set as one function of the virtual reality service provision control unit 240. Further, the communication identification number described above may be used as the virtual reality identification number. Since the virtual reality identification number is included in each packet transmitted and received in order to construct the virtual reality world related to the virtual reality service between the virtual reality presentation device 100 and the virtual reality service providing server 700, the description will be given later. The virtual reality related data acquisition unit 880 acquires each data used for construction of each virtual reality world for each virtual reality identification number as virtual reality related data, and stores it in the presented virtual reality related data storage DB 500.

The virtual reality service provision control unit 240 issues a virtual space generation instruction and a virtual reality object placement instruction to the virtual space generation processing unit 810 and the virtual reality object placement processing unit 820, respectively, when provision of the virtual reality service is started. .

In addition, the virtual reality service provision control unit 240 outputs a virtual reality presentation video data generation instruction and a virtual reality presentation audio data generation instruction to the display video generation processing unit 260 and the reproduction audio generation processing unit 270, respectively.

In addition, the virtual reality service provision control unit 240 instructs the virtual reality related data acquisition unit 880 to acquire data used to construct a virtual reality world related to the virtual reality service provided to the user.

The virtual reality content storage DB 400 stores virtual reality content that is data for realizing a virtual reality service. The virtual reality content includes a virtual reality object, a virtual reality service control program that controls the progress of the virtual reality service, a virtual reality object behavior control program that controls the behavior of the virtual reality object, and audio data for virtual reality presentation. It stores voice data for virtual reality presentation.

The virtual reality service provision control unit 240 performs various controls such as the start and end of the virtual reality service by executing the virtual reality service control program stored in the virtual reality content storage DB 400.

The virtual space generation processing unit 810 generates a virtual space used for the virtual reality service based on a virtual space generation instruction from the virtual reality service provision control unit 240. The virtual space is a coordinate space having a predetermined area, and is an information processing space created artificially so as to correspond to a space in the real world.

The virtual reality service provision control unit 240 issues a virtual space generation instruction corresponding to the virtual reality service to be started to the virtual space generation processing unit 810, and the virtual space generation processing unit 810 indicates the virtual space generation instruction. Generate a virtual space that is as large as possible.

Based on the virtual reality object placement instruction from the virtual reality service provision control unit 240, the virtual reality object placement processing unit 820 reads out the virtual reality object related to the virtual reality service from the virtual reality content storage DB 400, and generates a virtual space generation processing unit. 810 is arranged in the generated virtual space.

Here, when there are a plurality of virtual reality objects to be arranged, the virtual reality object arrangement processing unit 820 arranges the read virtual reality objects in virtual spaces of different layers.

That is, the virtual space generation processing unit 810 generates a virtual space of a plurality of layers having a common origin, and the virtual reality object placement processing unit 820 differs from the plurality of virtual reality objects read from the virtual reality content storage DB 400. Place in the virtual space of the layer.

FIG. 86 shows a case where the plurality of virtual reality objects are arranged in different layers of the virtual space. Three virtual reality objects, an elephant, a dog, and a lion, are arranged in three different layers of virtual space, each having a common origin.

The viewpoint control unit 830, based on the user state information received by the communication unit 210, in the virtual space generated by the virtual space generation processing unit 810, the viewpoint position coordinates and the line-of-sight direction corresponding to the user's eye position and line-of-sight direction And control. The viewpoint control unit 830 determines the viewpoint position coordinates corresponding to the position of the user's eyes and the line-of-sight direction from the change in the inclination of the user head and the change in the acceleration specified based on the inclination information and the acceleration information included in the user state information. And the line-of-sight direction are calculated and updated. Here, since the user state information includes user state detection time information that is a time stamp indicating the timing at which the corresponding user state is detected, the viewpoint control unit 830 determines the current viewpoint position coordinates and the line-of-sight direction. Manages the viewpoint position coordinates and the line-of-sight direction set based on the user state detected at which time.

The light source control unit 840 updates the position coordinates of the light source and the illuminance or luminance of the light source in the virtual space generated by the virtual space generation processing unit 810 based on the optical component information extracted by the image analysis processing unit 230. For example, the light source control unit 840 sets the illuminance or luminance of the light source at infinity based on the average luminance in the surrounding scene image obtained by the image analysis processing unit 230. Further, a local light source is newly arranged at a corresponding position in the virtual space based on the optical component information extracted based on the position of the shadow appearing in the surrounding scene image in the image analysis processing unit 230. In addition, the light source control unit 840 may set the color of the light source (color difference information) based on the optical component information extracted by the image analysis processing unit 230.

The model data arrangement processing unit 850 arranges model data such as an object existing in the real world around the user generated by the image analysis processing unit 230 in the virtual space. The model data arrangement processing unit 850 identifies from which position to which direction the surrounding landscape image is captured based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 830, and performs image analysis. The model data of each object generated by the processing unit 230 is arranged in the virtual space.

Here, it is preferable that the virtual space generation processing unit 810 generates a virtual space of a model data dedicated layer in which model data of an object existing in the real world is arranged. The model data arrangement processing unit 850 arranges the model data of the real world object generated by the image analysis processing unit 230 in the virtual space of the model data dedicated layer.

The virtual reality object behavior control unit 250 controls the behavior of the virtual reality object arranged in the virtual space. Some virtual reality objects arranged in each layer of the virtual space by the virtual reality object arrangement processing unit 820 are configured to be operable like a virtual reality object of a human being or an animal, for example. . The virtual reality object behavior control unit 250 reads and executes the behavior control program of the virtual reality object from the virtual reality content storage DB 400, thereby changing the appearance shape of the virtual reality object arranged in the virtual space. Or moving a virtual reality object in the virtual space.

The display video generation processing unit 260 generates a virtual reality presentation video. The display video generation processing unit 260 includes a drawing processing unit 261 and a video encoding processing unit 262.

The drawing processing unit 261 generates a virtual reality presentation video by drawing a virtual reality object arranged in the virtual space based on the virtual reality presentation video generation instruction from the virtual reality service provision control unit 240. . Here, the drawing processing unit 261 generates a virtual reality presentation video for each layer by performing drawing processing for each virtual reality layer.

The drawing processing unit 261 selects one of the layers in the virtual space and the model data dedicated layer, and in the virtual space of the selected layer based on the current viewpoint position coordinates and the line-of-sight direction set by the viewpoint control unit 830, respectively. Projection processing is performed for projecting the virtual reality object and the model of the object in the real world onto the image plane arranged forward in the line-of-sight direction from the viewpoint position coordinates. The drawing processing unit 261 deletes the model portion of the object in the real world from the two-dimensional projection image obtained by the projection processing, thereby obtaining the two-dimensional projection image of the virtual reality object arranged in the selected layer. get.

  The drawing processing unit 261 performs clipping processing that cuts the two-dimensional projection image obtained by the projection processing into a predetermined size. Here, the drawing processing unit 261 is characterized by cutting out in a range exceeding the resolution of the display panel of the virtual reality presentation device 100. For example, when the display panel of the virtual reality presentation device 100 has a resolution of 1080 pixels × 1920 pixels, the drawing processing unit 261 performs a clipping process of cutting the two-dimensional projection image by 1188 × 2112, which is 10% higher. It should be noted that how much the size is to be cut out may be determined based on the magnitude of change in the user state. This is because when the user is stationary, the subsequent correction process is not necessary, and it is not necessary to cut it out into large parts.

  Next, the drawing processing unit 261 performs a texture mapping process for pasting the texture mapping data of the virtual reality object to the two-dimensional projection image after the clipping process.

Next, the drawing processing unit 261 performs a shading process for applying a shadow to the two-dimensional projection image based on the light source set by the light source control unit 840. The drawing processing unit 261 performs anti-aliasing processing as necessary to obtain a virtual reality presentation video for the layer.

Here, the drawing processing unit 261 assigns information indicating the time at which the virtual reality presentation video of the layer generated by drawing is a video drawn based on the user state, and the virtual processing of the layer is performed. Reality-presenting video is generated. The viewpoint position coordinates and the line-of-sight direction used for the drawing process are updated and set by the viewpoint control unit 830. The time when the user state used when setting the currently used viewpoint position coordinates and the line-of-sight direction is detected is set. The user status detection time information, which is the time stamp shown, is given, and the virtual reality presentation video of the layer is generated.

The drawing processing unit 261 generates a virtual reality presentation video for each layer by drawing the virtual space of each layer generated by the virtual space generation processing unit 810 by parallel processing.

The video encoding processing unit 262 encodes the virtual reality presentation video of each layer generated by the drawing processing unit 261 to generate layered virtual reality presentation encoded video data. The layered virtual reality presentation encoded video data is transmitted from the communication unit 210 to the virtual reality presentation device 100.

The layer order update processing unit 860 updates the order of each layer in the virtual space generated by the virtual space generation processing unit 810. Since the rendering processing unit 261 performs rendering for each layer and generates virtual reality presentation videos for each layer, it is necessary to finally synthesize a plurality of virtual reality presentation videos for each layer in the virtual reality presentation device 100. is there. For this reason, the layer order update processing unit 860 sets the order of the layers in each virtual space, and updates the order of the layers.

The layer order is determined by the relationship between the position of the user and the position of the virtual reality object. That is, the layer order is determined in order from the layer where the virtual reality object closer to the user is arranged. The position of the user changes as appropriate, and the position of the virtual reality object also changes due to behavior control performed by the virtual reality object behavior control unit 250. Therefore, the layer order update processing unit 860 includes the viewpoint position coordinates (or user position coordinates) of the user controlled by the viewpoint control unit 830 based on the user state information received by the communication unit 210 and each layer in the virtual space. The layer order is updated based on the position coordinates of each virtual reality object arranged.

For example, in the case where virtual reality objects A to C are arranged in layers 1 to 3 of the virtual space as shown in FIG. 87, in state 1, the virtual reality objects A, B, and C are in order from the side closest to the user. . Therefore, the layer order update processing unit 860 assumes the order of layers 1, 2, and 3. Here, when the positional relationship of the state 2 is obtained due to the user moving in the real world, the virtual reality objects are in the order of B, C, and A from the side closer to the user. Therefore, the layer order update processing unit 860 manages the layer order by changing the layer order to layers 2, 3, and 1.

The layer information generation processing unit 870 generates layer information indicating the current layer order based on the update processing in the layer order update processing unit 860. The layer information generation processing unit 870 includes layer order information indicating the layer order, positional relationship information indicating the positional relationship between the user and each layer, the type and number of virtual reality objects included in each layer, and the importance of each layer. It is preferable to generate layer information including information such as degree. The positional relationship information is information indicating a distance and a vector from the user position to the virtual reality object included in each layer. The layer information is transmitted from the communication unit 210 to the virtual reality presentation device 100 together with the encoded video data for virtual reality presentation for each layer.

Based on the virtual reality presentation sound generation instruction from the virtual reality service provision control unit 240, the reproduction sound generation processing unit 270 reads the virtual reality presentation sound data related to the virtual reality service provided from the virtual reality content storage DB 400, Audio data for virtual reality presentation for each channel is generated.

For example, the playback audio generation processing unit 270 uses the virtual reality content storage DB 400 based on the relative relationship between the viewpoint position set by the viewpoint control unit 830 and the position of the virtual reality object arranged by the virtual reality object arrangement processing unit 820. The virtual reality presentation audio data is generated by distributing the read virtual reality presentation audio data to a plurality of channels.

Here, the reproduction sound generation processing unit 270 reads a plurality of virtual reality presentation sound data from the virtual reality content storage DB 400, distributes the sound data to the sound data of each channel, and synthesizes them, thereby synthesizing the virtual reality presentation sound for each channel. Generate data. Here, as a channel, it is preferable to provide a multi-channel audio output unit 530 such as 5.1Ch, 6.1Ch, 7.1Ch on the virtual reality presentation device 100 side so as to realize three-dimensional sound.

In addition, the reproduction sound generation processing unit 270 extracts the noise level of the user's surrounding environment from the surrounding sound information analyzed by the sound analysis processing unit 220, and reads the virtual reality presentation sound read based on the extracted noise level. It is even better if the data is amplified.

In addition, the reproduction sound generation processing unit 270 generates sound data for virtual reality presentation for each channel by encoding the generated sound data for virtual reality presentation for each channel in a predetermined encoding format. The channel-specific virtual reality presentation speech encoded data is transmitted from the communication unit 210 to the virtual reality presentation device 100.

Further, the reproduction voice generation processing unit 270 may generate the virtual reality presentation voice encoded data by generating the artificial voice by the voice synthesis process and encoding the voice data. In this case, the playback speech generation processing unit 270 includes a response speech context generation processing unit, a speech synthesis processing unit, and a speech encoding processing unit.

The response voice context generation processing unit is extracted by the user analysis information received by the communication unit 210, information about the voice context and emotion elements extracted by the voice analysis processing unit 220, and image analysis processing by the image analysis processing unit 230. Based on various kinds of information, a voice context of response voice generated by the virtual reality object currently arranged in the virtual space is generated. The speech synthesis processing unit joins the basic speech data (phoneme data) associated with the virtual reality object stored in the virtual reality content storage database 400 in accordance with the generated speech context, and then performs a predetermined filtering process. Is performed to generate response voice data. The response context generation processing unit further generates information related to the emotional element related to the response voice based on the input user state information, voice context, and emotional element information, and the voice synthesis processing unit It is further preferable that the synthesized voice data in which the pitch, the voice level, the conversation speed, and the like are generated by executing the filtering process with the filter coefficient based on the information related to the above-described information. The speech encoding processing unit uses the generated synthesized speech data as virtual reality presentation speech data, and generates speech encoding data for virtual reality presentation by performing speech encoding in an encoding format such as CELP. It is good also as the said structure.

The virtual reality related data acquisition unit 880 acquires various data used for realizing the virtual reality world presented to the user wearing the virtual reality presentation device 100 as virtual reality related data.

  For example, the virtual reality related data acquisition unit 880 monitors various types of information transmitted from the communication unit 210 to the virtual reality presentation device 100, and the virtual reality presentation video data and the virtual reality presentation audio data. Data transmitted to the reality presentation device 100 is acquired.

The virtual reality related data acquisition unit 880 stores the acquired virtual reality related data in the presented virtual reality related data storage DB 500. The presented virtual reality related data storage DB 500 stores and stores various types of information used to realize the virtual reality world presented to the user as presented virtual reality related data. For example, when the virtual reality service presented to the user is a virtual reality presentation service of “Idle X and 1 hour date”, the virtual reality related data storage DB 500 stores the video of the virtual reality object of the idle X and the idle Data such as sound emitted from the X virtual reality object is stored and stored as virtual reality related data. The presented virtual reality related data storage DB 500 stores and stores the virtual reality related data acquired by the virtual reality related data acquisition unit 880 as data used to reproduce the world of the virtual reality that has been presented.

The virtual reality presentation video data and the virtual reality presentation audio data acquired by the virtual reality related data acquisition unit 880 are data displayed and output by the virtual reality presentation device, respectively, for presenting the virtual reality world. Used. Therefore, by storing and storing these data, it is possible to reproduce the same virtual reality world by reading and reproducing these data from the presented virtual reality related data storage DB 500 at a later date. .

The virtual reality related data acquisition unit 880 stores the virtual reality related data in the database 500 in a state in which each virtual reality related data related to the acquired virtual reality world is associated with the virtual reality identification information.

The virtual reality presentation device 100 includes a user state detection unit 110, an ambient condition detection unit 120, a voice input unit 130, a display control unit 140, a display unit 150, a voice reproduction control unit 160, and a voice output unit 170. The virtual reality presentation control unit 180 and the communication unit 190 are provided.

The user state detection unit 110 includes an inclination sensor 111 that detects the inclination of the user's head and an acceleration sensor 112 that detects the acceleration of the user's head. The user state detection unit 110 includes an orientation sensor that detects the user's orientation, a position sensor that detects the user's position, a biological reaction sensor that detects biological reactions such as the user's body temperature, blood pressure, pulse, and brain waves, and the user. It may be configured with an internal camera that shoots the facial expression. Detection results of these sensors and the like are collected as user status information together with user status detection time information that is a time stamp indicating the detection time, and transmitted from the communication unit 190 to the virtual reality service providing server 800. A part of the user status information is used for video correction processing in the display control unit 140.

The ambient condition detection unit 120 includes a line-of-sight camera 121 that captures the front of the user corresponding to the user's line of sight, and a peripheral camera 122 that captures the user's periphery such as both sides and the back of the user. The line-of-sight camera 121 is arranged at the position of the user's left eye and right eye, respectively. Ambient scene images captured and acquired by these cameras are collected as ambient status information together with ambient status detection time information that is a time stamp indicating the shooting time, and transmitted from the communication unit 190 to the virtual reality service providing server 800. .

The voice input unit 130 is installed near the user's mouth, collects a voice uttered by the user and inputs user voice information, and collects a voice from an external environment around the user and collects the ambient sound. And a peripheral microphone 132 for inputting voice information. Each piece of audio information input by the audio input unit 130 is transmitted from the communication unit 190 to the virtual reality service providing server 800.

The communication unit 190 communicates with the virtual reality service providing server 800 via a communication network such as the Internet. The communication unit 190 wirelessly transmits user status information detected by the user status detection unit 110, ambient status information detected by the ambient status detection unit 120, and other audio information input by the audio input unit 130. And a wireless reception unit for wirelessly receiving video encoded data for virtual reality presentation and layer information, audio encoded data for virtual reality presentation, various notification information, and the like transmitted from the virtual reality service providing server 800.

The display control unit 140 includes a video decoding processing unit 141, a video correction processing unit 142, a video composition processing unit 143, and a display area selection unit 144.

The video decoding processing unit 141 performs a decoding process on the virtual reality presentation video encoded data for each layer received by the communication unit 190, and acquires virtual reality presentation video data for each layer.

The video correction processing unit 142 corrects the position of the decrypted virtual reality presentation video data for each layer, and the positional relationship included in the user status detected by the user status detection unit 110 and the layer information received by the communication unit 190 Based on information.

Each virtual reality presentation video data generated by the display video generation processing unit 260 is generated based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 830 based on the user state. Here, the user's viewpoint position and line-of-sight direction change during the communication lag between the virtual reality presentation device 100 and the virtual reality service providing server 800, and the time required for drawing processing, encoding processing, decoding processing, and the like. There is a possibility. For this reason, the video correction processing unit 142 performs a predetermined correction on the generated virtual reality presentation video, thereby correcting the video so as to be a video suitable for the current user state.

For example, as shown in FIG. 88, there are two virtual reality presentation images of layer 1 and layer 2, and relative position vectors (x, x) to the virtual reality object included in each layer indicated by the positional relationship information of each layer. y, z) are (+0.3, +2.2, −0.4) and (+2.1, +21.4, +0.6), respectively. Here, when the user is moving in the −X direction, the viewpoint position and the line-of-sight direction are different between when the video is generated and at the present time. Therefore, the video correction processing unit 142 determines the position of the virtual reality object included in the video based on the user status detected between the user status detection time information given to the virtual reality presentation video and the current time. Correct the size. Since how the video of the virtual reality object is corrected according to changes in the viewpoint position and the line-of-sight direction is determined by the relative positional relationship between the viewpoint position and line-of-sight direction and the virtual reality object, the video correction processing unit 142 Performs a correction process after calculating a correction amount for the virtual reality presentation video for each layer.

The video composition processing unit 143 performs a process of synthesizing the virtual reality presentation video of each layer after the correction processing by superimposing according to the layer order information included in the layer information. The video of the virtual reality object included in the rear layer is hidden by the virtual reality object included in the front layer.

The display area selection unit 144 selects an area to be displayed on the display panel 151 from the virtual reality presentation video after the synthesis process. Since each virtual reality presentation video is a video having a resolution higher than the resolution of the display panel 151 so as to be compatible with the correction processing, the virtual reality presentation video is displayed on the display panel 151 from the virtual reality presentation video after the correction processing and the synthesis processing. Select the range to display.

The display unit 150 displays a virtual reality presentation video according to the control from the display control unit 140. The display unit 150 is two display panels 151a and 151b that are specifically arranged in front of the left and right eyes of the user, and each virtual reality presentation selected by the display region selection unit 144 of the display control unit 140 Display the video. The viewpoint control unit 830 sets two viewpoint position coordinates and line-of-sight directions corresponding to the user's left eye and right eye, and the display image generation processing unit 260 draws from each of these two viewpoints for virtual reality presentation for the right eye. The virtual reality presentation video for the video and the left eye is generated. The generated virtual reality presentation video is displayed on the right display panel 151a and the left display panel 151b through correction processing and layer synthesis processing.

The reproduction audio control unit 160 performs a decoding process on the virtual reality presentation audio encoded data received by the communication unit 190, and distributes the audio data after the decoding process to each channel as necessary. Or, an amplification process for adjusting the sound level is performed.

The audio output unit 170 outputs the virtual reality presentation audio of each channel in accordance with the control from the reproduction audio control unit 160. Specifically, the audio output unit 170 includes a plurality of speakers 171a171b, performs DA conversion processing on the virtual reality presentation audio data reproduced by the reproduction audio control unit 160, converts the audio data into an analog electric signal, and further converts the audio data. Sound is output by converting it into vibration and emitting it to the outside.

The virtual reality presentation control unit 180 controls each process for virtual reality presentation on the virtual reality presentation device 100 side. Based on various notification information and parameter change information received by the communication unit 190, the virtual reality presentation control unit 180 encodes / decodes data in transmission / reception, a detection cycle of a user state and surrounding conditions, a frame rate of video display, and the like. Set the format. For example, the virtual reality presentation control unit 180 sets the type of user state to be detected by the user state detection unit 110 based on the detected user state designation information of the virtual reality control parameter information included in the virtual reality service provision start information, Based on the user state detection frequency, a user state detection frequency to be detected by the user state detection unit 110 is set.

Further, the virtual reality presentation control unit 180 monitors the state of the virtual reality presentation device 100 such as the remaining battery level, temperature, resource consumption, and wireless communication speed, and collects these information in a report to provide a virtual reality service providing server 800. Send to. The report is sent to the virtual reality service provision control unit 240 and used for various controls.

The communication unit 190 transmits user state information, ambient state information, audio information, and various control information generated in the device 100 to the virtual reality service providing server 800, and the virtual reality presentation transmitted from the virtual reality service providing server 800. Video encoded data, virtual reality presentation audio encoded data, various control information and notification information are received.

As described above, in the virtual reality presentation system according to the seventeenth embodiment, related data related to the virtual reality world is stored and stored in a state where the virtual reality world presented to the user can be reproduced. Therefore, when it is no longer possible to determine whether the world that you have experienced is the virtual reality world or the real world, it is possible to reproduce the same virtual reality world based on the relevant stored data. The user can appropriately determine that it was a virtual reality world.

In the above description, the case where the virtual reality related data acquisition unit 880 acquires the virtual reality presentation video data and the virtual reality presentation audio data transmitted to the virtual reality presentation device 100 and stores them in the database 500 has been described. However, it is even better if the configuration is such that the ambient state information is acquired together and stored in the database 500.

The user senses the virtual reality world by simultaneously viewing the virtual reality presentation video and the outside scene. Here, when the virtual reality world is reproduced at a later date, the scenery of the outside world may be different. For example, if the user is receiving a virtual reality service called “Idol X and One Hour Date” while in the park, the user will still have the experience of dating the Idol X and the park for an hour. On the other hand, when reproducing at home at a later date, the scenery of the outside world is the scenery of the home, so even if the same video and audio are reproduced, the impression of the virtual reality world felt at that time is different.

Therefore, the surrounding scene image which is the surrounding situation information detected by the surrounding situation detection unit 120 is stored together, and at the time of reproduction, the gaze scene image photographed by the gaze camera 121 and the virtual reality presentation video are synthesized. It is possible to reproduce a world similar to the virtual reality world at that time. This is because the gaze scenery image includes a park scenery that is equivalent to the scenery of the outside world that the user was viewing at that time, and the virtual reality object image of the idol X that is the virtual reality presentation video is included in the image. By multiplexing and synthesizing the video, the scene that was perceived by the user at that time is reproduced.

It is further preferable that the virtual reality related data acquisition unit 880 further acquires voice information input by the virtual reality presentation device 100 and stores it in the database 500. Since it is possible to reproduce what kind of words the user himself has spoken and what kind of noise or the like was heard in the surroundings, the degree of reproduction of the virtual reality world can be increased.

In the above description, the case where the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted from the communication unit 210 is stored as virtual reality related data has been described. Therefore, there arises a problem that a sufficient space must be secured in the database 500. The virtual reality presentation video is a video controlled based on the user state at the time, but the user state at the time of reproduction is different from the user state at the time, so the virtual reality object is displayed in the display panel at the time of reproduction. Will move without permission.

Therefore, the virtual reality related data acquisition unit 880 can be configured to acquire other information as virtual reality related data. FIG. 89 is a diagram illustrating an example of a combination of virtual reality related data acquired by the virtual reality related data acquisition unit 880 so that it can be reproduced at a later date.

In the pattern 01, the virtual reality related data acquisition unit 880 sequentially acquires and collects the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted to the virtual reality presentation device 100 as described above. And stored in the presented virtual reality related data storage DB 500. In this case, the reproducibility when reproducing the virtual reality world presented to the user based on the acquired virtual reality related data is 5 in the 10-level evaluation. This is because the virtual reality world sensed by the user has elements of ambient noise and surrounding scenery, but these elements cannot be reproduced only by such information.

In the pattern 02, the virtual reality related data acquisition unit 880 receives the line of sight received by the communication unit 210 in addition to the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted to the virtual reality presentation device 100. Scenery images are sequentially acquired and collected and stored in the presented virtual reality related data storage DB 500. By acquiring a gaze-scene image and storing it in time series, the virtual reality-presentation video obtained by decoding the virtual reality-presentation video-encoded data is multiplexed and synthesized with the gaze-scene image. Can be reproduced at a high level. Therefore, the reproducibility in this case is set to 6 which is higher.

In the pattern 03, the virtual reality related data acquisition unit 880 receives the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted from the communication unit 210 to the virtual reality presentation device 100, and the communication unit 210 receives them. The acquired line-of-sight image and user audio information are sequentially acquired and collected, and stored in the presented virtual reality related data storage DB 500. By acquiring and storing user voice information together, it is possible to reproduce what kind of words the user was talking to the virtual reality object at that time, so that the situation at that time can be reproduced at a higher level . Therefore, the reproducibility in this case is set to 6 which is even higher. Note that the virtual reality-related data acquisition unit 880 further increases the reproducibility when it is configured to further acquire the peripheral audio information and store it in the database 500.

In the pattern 04, the virtual reality related data acquisition unit 880 controls the user status information received by the communication unit 210, the surrounding state information, the voice information (user voice information and surrounding voice information), and the control program execution history. The program execution history information is sequentially acquired / collected and stored in the presented virtual reality related data storage DB 500.

The information necessary for reproducing the virtual reality world is an output from the virtual reality service providing server 800. The virtual reality presentation video and the virtual reality presentation audio, which are the outputs, are input to the server 800. And generated by a program executed by the input. Therefore, by sequentially acquiring and storing the input and the history of the executed program, it is possible to obtain the output again by simulating based on this information, and reproduce the virtual reality world. be able to.

Here, the control program execution history information is information that summarizes the relationship between the types of control programs and the times that the virtual reality service provision control unit 240 reads from the virtual reality content storage DB 400 and executes. The virtual reality related data acquisition unit 880 monitors the virtual reality service provision control unit 240, and displays control program execution history information that summarizes the types and times of control programs executed by the virtual reality service provision control unit 240. get. The virtual reality related data acquisition unit 880 is more preferably configured to acquire the control program execution history information further together with the type and time of the virtual reality object behavior control program executed by the virtual reality object behavior control unit 250. It is. The reproducibility in this pattern 04 is 5.

In the pattern 05, the virtual reality related data acquisition unit 880 includes viewpoint information on the viewpoint position coordinates and the line-of-sight direction controlled by the viewpoint control unit 830, the position coordinates of the light source controlled by the light source control unit 840, the light source illuminance, and the light source Light source information relating to color, model data information relating to model data of an object in the real world arranged by the model data arrangement processing unit 850, virtual reality object history information indicating a history of virtual reality objects arranged in the virtual space, Are sequentially acquired and stored in the database 500.

The user status information and ambient status information received by the communication unit 210 are information used for obtaining coordinate information, light source information, and the like. Therefore, coordinate information and light source information, which are results obtained by processing these pieces of information, instead of user state information and surrounding state information, are acquired as virtual reality related data. With this configuration, the amount of information to be stored can be further compressed. Coordinate information is history information that summarizes temporal transitions of viewpoint position coordinates and line-of-sight directions in virtual space, and light source information summarizes temporal transitions of information related to light source position coordinates, light source illuminance, light source color, etc. in virtual space. It is history information.

The virtual reality object history information includes identification information (identification number) for identifying a virtual reality object arranged in the virtual space, position coordinates of the virtual reality object in the virtual space, information on the behavior of the virtual reality object, and the like. The history information is summarized in time series. The appearance of the virtual reality object changes by changing points such as joint data and skeleton data or changing the type of texture data pasted on the surface by the virtual reality object behavior control program. The information related to the behavior of the virtual reality object includes, for example, joint data that defines the position coordinates and angles of joint points that are movable parts in the virtual reality object, parameter values that specify the type and expression of the texture attached to the surface, etc. It is. The virtual reality related data acquisition unit 880 monitors a virtual reality object placed in the virtual space by the virtual reality object placement processing unit 820, and monitors temporal changes in joint data, skeleton data, and texture data in the virtual reality object. Thus, virtual reality object history information is acquired.

According to the virtual reality object history information, the virtual reality object is read from the virtual reality content storage DB 400, and the arrangement and behavior are controlled again in the virtual space, projected from the viewpoint position indicated by the viewpoint information onto the image plane, and shaded based on the light source information By performing the above, it is possible to generate the video generated at that time again. Therefore, the virtual reality world at that time can be reproduced. In this case, the reproducibility is 3.

In the pattern 06, the virtual reality related data acquisition unit 880 acquires viewpoint information, light source information, virtual reality object history information, line-of-sight scene images, and user audio information, and stores them in the database 500. Here, the model data of the pattern 05 is data used for hidden surface processing at that time, but since the environment around the user has changed at the time of reproduction, this processing is a process of regenerating the virtual reality presentation video. However, since it makes little sense, it is not acquired as virtual reality related data. Instead, by storing the line-of-sight scene image and the user voice information, it is possible to reproduce the scene at that time and the voice generated by itself. In this case, since the reproducibility of the virtual reality world is relatively high, the reproducibility is set to 8.

In the pattern 07, the virtual reality related data acquisition unit 880 acquires the light source information, the virtual reality object history information, and the reproduction sound generation history information and stores them in the database 500. The virtual reality presentation audio is generated by the reproduction audio generation processing unit 270 appropriately reading out relevant audio data from the virtual reality content storage DB 400. If the history of which audio data is read at which time is known, the virtual reality presentation audio is generated. Can be reproduced. Therefore, the virtual reality related data acquisition unit 880 monitors the playback audio generation processing unit 270 and acquires the history of audio data read by the playback audio generation processing unit 270 as playback audio generation history information.

Since the viewpoint information is not acquired compared to the pattern 05, the virtual reality object cannot be observed from the same angle or distance as that time, but the virtual reality object can be observed from the third party's position. The world of virtual reality can be reproduced. However, the reproducibility in this case is relatively low, and the reproducibility is 4.

In the pattern 08, the virtual reality related data acquisition unit 880 acquires access history information to the virtual reality content storage DB 400 as virtual reality related data and stores it in the database 500.

Since the output from the virtual reality service providing server 800 to the virtual reality presentation device 100 is obtained by executing various data and programs read from the virtual reality content storage DB 400, the access history to the virtual reality content storage DB 400 is suppressed. By doing so, it is possible to reproduce the virtual reality world.

In the pattern 09, the virtual reality related data acquisition unit 880 has access history information to the virtual reality content storage DB 400, user status information received by the communication unit 210, and a voice context that is a voice analysis result in the voice analysis processing unit 220. The information and the image analysis result in the image analysis processing unit 230 are acquired as virtual reality related data and stored in the database 500.

As described above, the virtual reality related data acquisition unit 880 acquires data used for constructing the virtual reality world presented to the user by the virtual reality presentation device 100 as virtual reality related data, and the presented virtual reality related data The storage DB 500 stores the virtual reality related data. The reproducibility (reproduction level) of the virtual reality world to be reproduced changes depending on what kind of data is acquired and stored as virtual reality related data.

In general, the greater the amount of virtual reality-related data to be stored, the higher the reproducibility of the virtual reality world. On the other hand, storing a large amount of data that cannot be used reduces the efficiency of database use. Therefore, the selection is appropriately made in consideration of the relationship between the reproducibility of the virtual reality world and the data amount.

Here, in order to reproduce the virtual reality world, a video of the virtual reality object represented in the virtual reality world is required. Therefore, at least the virtual reality presentation video data for which the virtual reality object has been drawn or information for deriving the virtual reality object arranged in the virtual space is required. By storing these pieces of information, it is possible to reproduce a virtual reality world with a level of reproducibility of 1.

Here, the virtual reality object operates in the virtual reality world. Therefore, even if the virtual reality object represented in the presented virtual reality world can be identified, there is a difference from the virtual reality world felt by the user at that time alone. Therefore, it is preferable to store the virtual reality presentation video data or to further store information for deriving the motion history of the virtual reality object because the reproducibility is increased.

In the virtual reality world, the virtual reality object emits sound. Users who are impressed with this voice may not believe it was virtual unless they listen to it again. For this reason, it is preferable to further store information for guiding the virtual reality presentation audio data or the audio generated by the virtual reality object because the reproducibility increases.

In addition, as described above, the user senses the virtual reality world by detecting the virtual reality presentation video and the virtual reality presentation sound, and the scenery and sound from the real world together. Sense. Therefore, it is preferable to memorize the scenery in the real world at that time and the surrounding sound in the real world in order to reproduce the virtual reality world sensed by the user at a high level.

Thus, as an example, the virtual reality related data acquisition unit 880 uses the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server 800 to the virtual reality presentation device 100 as virtual reality. The acquired virtual reality related data storage database 500 acquired as related data stores the acquired virtual reality presentation video data and virtual reality presentation audio data.

More preferably, the virtual reality related data acquisition unit 880 acquires, as virtual reality related data, forward scenery images sequentially transmitted from the virtual reality presentation device 100 to the virtual reality service providing server 800, and presents the virtual reality related data storage database that has been presented. 500 stores the acquired forward scene image as virtual reality related data. It is further preferable that the virtual reality related data acquisition unit 880 acquires the surrounding scenery image together and the presented virtual reality related data storage database 500 stores the surrounding scenery image.

  More preferably, the virtual reality related data acquisition unit 880 sequentially acquires user status information transmitted from the virtual reality presentation device 100 to the virtual reality service providing server 800, and the presented virtual reality related data storage database 500 stores the acquisition. The user state history information in which the obtained user state information is collected in time series is stored. If the user status information includes an image taken with an internal camera that captured the user's facial expression, the user's facial expression is obsessed with the virtual reality world. Can be reviewed more objectively.

More preferably, the virtual reality related data acquisition unit 880 acquires various audio data such as user audio data and peripheral audio data input by the virtual reality presentation device 100, and the presented virtual reality related data storage database 500 Voice history information in which the acquired voice data is collected in time series is stored. Since users who are in the virtual reality world have a two-way conversation with the virtual reality object, it is possible to store voices uttered from the user side and voices from the external environment together. , Increase the reproducibility of the virtual reality world.

As another form, the virtual reality related data acquisition unit 880 sequentially acquires and presents virtual reality object identification information for identifying virtual reality objects arranged in at least the virtual space generated by the virtual space generation processing unit 810. The virtual reality related data storage database 500 stores virtual reality object history information in which virtual reality object identification information for identifying virtual reality objects arranged in the acquired virtual space is collected in time series. According to this configuration, according to the virtual reality object history information, the virtual reality world identified by the virtual reality object identification information is again read from the virtual reality content storage database 400 and arranged in the virtual space. Can be reproduced at a certain level.

Here, it is even better if the virtual reality related data acquisition unit 880 acquires the position coordinates of the virtual reality object in the virtual space and information related to the behavior of the virtual reality object as virtual reality related data. The presented virtual reality related data storage database 500 includes virtual reality object history information in which position coordinates of the acquired virtual reality object in the virtual space and information related to the behavior of the virtual reality object are further collected in time series. Remember. With this configuration, when multiple virtual reality objects are placed in the virtual space at the same time, the relative positional relationship of the virtual reality objects is an important factor in reproducing the virtual reality world. It is possible to reproduce the relative positional relationship. Further, when the virtual reality object is an operable object, the virtual reality world at that time can be reproduced by reproducing the motion.

In addition, the virtual reality related data acquisition unit 880 acquires, as virtual reality related data, an execution history of a behavior control program that controls at least the behavior of the virtual reality object arranged in the virtual space generated by the virtual space generation processing unit 810. The presented virtual reality related data storage database 500 may be configured to store behavior control program execution history information in which the acquired joint control program execution history is collected in time series.

More preferably, the virtual reality related data acquisition unit 880 is a voice context sequentially extracted by the voice analysis processing unit 220 (voice context extraction unit) from the user voice input by the user microphone 131 of the virtual reality presentation device 100. Information related to emotional elements is acquired as virtual reality related data, and the presented virtual reality related data storage database 500 stores voice context history information in which acquired voice contexts and emotional elements are sequentially collected in time series.

Further, the virtual reality related data acquisition unit 880 acquires an access history for the virtual reality content storage database 500 as virtual reality related data, and the presented virtual reality related data storage database 500 is an access history for the virtual reality content storage database 500. May be configured to store access history information gathered in time series.

The presented virtual reality related data storage database 500 stores the above-described various data in association with each virtual reality identification information for identifying the virtual reality world, so that the data related to the virtual reality world to be reproduced can be read collectively. preferable.

FIG. 90 is a block diagram of the entire system including the virtual reality reproduction request processing server 900 that reproduces the presented virtual reality world. The virtual reality reproduction request processing server 900 includes a communication unit 910, a reproduction request processing unit 920, a virtual reality related data reading unit 930, an audio analysis processing unit 220, an image analysis processing unit 230, and a virtual reality service provision control unit. 240, a virtual reality object behavior control unit 250, a display video generation processing unit 260, a reproduction audio generation processing unit 270, a virtual space generation processing unit 810, a virtual reality object arrangement processing unit 820, and a viewpoint control unit 830 A light source control unit 840.

The communication unit 910 communicates with the virtual reality presentation device 100n and the PC 20 that request reproduction of virtual reality. The communication unit 910 includes a reproduction request receiving unit 911 that receives a virtual reality reproduction request that requests reproduction of the virtual reality world, and a reproduction virtual reality that transmits virtual reality presentation data for constructing the virtual reality world to be reproduced. A data transmission unit 912 for presentation.

The reproduction request processing unit 920 determines whether to execute processing for reproducing the virtual reality world based on the reproduction request received by the reproduction request receiving unit 911. The reproduction request includes at least the user ID for identifying the user, the user password, and the presented virtual reality identification information for specifying the virtual reality world for which reproduction is requested. The reproduction request processing unit 920 performs authentication processing by referring to the contractor information stored in the contractor information storage DB 300 as to whether the set of the user ID and the password included in the reproduction request matches. If the reproduction request is appropriate, the reproduction request processing unit 920 causes the virtual reality related data reading unit 930 to store virtual reality related data related to the virtual reality world specified by the virtual reality identification information included in the reproduction request. A read instruction is issued.

Based on the instruction, the virtual reality related data reading unit 930 sequentially reads each virtual reality related data identified by the virtual reality identification information from the presented virtual reality related data storage DB 500 and outputs it to the corresponding units.

For example, when the virtual reality presentation video data and the virtual reality presentation audio data are stored as they are in the presented virtual reality related data storage DB 500 as virtual reality related data, the virtual reality related data reading unit 930 may The data is read out and output to the reproduced virtual reality presentation data transmission unit 912. The reproduction virtual reality presentation data transmission unit 912 transmits the virtual reality presentation video data and the virtual reality presentation audio data read by the virtual reality related data reading unit 930 to the reproduction request source apparatus.

In addition, when the gaze scene image is included in the virtual reality related data, the gaze scene image may be combined and transmitted to the requesting apparatus, or a video composition processing unit may be included in the virtual reality reproduction request processing server 900. May be provided, and the combined virtual reality presentation video data obtained by synthesizing the virtual reality presentation video data and the surrounding scene image may be transmitted to the reproduction request source apparatus.

When virtual reality related data is stored in the above format, other functional blocks are not necessary. However, when virtual reality related data is stored in the form of control program execution history information, a virtual The reality reproduction processing server 900 needs to regenerate a virtual reality presentation video or the like. In this case, the virtual reality related data reading unit 930 outputs the read control program execution history information to the virtual reality service provision control unit 240. The virtual reality service provision control unit 240 reads the corresponding control program from the virtual reality content storage DB 400 and executes it at each timing indicated by the control program execution history information, thereby generating the display video generation processing unit 260 and the reproduction audio generation. The virtual reality presentation video data and the virtual reality presentation audio data related to the virtual reality world already presented by the processing unit 270 are regenerated. The reproduction virtual reality presentation data transmission unit 912 transmits the regenerated virtual reality presentation video data and virtual reality presentation audio data to the reproduction request source apparatus.

When the virtual reality related data is stored in the format of access history information to the virtual reality content storage DB 400, the virtual reality related data reading unit 930 uses the read access history information as the virtual reality service provision control unit 240, The virtual reality object behavior control unit 250, the virtual reality object arrangement processing unit 820, the playback sound generation processing unit 270, and the like are output to each unit that reads corresponding data and programs from the virtual reality content storage DB 400. Each of these units reads and processes the specified data and programs at the timing indicated by the access history information, thereby processing the virtual reality presentation video data and virtual reality presentation related to the presented virtual reality world. The audio data is regenerated by the display video generation processing unit 260 and the reproduction audio generation processing unit 270. The reproduction virtual reality presentation data transmission unit 912 transmits the regenerated virtual reality presentation video data and virtual reality presentation audio data to the reproduction request source apparatus.

As described above in Embodiments 15 to 17, according to the present invention, a user who has experienced a virtual reality world can request to reproduce the same virtual reality world again.

By changing the image displayed according to the change in the user's viewpoint position and line-of-sight direction, the object or person on the image displayed on the display panel is presented to the user so as to actually exist in the real world. A virtual reality world is built. Here, the virtual reality presentation device is not limited to the transmissive head-mounted display device. For example, the virtual reality world can be displayed to the user by synthesizing and displaying the virtual reality presentation video in which the virtual reality object is drawn on the video captured by the line-of-sight cameras respectively disposed at the positions of the left and right eyes of the user. It can be set as the structure which presents.

Note that the above-described voice analysis processing unit and image analysis processing unit have a configuration in which a dedicated voice analysis server and image analysis server are separately provided to communicate with the virtual reality service providing server. In addition, each processing unit described above may be included in one information processing server, or a plurality of information processing servers may be linked.

Further, in the above description, the case where the reproduction sound generation processing unit included in the virtual reality service providing server generates sound data for virtual reality presentation has been described. However, virtual reality presentation in a state that can be reproduced in advance in the virtual reality content storage DB 400 has been described. When the audio data is stored, the reproduction audio generation processing unit may be configured to read out the virtual reality presentation audio data and transmit it to the virtual reality presentation device.

The line-of-sight camera and the peripheral camera may be configured to capture a still image and transmit it to the virtual reality service providing server, or may be configured to capture a moving image and transmit it to the virtual reality service providing server.

  Note that the virtual reality related data acquisition unit 880 may be configured to store the data after reducing the resolution of the video data for virtual reality presentation transmitted to the virtual reality presentation device 100 to reduce the data.

Note that it is better if the above-described virtual reality content includes special video effect data such as snow, afterglow, rain, and thunder. The display video generation processing unit is more preferably configured to generate a composite video in which a layer of the special video effect is superimposed on a video in which a virtual reality object is drawn as a virtual reality presentation video.

Note that the above-described method of taking virtual reality related data is an example, and a company that provides a virtual reality service can appropriately select and store the data. In general, in order to be able to reproduce the virtual world with high reproducibility, the amount of virtual reality-related data increases, so it is necessary to prepare a database with a large capacity. For this reason, the data amount of the virtual reality related data may be suppressed even if the reproducibility is lowered. Further, the virtual reality reproduction request processing server may include an erasing unit that erases the virtual reality related data stored in the presented virtual reality related data storage DB 500 when a certain period of time has passed.

Further, the virtual reality reproduction request processing server and the virtual reality service providing server described above may be configured integrally because there are many common processing units.

Each processing unit described above can be realized by hardware and software working by causing an information processing apparatus such as a CPU to execute a predetermined program.

Moreover, it is also possible to combine the structure of each embodiment mentioned above suitably. For example, the virtual reality presentation system of the first embodiment, the virtual reality presentation system of the tenth embodiment, and the virtual reality presentation system of the fifteenth embodiment are combined, and the virtual reality presentation device and the virtual reality service providing server included in each virtual reality presentation system The virtual reality presentation device and the virtual reality service providing server each having the above functions may be used. Further, for example, by integrating the virtual reality presentation systems shown in the seventh embodiment, the fourteenth embodiment, and the seventeenth embodiment, when providing a virtual reality service, the discontinuity between the real world and the virtual reality world can be suppressed or the virtual reality can be reduced. High illusion is realized by presenting the virtual reality world to the user while controlling it, making it possible to immerse the user in the virtual reality world and increasing the user's satisfaction while providing it to the user By making it possible to reproduce the history of the virtual reality service at a later date, it may be configured to prevent the distinction between the real world and the virtual reality world even after the virtual reality service is fully enjoyed. The virtual reality presentation device used in the virtual reality presentation system integrated in this way has the functions of the virtual reality presentation device included in each of the seventh, fourteenth, and seventeenth embodiments. Similarly, the virtual reality service providing server is the same as the embodiment. It is preferable that the virtual reality service providing server included in each of 7, 14, and 17 has each function, and other databases and auxiliary servers are arranged in the integrated virtual reality presentation system. Further, a part of the virtual reality service providing system may be arranged in the home in the form of a home server.

Virtual reality is sometimes referred to as augmented reality. Further, a part of the video presented to the user may be configured to display an area determined based on the user state from an image generated in advance as a background image instead of a video drawn in real time.

In addition, as described in the above embodiments 10 to 14, according to the present invention, according to newly introduced parameters, various data for virtual reality presentation are generated and presented so as to be within the allowable parameter range. . Therefore, the virtual reality world is appropriately presented to the user.

  The virtual reality content storage unit described above is preferably configured to store and manage various types of virtual reality content in the form of a virtual reality content storage database. In addition, it is preferable that a dedicated image analysis server is separately provided for the image analysis processing unit so as to communicate with the virtual reality service providing server. In addition, each processing unit described above may be included in one information processing server, or a plurality of information processing servers may be linked.

It should be noted that the method for obtaining and calculating various parameters such as the presentable virtual reality, the presenting virtual reality, and the illusion are as described above depending on the virtual reality presentation device manufacturer and the virtual reality service providing system. It is possible to select.

The virtual reality that can be presented may be set for each virtual reality service presented to the user, or may be set for each user. In a virtual reality service that presents a virtual reality world with a high fantasy virtual reality, the virtual reality is high. Therefore, it is preferable to calculate a virtual reality with a high presentable virtual reality. Also, in order to realize a virtual reality world with generally high virtual reality, various virtual reality objects are arranged, special video effects etc. are used, the frame rate of the generated video is increased, The amount of processing tends to increase. Since the amount of resources consumed in the virtual reality service providing system increases, a configuration in which a high virtual reality is set for a user who has a high price contract and a cheap virtual reality is assigned to a free user or the like This is preferable because a business model is established. That is, the provisionable virtual reality calculation unit 1020 reads the contract information of the user who provides the virtual reality service from the contract information storage unit 3060, and based on the contract information, the virtual reality degree related to the virtual reality world that can be provided It is good to have a configuration to set.

Moreover, it is good to set it as a virtual reality which can provide a high virtual reality at the time of a user's waking and going to bed. Since the user is asleep when waking up, the boundary between the dream and the real world is unclear. Therefore, the user goes to bed while wearing the virtual reality presentation device, and presents the virtual reality world with a high virtual reality to the user when the user wakes up. Since the user who sees the virtual reality world is sleeping and has a low judgment power, the user sees the scenery he / she sees as a dream image. Therefore, the user can enjoy the virtual reality world presented by the virtual reality presentation device with a feeling of dreaming.

Further, each function described above can be realized by hardware and software working by causing an information processing apparatus such as a CPU to execute a predetermined program.

Moreover, it is also possible to combine the structure of each embodiment mentioned above suitably. Further, a part of the virtual reality service providing system may be arranged in the home in the form of a home server.

Also, the virtual reality described above may be referred to as augmented reality. Accordingly, the virtual reality presentation device may be referred to as an augmented reality presentation device, the virtual reality presentation system as an augmented reality presentation system, the virtual reality service providing server as an augmented reality service providing server, and the virtual reality service as an augmented reality service. is there. Further, a part of the video presented to the user may be configured to display an area determined based on the user state from an image generated in advance as a background image instead of a video drawn in real time.

In addition, as described in the first to ninth embodiments, according to the virtual reality presentation system according to the present invention, the virtual reality world is not immediately presented to the user based on a clear instruction or request from the user. Rather, the presentation timing is controlled based on the user's state and presented to the user. With this configuration, the user can naturally enter the virtual reality world.

Each embodiment described above shows an example of the present invention, and can be changed as appropriate. For example, the image analysis unit included in the virtual reality service providing server may be configured to be arranged in the virtual reality service providing system independently as the image analysis server. Moreover, it is also possible to combine the structure of each embodiment.

It is also possible to combine the above-described embodiments. In addition, the present invention can be configured as follows.
(Appendix 1)
User status detection means for detecting the user status;
Video generation means for generating video for virtual reality presentation based on the user's state;
Video display means for displaying the generated virtual reality presentation video;
A determination unit that determines whether a user state detected by the user state detection unit satisfies a predetermined criterion;
Video display control means for performing control to display the virtual reality presentation video on the video display means when it is determined that the criterion is satisfied;
A virtual reality presentation system comprising:
(Appendix 2)
Virtual reality object storage means for storing virtual reality objects;
Virtual space setting means for setting the virtual space;
Virtual reality object placement means for reading the virtual reality object from the virtual reality object storage means and placing it in the set virtual space;
Viewpoint control means for updating the position coordinates and the line-of-sight direction of the viewpoint in the set virtual space based on the state of the user detected by the user state detection means;
Further comprising
The video generation means renders the virtual reality object by drawing the virtual reality object arranged in the virtual space based on the position coordinates and the line-of-sight direction of the viewpoint updated by the viewpoint control means. Generate video for
The virtual reality presentation system according to attachment 1.
(Appendix 3)
Ambient shooting means for shooting the surrounding scenery and obtaining the surrounding scenery image;
Optical component extraction means for extracting a predetermined optical component from the surrounding scenery image;
Light source control means for updating the position coordinates of the light source in the virtual space and the luminance or illuminance of the light source based on the extracted optical component;
Further comprising
The video generation means performs a shading process based on the position coordinates of the light source and the luminance or illuminance of the light source updated by the light source control means when the virtual reality object arranged in the virtual space is drawn. To generate the video for presenting the virtual reality,
The virtual reality presentation system according to attachment 2.
(Appendix 4)
Modeling processing means for modeling each object in the real world reflected in the surrounding landscape image and generating a real object;
Model data placement means for placing the real object generated by the modeling processing means in the virtual space;
Further comprising
The video generation means hides the virtual reality object arranged by the virtual reality object arrangement means with the real object arranged by the model data arrangement means when the virtual reality object arranged in the virtual space is drawn. By performing processing, the virtual reality presentation video is generated.
The virtual reality presentation system according to appendix 2 or 3.
(Appendix 5)
The virtual reality object arranging means arranges a plurality of virtual reality objects in different layers of the virtual space,
The video generation means generates a virtual reality presentation video for each layer by drawing a virtual reality object for each layer of the virtual space,
The display control means synthesizes the layered virtual reality presentation video, and performs control to display the synthesized video on the display means.
The virtual reality presentation system according to attachment 4.
(Appendix 6)
The virtual reality object placement means places a newly placed virtual reality object in a new virtual space layer,
The video generation means generates a virtual reality presentation video for each layer by drawing a virtual reality object for each layer of the virtual space,
The determination means determines whether to start displaying the video of the new virtual reality object,
When the determination unit determines that the display of the video of the new virtual reality object is to be started, the display unit displays another virtual reality presentation video for each layer in which the new virtual reality object is arranged. Is combined with the virtual reality presentation video of each layer and performs control to display the composited video on the display means.
The virtual reality presentation system according to attachment 5.
(Appendix 7)
User status detection means for detecting the user status;
Video display means for displaying video;
A determination unit that determines whether a user state detected by the user state detection unit satisfies a predetermined criterion;
Display that performs control to display on the video display means a video for virtual reality presentation that is controlled based on the user status detected by the user status detection means when it is determined that the standard is satisfied Control means;
A virtual reality presentation device.
(Appendix 8)
The user state detection means detects the state of the user's face by photographing a part or all of the user's face,
The determination means determines whether the user's eyes are closed for a predetermined time based on the video imaged by the user state detection means,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the user's eyes are closed for a predetermined time.
The virtual reality presentation device according to appendix 7.
(Appendix 9)
The determination means determines whether the user has closed eyes deeply based on the video taken by the user state detection means,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the user has closed eyes deeply;
The virtual reality presentation device according to appendix 8.
(Appendix 10)
The user state detection means further detects the user's brain waves,
The determination unit determines whether the user's brain wave value detected by the user state detection unit satisfies a predetermined criterion,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the value of the user's brain wave satisfies a predetermined criterion;
The virtual reality presentation device according to any one of appendices 7 to 9.
(Appendix 11)
The user state detection means detects the direction the user is facing,
The determination unit determines whether the direction of the user detected by the user state detection unit has changed by a predetermined reference angle or more,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the direction in which the user is facing has changed by a predetermined reference angle or more.
The virtual reality presentation device according to any one of appendices 7 to 10.
(Appendix 12)
Transmitting means for transmitting information indicating the state of the user detected by the user state detecting means;
Receiving means for receiving the information indicating the predetermined reference and the virtual reality presentation video;
Further comprising setting means for setting the predetermined reference based on the received information;
The determination unit determines whether the user's state detected by the user state detection unit satisfies the predetermined criterion set by the setting unit;
The display control means is the receiving means when the determining means determines that the user's condition detected by the user condition detecting means satisfies the predetermined reference set by the setting means. Control to display the received virtual reality presentation video on the video display means;
The virtual reality presentation device according to any one of appendices 7 to 11.
(Appendix 13)
A user status detection step for detecting a user status;
A determination step of determining whether the detected state of the user satisfies a predetermined criterion;
A video generation step of generating a video for virtual reality presentation based on the detected state of the user;
A video display control step of performing control to display the virtual reality presentation video when it is determined that the criterion is satisfied;
Based on the control, a video display step for displaying the generated video for virtual reality presentation;
A virtual reality presentation method comprising:
(Appendix 14)
User status detection means for detecting the user status;
Video display means for displaying video;
A determination unit that determines whether a user state detected by the user state detection unit satisfies a predetermined criterion;
Display that performs control to display on the video display means a video for virtual reality presentation that is controlled based on the user status detected by the user status detection means when it is determined that the standard is satisfied Control means;
A head-mounted display device comprising:
(Appendix 101) (Basic design)
User status detection means for detecting the user status;
Presentable virtual reality calculation means for calculating the virtual reality of the present virtual reality world;
Video generation processing means for generating a video for virtual reality presentation so as to be within the virtual reality calculated by the presentable virtual reality calculation means based on the state of the user;
Display means for displaying the generated video;
A virtual reality presentation system comprising:
(Supplementary Note 102) (Virtual Reality Calculation Method)
A virtual reality calculation means for presenting that calculates the virtual reality of the virtual reality world currently being presented;
The presentable virtual reality calculation means is based on the virtual reality of the present virtual reality world calculated by the presenting virtual reality calculation means. Calculate the degree,
The virtual reality presentation system according to attachment 101.
(Supplementary Note 103) (Illusion degree)
An illusion degree calculating means for calculating the illusion degree of the user is provided,
The presentable virtual reality calculation unit is configured to calculate the illusion degree calculated by the illusion degree calculation unit based on the virtual reality degree of the currently presented virtual reality calculated by the presenting virtual reality calculation unit. Calculate the virtual reality of the virtual reality world that can be presented by adding, subtracting, multiplying, or dividing the value determined based on
The virtual reality presentation system according to attachment 102.
(Appendix 104) (Degree of Illusion)
The user state detection means includes an electroencephalogram sensor that detects a user's brain wave, an internal camera that captures part or all of the user's face, a blood pressure sensor that measures the user's blood pressure, a pulse sensor that measures the user's heart rate, and the user Comprising at least one perspiration amount sensor for detecting the perspiration amount of
The illusion degree calculating means calculates the illusion degree of the user based on a detection result in any one of the sensors provided in the user state detecting means.
The virtual reality presentation system according to attachment 103.
(Supplementary Note 105) (Virtual Reality Object)
Virtual reality content storage means for storing at least a plurality of virtual reality objects;
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object in the virtual space;
Viewpoint control means for controlling viewpoint coordinates and line-of-sight direction in the virtual space based on the user state;
Drawing processing means for drawing the virtual reality object arranged in the virtual space based on the viewpoint coordinates and the line-of-sight direction updated by the viewpoint control means;
With
The virtual reality object arranging means arranges the virtual reality object in the virtual space so as to be within the virtual reality calculated by the presentable virtual reality calculating means;
The virtual reality presentation system according to any one of appendices 101 to 104.
(Supplementary Note 106) (Delete Virtual Reality Object)
The virtual reality object placement unit is configured to display a virtual reality object being placed in a virtual space when the virtual reality of the presentable virtual reality world is lower than the virtual reality of the virtual reality world currently being presented. By removing the virtual reality from the virtual space so that the virtual reality of the virtual reality world to be presented falls within the virtual reality of the virtual reality world that can be presented.
The virtual reality presentation system according to attachment 105.
(Supplementary Note 107) (Supports virtual reality)
The virtual reality content storage means stores the virtual reality object in association with the virtual reality set for each of the virtual reality objects;
The virtual reality object placement unit reads the virtual reality object associated with a virtual reality lower than the virtual reality calculated by the presentable virtual reality calculation unit from the virtual reality content storage unit and reads the virtual reality object Place it in virtual space,
The virtual reality presentation system according to attachment 106.
(Supplementary Note 108) (Excess Virtual Reality)
The virtual reality object placement unit obtains an excess virtual reality obtained by subtracting the virtual reality calculated by the presenting virtual reality calculating unit from the virtual reality calculated by the presentable virtual reality calculating unit. The virtual reality object associated with a lower virtual reality degree is read from the virtual reality content storage means and arranged in the virtual space;
108. A virtual reality presentation system according to appendix 107.
(Appendix 109) (by ambient conditions)
Ambient condition detection means for detecting the situation of the user's surroundings,
Ambient situation specifying means for specifying a situation around the user based on a detection result in the surrounding situation detection means,
With
The virtual reality content storage means stores a correspondence relationship between the virtual reality object and virtual reality for each surrounding situation,
The virtual reality object placement means is a virtual reality degree calculation means that provides a virtual reality degree corresponding to a situation around the user specified by the surrounding situation specifying means of the virtual reality object placed in the virtual space. The virtual reality object is selected and read from the virtual reality object storage means so as to be lower than the calculated virtual reality, and is arranged in the virtual space.
The virtual reality presentation system according to any one of appendices 105 to 108.
(Appendix 110) (Shading)
The ambient condition detection means captures an ambient scene image by photographing the situation of the user,
Light source control means for updating the position coordinates of the light source and the illuminance or luminance of the light source based on the surrounding scenery image;
Model data generating means for generating model data of an object included in the surrounding scenery image;
Further comprising
The drawing processing unit projects a virtual reality object arranged in the virtual space to generate a two-dimensional image based on the viewpoint coordinates and the line-of-sight direction updated by the viewpoint control unit; A shading process for calculating the brightness of the virtual reality object based on the position coordinates of the light source updated by the light source control unit and the illuminance or brightness of the light source; and a hidden surface process for hiding a part of the virtual reality object based on the model data; The virtual reality presentation video is generated by executing at least
The virtual reality presentation system according to any one of supplementary notes 105 to 109 (Supplementary note 201) (system basic design)
A virtual reality presentation device that presents a virtual reality world related to a virtual reality service to a user;
A virtual reality service providing server that generates at least virtual reality presentation video data to be used to construct the virtual reality world and transmits the virtual reality presentation video data to the virtual reality presentation device;
A presented virtual reality-related data storage database that stores data related to construction of the virtual reality world as virtual reality-related data in a state in which the virtual reality presentation device can reproduce the virtual reality world presented to the user; ,
A virtual reality presentation system comprising:
(Supplementary Note 202) (Present Virtual Reality History Information)
The virtual reality service providing server comprises virtual reality identification information assigning means for assigning virtual reality identification information for identifying the virtual reality world to be presented,
A virtual reality service for storing provided virtual reality history information in which correspondences between identification information for identifying a user and the virtual reality identification information assigned to the virtual reality world presented to the user are summarized in time series Further comprising a history information storage database;
The virtual reality presentation system according to attachment 201.
(Supplementary Note 203) (Provided Virtual Reality Service History Information)
Virtual reality service history information storing provided virtual reality service history information in which correspondences between identification information for identifying a user and virtual reality service identification information for identifying the virtual reality service provided to the user are summarized in time series Further comprising a storage database;
The virtual reality presentation system according to attachment 201.
(Supplementary Note 204) (Virtual Reality Object)
A virtual reality content storage database for storing virtual reality content used to construct the virtual reality world including at least the virtual reality object related to the virtual reality service;
The virtual reality service providing server includes:
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object related to the virtual reality service in the virtual space;
Video generation means for sequentially generating the virtual reality presentation video data by rendering the virtual reality object arranged in the virtual space in real time;
Comprising at least
The virtual reality presentation device presents the virtual reality world to a user by displaying a video including the virtual reality object based on the generated virtual reality presentation video data.
The virtual reality presentation system according to appendix 201-203.
(Supplementary Note 205) (Drawing Based on User Status)
The virtual reality presentation device includes:
User status detection means for detecting the user status;
Wireless transmission means for wirelessly transmitting user status information indicating the detected user status;
Wireless receiving means for wirelessly receiving the virtual reality presentation video data;
Display means for displaying a virtual reality presentation video based on the wirelessly received virtual reality presentation video data;
Comprising
The virtual reality service providing server includes:
Receiving means for receiving the user status information;
Transmission means for transmitting the virtual reality presentation video data sequentially generated by the video generation means;
Further comprising
The video generation means renders the virtual reality presentation video in real time by drawing the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction specified based on the user state information. Generate data sequentially,
The virtual reality presentation system according to attachment 204.
(Supplementary Note 206) (Sound for virtual reality presentation)
The virtual reality content storage database stores audio data related to the virtual reality service as the virtual reality content,
The virtual reality service providing server further includes voice generation processing means for generating virtual reality presentation voice data based on voice data related to the virtual reality service stored in the virtual reality content storage database,
The virtual reality presentation device further includes voice output means for converting the virtual reality presentation voice data into an analog electrical signal and outputting the virtual reality presentation voice to the outside.
The virtual reality presentation system according to attachment 205.
(Supplementary Note 207) (Landscape Image)
The virtual reality presentation device includes:
Voice input means for collecting user voice data by collecting voice uttered by the user;
Image photographing means for photographing at least the front direction of the user and acquiring a front scenery image;
Further comprising
The virtual reality service providing server includes:
Voice context extraction means for extracting a voice context included in the user voice data;
Light source control means for extracting optical information relating to illuminance or luminance from the front scenery image, and updating the position coordinates of the light source and the illuminance or luminance of the light source;
Viewpoint control means for updating the viewpoint position coordinates and the line-of-sight direction based on the user state information;
Further comprising
The video generation unit is configured to store the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit, and the light source position coordinates and the illuminance or luminance of the light source updated by the light source control unit. Sequentially generating the virtual reality presentation video data by rendering the arranged virtual reality objects in real time;
The virtual reality presentation system according to attachment 206.
(Appendix 208) (Shading)
The video generation unit generates a projection image by projecting the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit, and the light source control unit Generating the virtual reality presentation video by calculating the luminance or illuminance of the virtual reality object in the projection image based on the position coordinates of the light source and the illuminance or luminance of the light source updated by
The virtual reality presentation system according to appendix 207.
(Supplementary Note 209) (Virtual Reality Object Behavior Control)
The virtual reality content storage database further stores a virtual reality object behavior control program for controlling the behavior of the virtual reality object as the virtual reality content,
The virtual reality service providing server includes:
A virtual reality object behavior control means for controlling the behavior of the virtual reality object placed in the virtual space by executing the virtual reality object behavior control program read from the virtual reality content storage database;
The virtual reality presentation system according to appendix 207 or 208.
(Supplementary Note 210) (Type of Virtual Reality Related Data: Virtual Reality Presentation Data)
The presented virtual reality related data storage database stores the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server to the virtual reality presentation device.
The virtual reality presentation system according to appendix 209.
(Supplementary Note 211) (Type of Virtual Reality Related Data: Ambient Scene Image)
The presented virtual reality related data storage database stores the front scene image transmitted from the virtual reality presentation device to a virtual reality service providing server,
The virtual reality presentation system according to attachment 210.
(Supplementary Note 212) (Type of Virtual Reality Related Data: User Status Information)
The presented virtual reality related data storage database stores user status history information in which the user status information transmitted from the virtual reality presentation device to the virtual reality service providing server is summarized in time series.
The virtual reality presentation system according to attachment 210 or 211.
(Supplementary Note 213) (Type of Virtual Reality Related Data: User Voice Data)
The presented virtual reality related data storage database stores user voice history information in which the user voice data input by the virtual reality presentation device is summarized in time series.
The virtual reality presentation system according to any one of appendices 210 to 212.
(Supplementary Note 214) (Type of Virtual Reality Related Data: Virtual Reality Object History)
The presented virtual reality related data storage database stores virtual reality object history information in which virtual reality object identification information for identifying at least the virtual reality object arranged in the virtual space is collected in time series.
The virtual reality presentation system according to appendix 209.
(Supplementary Note 215) (Type of Virtual Reality Related Data: Virtual Reality Object History)
The presented virtual reality related data storage database includes the virtual reality object history information in which the position coordinates of the virtual reality object in the virtual space and the information related to the behavior of the virtual reality object are further summarized in time series. Remember,
The virtual reality presentation system according to attachment 214.
(Supplementary Note 216) (Type of Virtual Reality Related Data: Control Program Execution History)
The presented virtual reality related data storage database stores behavior control program execution history information in which the execution history of a behavior control program for controlling the behavior of the virtual reality object arranged in the virtual space is summarized in time series. ,
The virtual reality presentation system according to appendix 209.
(Supplementary Note 217) (Type of Virtual Reality Related Data: Control Program Execution History)
The presented virtual reality related data storage database stores audio context history information in which the audio contexts extracted by the audio context extraction unit are summarized in time series.
The virtual reality presentation system according to attachment 216.
(Appendix 218) (Method)
A generation step of generating at least virtual reality presentation video data used to construct a virtual reality world related to a virtual reality service;
A presentation step of presenting the virtual reality world to the user by displaying the generated virtual reality presentation video data;
Presented virtual reality related data storage step for storing data related to construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user can be reproduced;
A virtual reality presentation method comprising:

  The present invention can be widely used to provide a virtual reality world with high creativity.

(About Embodiment 1-9)
DESCRIPTION OF SYMBOLS 100 Virtual reality presentation apparatus 110 User state detection part 111 Internal camera 112 Inclination sensor 113 Acceleration sensor 114 EEG sensor 115 Biosensor 120 Image | video display start determination processing part 130 Display control part 131 Image | video reading part 132 Image | video decoding process part 133 Image | video composition process Unit 134 layer order management unit 135 video correction processing unit 140 display unit 141a right display panel 141b left display panel 142 light source 143 light guide plate 144 LCD (liquid crystal display)
145 LCD (For external light transmission control)
146a Lens group 146b PLC
146c first reflecting mirror 146d R reflecting dichroic mirror 146e G reflecting dichroic mirror 146f second reflecting mirror 146g third reflecting mirror 146h first liquid crystal panel 146j cross dichroic prism 146k fourth reflecting mirror 146m second liquid crystal panel 146n third liquid crystal panel 146p Galvano mirror 146q Irradiation window 200 Virtual reality presentation device
210 voice input unit 211 user microphone 212 surrounding microphone 220 device side virtual reality presentation control unit 230 communication unit 231 wireless transmission unit 232 wireless reception unit 240 video data temporary storage unit 300 virtual reality presentation device
310 Ambient Condition Detection Unit 311 Eye Camera (Front Camera)
312 Peripheral camera 320 Video display start condition setting unit 400 Virtual reality presentation device
410 Virtual reality presentation data storage unit 420 Virtual reality presentation condition setting unit 430 Audio reproduction start determination processing unit 440 Audio reproduction control unit 441 Audio data reading unit 442 Audio decoding processing unit 450 Audio output unit 451a Right speaker 451b Left speaker
500 Virtual reality presentation device 510 Video display end determination processing unit 520 Virtual reality presentation end condition setting unit 530 Audio reproduction end determination processing unit 600 Virtual reality presentation device 700 Virtual reality presentation device 710 Video display condition setting unit 720 Video display determination processing unit 720
800 Virtual reality presentation device 900 Virtual reality presentation device 2000 Virtual reality presentation system 2010 Virtual reality presentation control unit 2020 Display video generation processing unit 2100 Virtual reality service providing server 2110 Communication unit 2120 Audio context acquisition unit 2130 Server-side virtual reality presentation control unit 2131 Virtual reality service provision availability determination unit 2132 Virtual reality service determination processing unit 2133 Virtual reality service control unit 2134 Notification information generation processing unit 2140 Display image generation processing unit 2141 View point control unit 2142 Light source control unit 2143 Drawing processing unit 2144 Video encoding processing unit 2200 Voice analysis server 2210 Voice analysis processing part 2220 Voice context extraction part 2300 Virtual reality content storage DB
3000 Virtual reality presentation system 3100 Virtual reality service providing server 3110 Image analysis processing unit 3111 Object position specifying unit 3112 Optical component extraction unit 3120 Virtual reality object reading unit 3130 Virtual space setting unit 3131 Virtual space generation processing unit 3132 Virtual space update management unit 3140 Virtual reality object placement unit 3200 Contractor information storage DB
4000 Virtual reality presentation system 4100 Virtual reality service providing server 4110 Virtual reality content reading unit 4120 Playback audio generation processing unit 4121 Auditory point control unit 4122 Audio distribution processing unit
4122a Virtual reality object arrangement position acquisition unit 4122b Relative positional relationship calculation unit 4122c Channel-specific audio data generation processing unit 4123 Audio encoding processing unit 5000 Virtual reality presentation system 5100 Virtual reality service providing server 5110 Virtual reality object behavior control unit 6000 Virtual reality presentation System 6100 Virtual reality service provision server 6110 Virtual reality presentation start determination processing unit 7000 Virtual reality presentation system 7100 Virtual reality service provision server 8000 Virtual reality presentation system 8100 Virtual reality service provision server 8110 Accompanying information generation processing unit (about embodiments 10 to 14) )
DESCRIPTION OF SYMBOLS 500 Virtual reality presentation apparatus 510 Communication part 511 Wireless transmission part 512 Wireless reception part 520 Virtual reality presentation control part 1000 Virtual reality presentation system 1010 User state detection part 1011 Inclination sensor 1012 Acceleration sensor 1013 Direction sensor 1014 Position sensor 1015 Internal camera 1016 EEG sensor 1017 Blood pressure sensor 1018 Pulse sensor 1019 Sweat sensor 1020 Presentable virtual reality calculation unit 1030 Video generation processing unit 1031 Drawing processing unit 1032 Video encoding processing unit 1040 Video display unit 1041 Video related data storage unit 1042 Video decoding processing unit 1043 Image correction processing unit 1044 Image composition processing unit 1045 Display area selection unit 1046 Display panel 1046-1 Light source 1046-2 Light guide plate 1046-3 LCD (for contrast) 146-4 LCD (For blocking external light)
1046-7a lens group 1046-7b PLC
1046-7c first reflection mirror
1046-7d R reflective dichroic mirror 1046-7e G reflective dichroic mirror
1046-7f Second reflecting mirror 1046-7g Third reflecting mirror 1046-7h First liquid crystal panel 1046-7j Cross dichroic prism 1046-7k Fourth reflecting mirror 1046-7m Second liquid crystal panel 1046-7n Third liquid crystal panel 1046 7p Galvano mirror 1046-7q Irradiation window 1050 Virtual reality content storage unit 1060 Presented virtual reality calculation unit 1070 Audio generation processing unit 1080 Audio output unit 1090 Virtual reality presentation data generation processing unit 1100 Virtual reality presentation unit 2000 Virtual reality presentation system 2010 Illusion Degree Calculation Unit 3000 Virtual Reality Presentation System 3010 Virtual Reality Presentation Control Unit 3020 Virtual Space Generation Processing Unit 3030 Virtual Reality Object Placement Processing Unit 3040 Viewpoint Control Unit 3050 Virtual Reality Service Request Unit 3060 Contractor Information Information storage unit 4000 Virtual reality presentation system 4010 Ambient condition detection unit 4011 Gaze camera (front camera) 4012 Peripheral camera 4020 Ambient condition identification unit 4030 Image analysis processing unit 4031 Object identification processing unit 4032 Optical component extraction unit 4033 Object identification processing unit 4034 Model Data generation processing unit 5000 Virtual reality service providing system 5010 Communication unit 5011 Reception unit 5012 Transmission unit 5020 Virtual reality service control unit 5030 Virtual reality object behavior control unit 5040 Layer order update processing unit 5050 Layer information generation processing unit (Embodiments 15 to 17) about)
10 communication network 20 PC
DESCRIPTION OF SYMBOLS 100 Virtual reality presentation apparatus 110 User state detection part 111 Inclination sensor 112 Acceleration sensor 120 Ambient condition detection part 121 Gaze camera 122 Peripheral camera 130 Audio | voice input part 131 User microphone 132 Peripheral microphone 140 Display control part 141 Video decoding process part 142 Video correction | amendment Processing unit 143 Video composition processing unit 144 Display area selection unit 150 Display unit 151a Right display panel 151b Left display panel 160 Audio reproduction control unit 170 Audio output unit 171a Right speaker 171b Left speaker 180 Virtual reality presentation control unit 190 Communication unit 200 Virtual reality Service providing server 210 Communication unit 220 Audio analysis processing unit 230 Image analysis processing unit 240 Virtual reality service providing server 250 Virtual reality object behavior control unit 260 Display video generation processing unit 26 Drawing processing unit 262 image decoding processing unit 270 reproducing the sound generation processing unit 300 subscriber information storage DB 400 virtual reality contents storage DB
500 Presented virtual reality related data storage DB
600 history information storage DB 700 virtual reality reproduction request processing server 800 virtual reality service providing server 810 virtual space generation processing unit 820 virtual reality object arrangement processing unit 830 viewpoint control unit 840 light source control unit 850 model data arrangement processing unit 860 layer order update processing Unit 870 layer information generation processing unit 900 virtual reality reproduction request processing server 910 communication unit 911 reproduction request receiving unit 912 reproduction virtual reality presentation data transmission unit 920 reproduction request processing unit 930 virtual reality related data reading unit

Claims (39)

User status detection means for detecting the user status;
Video generation means for generating video for virtual reality presentation based on the user's state;
Video display means for displaying the generated virtual reality presentation video;
A determination unit that determines whether a user state detected by the user state detection unit satisfies a predetermined criterion;
Video display control means for performing control to display the virtual reality presentation video on the video display means when it is determined that the criterion is satisfied;
A virtual reality presentation system comprising: Virtual reality object storage means for storing virtual reality objects;
Virtual space setting means for setting the virtual space;
Virtual reality object placement means for reading the virtual reality object from the virtual reality object storage means and placing it in the set virtual space;
Viewpoint control means for updating the position coordinates and the line-of-sight direction of the viewpoint in the set virtual space based on the state of the user detected by the user state detection means;
Further comprising
The video generation means renders the virtual reality object by drawing the virtual reality object arranged in the virtual space based on the position coordinates and the line-of-sight direction of the viewpoint updated by the viewpoint control means. Generate video for
The virtual reality presentation system according to claim 1. Ambient shooting means for shooting the surrounding scenery and obtaining the surrounding scenery image;
Optical component extraction means for extracting a predetermined optical component from the surrounding scenery image;
Light source control means for updating the position coordinates of the light source in the virtual space and the luminance or illuminance of the light source based on the extracted optical component;
Further comprising
The video generation means performs a shading process based on the position coordinates of the light source and the luminance or illuminance of the light source updated by the light source control means when the virtual reality object arranged in the virtual space is drawn. To generate the video for presenting the virtual reality,
The virtual reality presentation system according to claim 2. Modeling processing means for modeling each object in the real world reflected in the surrounding landscape image and generating a real object;
Model data placement means for placing the real object generated by the modeling processing means in the virtual space;
Further comprising
The video generation means hides the virtual reality object arranged by the virtual reality object arrangement means with the real object arranged by the model data arrangement means when the virtual reality object arranged in the virtual space is drawn. By performing processing, the virtual reality presentation video is generated.
The virtual reality presentation system according to claim 2 or 3. The virtual reality object arranging means arranges a plurality of virtual reality objects in different layers of the virtual space,
The video generation means generates a virtual reality presentation video for each layer by drawing a virtual reality object for each layer of the virtual space,
The display control means synthesizes the layered virtual reality presentation video, and performs control to display the synthesized video on the display means.
The virtual reality presentation system according to claim 4. The virtual reality object placement means places a newly placed virtual reality object in a new virtual space layer,
The video generation means generates a virtual reality presentation video for each layer by drawing a virtual reality object for each layer of the virtual space,
The determination means determines whether to start displaying the video of the new virtual reality object,
When the determination unit determines that the display of the video of the new virtual reality object is to be started, the display unit displays another virtual reality presentation video for each layer in which the new virtual reality object is arranged. Is combined with the virtual reality presentation video of each layer and performs control to display the composited video on the display means.
The virtual reality presentation system according to claim 5. User status detection means for detecting the user status;
Video display means for displaying video;
A determination unit that determines whether a user state detected by the user state detection unit satisfies a predetermined criterion;
Display that performs control to display on the video display means a video for virtual reality presentation that is controlled based on the user status detected by the user status detection means when it is determined that the standard is satisfied Control means;
A virtual reality presentation device. The user state detection means detects the state of the user's face by photographing a part or all of the user's face,
The determination means determines whether the user's eyes are closed for a predetermined time based on the video imaged by the user state detection means,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the user's eyes are closed for a predetermined time.
The virtual reality presentation apparatus according to claim 7. The determination means determines whether the user has closed eyes deeply based on the video taken by the user state detection means,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the user has closed eyes deeply;
The virtual reality presentation apparatus according to claim 8. The user state detection means further detects the user's brain waves,
The determination unit determines whether the user's brain wave value detected by the user state detection unit satisfies a predetermined criterion,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the value of the user's brain wave satisfies a predetermined criterion;
The virtual reality presentation apparatus according to any one of claims 7 to 9. The user state detection means detects the direction the user is facing,
The determination unit determines whether the direction of the user detected by the user state detection unit has changed by a predetermined reference angle or more,
The display control means starts displaying the virtual reality presentation video on the video display means when it is determined that the direction in which the user is facing has changed by a predetermined reference angle or more.
The virtual reality presentation apparatus according to claim 7. Transmitting means for transmitting information indicating the state of the user detected by the user state detecting means;
Receiving means for receiving the information indicating the predetermined reference and the virtual reality presentation video;
Further comprising setting means for setting the predetermined reference based on the received information;
The determination unit determines whether the user's state detected by the user state detection unit satisfies the predetermined criterion set by the setting unit;
The display control means is the receiving means when the determining means determines that the user's condition detected by the user condition detecting means satisfies the predetermined reference set by the setting means. Control to display the received virtual reality presentation video on the video display means;
The virtual reality presentation device according to claim 7. A user status detection step for detecting a user status;
A determination step of determining whether the detected state of the user satisfies a predetermined criterion;
A video generation step of generating a video for virtual reality presentation based on the detected state of the user;
A video display control step of performing control to display the virtual reality presentation video when it is determined that the criterion is satisfied;
Based on the control, a video display step for displaying the generated video for virtual reality presentation;
A virtual reality presentation method comprising: User status detection means for detecting the user status;
Presentable virtual reality calculation means for calculating the virtual reality of the present virtual reality world;
Video generation processing means for generating a video for virtual reality presentation so as to be within the virtual reality calculated by the presentable virtual reality calculation means based on the state of the user;
Display means for displaying the generated video;
A virtual reality presentation system comprising: A virtual reality calculation means for presenting that calculates the virtual reality of the virtual reality world currently being presented;
The presentable virtual reality calculation means is based on the virtual reality of the present virtual reality world calculated by the presenting virtual reality calculation means. Calculate the degree,
The virtual reality presentation system according to claim 14. An illusion degree calculating means for calculating the illusion degree of the user is provided,
The presentable virtual reality calculation unit is configured to calculate the illusion degree calculated by the illusion degree calculation unit based on the virtual reality degree of the currently presented virtual reality calculated by the presenting virtual reality calculation unit. Calculate the virtual reality of the virtual reality world that can be presented by adding, subtracting, multiplying, or dividing the value determined based on
The virtual reality presentation system according to claim 15. The user state detection means includes an electroencephalogram sensor that detects a user's brain wave, an internal camera that captures part or all of the user's face, a blood pressure sensor that measures the user's blood pressure, a pulse sensor that measures the user's heart rate, and the user Comprising at least one perspiration amount sensor for detecting the perspiration amount of
The illusion degree calculating means calculates the illusion degree of the user based on a detection result in any one of the sensors provided in the user state detecting means.
The virtual reality presentation system according to claim 16. Virtual reality content storage means for storing at least a plurality of virtual reality objects;
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object in the virtual space;
Viewpoint control means for controlling viewpoint coordinates and line-of-sight direction in the virtual space based on the user state;
Drawing processing means for drawing the virtual reality object arranged in the virtual space based on the viewpoint coordinates and the line-of-sight direction updated by the viewpoint control means;
With
The virtual reality object arranging means arranges the virtual reality object in the virtual space so as to be within the virtual reality calculated by the presentable virtual reality calculating means;
The virtual reality presentation system according to any one of claims 14 to 17. The virtual reality object placement unit is configured to display a virtual reality object being placed in a virtual space when the virtual reality of the presentable virtual reality world is lower than the virtual reality of the virtual reality world currently being presented. By removing the virtual reality from the virtual space so that the virtual reality of the virtual reality world to be presented falls within the virtual reality of the virtual reality world that can be presented.
The virtual reality presentation system according to claim 18. The virtual reality content storage means stores the virtual reality object in association with the virtual reality set for each of the virtual reality objects;
The virtual reality object placement unit reads the virtual reality object associated with a virtual reality lower than the virtual reality calculated by the presentable virtual reality calculation unit from the virtual reality content storage unit and reads the virtual reality object Place it in virtual space,
The virtual reality presentation system according to claim 19. The virtual reality object placement unit obtains an excess virtual reality obtained by subtracting the virtual reality calculated by the presenting virtual reality calculating unit from the virtual reality calculated by the presentable virtual reality calculating unit. The virtual reality object associated with a lower virtual reality degree is read from the virtual reality content storage means and arranged in the virtual space;
The virtual reality presentation system according to claim 20. Ambient condition detection means for detecting the situation of the user's surroundings,
Ambient situation specifying means for specifying a situation around the user based on a detection result in the surrounding situation detection means,
With
The virtual reality content storage means stores a correspondence relationship between the virtual reality object and virtual reality for each surrounding situation,
The virtual reality object placement means is a virtual reality degree calculation means that provides a virtual reality degree corresponding to a situation around the user specified by the surrounding situation specifying means of the virtual reality object placed in the virtual space. The virtual reality object is selected and read from the virtual reality object storage means so as to be lower than the calculated virtual reality, and is arranged in the virtual space.
The virtual reality presentation system according to any one of claims 18 to 21. The ambient condition detection means captures an ambient scene image by photographing the situation of the user,
Light source control means for updating the position coordinates of the light source and the illuminance or luminance of the light source based on the surrounding scenery image;
Model data generating means for generating model data of an object included in the surrounding scenery image;
Further comprising
The drawing processing unit projects a virtual reality object arranged in the virtual space to generate a two-dimensional image based on the viewpoint coordinates and the line-of-sight direction updated by the viewpoint control unit; A shading process for calculating the brightness of the virtual reality object based on the position coordinates of the light source updated by the light source control unit and the illuminance or brightness of the light source; and a hidden surface process for hiding a part of the virtual reality object based on the model data; The virtual reality presentation video is generated by executing at least
The virtual reality presentation system according to any one of claims 18 to 22. A user status detection step for detecting a user status;
A presentable virtual reality calculation step for calculating a virtual reality of a virtual reality world that can be currently presented;
Based on the state of the user, a video generation step of generating a video for virtual reality presentation so as to fit within the calculated virtual reality degree;
A display step for displaying the generated video;
A virtual reality presentation method comprising: A virtual reality presentation device that presents a virtual reality world related to a virtual reality service to a user;
A virtual reality service providing server that generates at least virtual reality presentation video data to be used to construct the virtual reality world and transmits the virtual reality presentation video data to the virtual reality presentation device;
A presented virtual reality-related data storage database that stores data related to construction of the virtual reality world as virtual reality-related data in a state in which the virtual reality presentation device can reproduce the virtual reality world presented to the user; ,
A virtual reality presentation system comprising: A virtual reality content storage database for storing virtual reality content used to construct the virtual reality world including at least the virtual reality object related to the virtual reality service;
The virtual reality service providing server includes:
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object related to the virtual reality service in the virtual space;
Video generation means for sequentially generating the virtual reality presentation video data by rendering the virtual reality object arranged in the virtual space in real time;
Comprising at least
The virtual reality presentation device presents the virtual reality world to the user by displaying a virtual reality presentation video including the virtual reality object based on the generated virtual reality presentation video data.
The virtual reality presentation system according to claim. The virtual reality presentation device includes:
User status detection means for detecting the user status;
Wireless transmission means for wirelessly transmitting user status information indicating the detected user status;
Wireless receiving means for wirelessly receiving the virtual reality presentation video data;
Display means for displaying the virtual reality presentation video based on the wirelessly received virtual reality presentation video data;
Comprising
The virtual reality service providing server includes:
Receiving means for receiving the user status information;
Transmission means for transmitting the virtual reality presentation video data sequentially generated by the video generation means;
Further comprising
The video generation means renders the virtual reality presentation video in real time by drawing the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction specified based on the user state information. Generate data sequentially,
The virtual reality presentation system according to claim 26. The virtual reality content storage database stores audio data related to the virtual reality service as the virtual reality content,
The virtual reality service providing server further includes voice generation processing means for generating virtual reality presentation voice data based on voice data related to the virtual reality service stored in the virtual reality content storage database,
The virtual reality presentation device further includes voice output means for converting the virtual reality presentation voice data into an analog electrical signal and outputting the virtual reality presentation voice to the outside.
The virtual reality presentation system according to claim 27. The virtual reality presentation device includes:
Voice input means for collecting user voice data by collecting voice uttered by the user;
Image photographing means for photographing at least the front direction of the user and acquiring a front scenery image;
Further comprising
The virtual reality service providing server includes:
Voice context extraction means for extracting a voice context included in the user voice data;
Light source control means for extracting optical information relating to illuminance or luminance from the front scenery image, and updating the position coordinates of the light source and the illuminance or luminance of the light source;
Viewpoint control means for updating the viewpoint position coordinates and the line-of-sight direction based on the user state information;
Further comprising
The video generation unit is configured to store the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit and the light source position coordinates and the illuminance or luminance of the light source updated by the light source control unit. Sequentially generating the virtual reality presentation video data by rendering the arranged virtual reality objects in real time;
The virtual reality presentation system according to claim 28. The virtual reality content storage database further stores a virtual reality object behavior control program for controlling the behavior of the virtual reality object as the virtual reality content,
The virtual reality service providing server includes:
A virtual reality object behavior control means for controlling the behavior of the virtual reality object placed in the virtual space by executing the virtual reality object behavior control program read from the virtual reality content storage database;
The virtual reality presentation system according to claim 29. The presented virtual reality related data storage database stores the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server to the virtual reality presentation device.
The virtual reality presentation system according to claim 30. The presented virtual reality related data storage database stores the front scene image transmitted from the virtual reality presentation device to a virtual reality service providing server,
The virtual reality presentation system according to claim 31. The presented virtual reality related data storage database stores user status history information in which the user status information transmitted from the virtual reality presentation device to the virtual reality service providing server is summarized in time series.
The virtual reality presentation system according to claim 31 or 32. The presented virtual reality related data storage database stores user voice history information in which the user voice data input by the virtual reality presentation device is summarized in time series.
The virtual reality presentation system according to any one of claims 31 to 33. The presented virtual reality related data storage database stores virtual reality object history information in which virtual reality object identification information for identifying at least the virtual reality object arranged in the virtual space is collected in time series.
The virtual reality presentation system according to claim 30. The presented virtual reality related data storage database includes the virtual reality object history information in which the position coordinates of the virtual reality object in the virtual space and the information related to the behavior of the virtual reality object are further summarized in time series. Remember,
36. The virtual reality presentation system according to claim 35. (Type of virtual reality related data: control program execution history)
The presented virtual reality related data storage database stores access history information in which at least the access history from the virtual reality service providing server for the virtual reality content storage database is summarized in time series.
The virtual reality presentation system according to claim 30. The presented virtual reality related data storage database further stores an execution history of the control program read from the virtual reality content storage database.
The virtual reality presentation system according to claim 37. A generation step of generating at least virtual reality presentation video data used to construct a virtual reality world related to a virtual reality service;
A presentation step of presenting the virtual reality world to the user by displaying the generated virtual reality presentation video data;
Presented virtual reality related data storage step for storing data related to construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user can be reproduced;
A virtual reality presentation method comprising: