JP2014187559A - Virtual reality presentation system and virtual reality presentation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a virtual reality presentation system which ensures that a user accepts the fact that a world of virtual reality experienced in the past was not a real world.SOLUTION: The virtual reality presentation system comprises: virtual reality presentation apparatuses for presenting to a user the world of virtual reality relating to a virtual reality service; a virtual reality service providing server which generates at least image data for presenting virtual reality to be used to construct the world of virtual reality and transmits it to the virtual reality presentation apparatus; and a presented virtual reality related data storage database which stores, as virtual reality related data, data relating to the construction of the world of virtual reality, while the world of virtual reality presented to the user by a virtual reality presentation apparatus can be reproduced.

Description

  The present invention relates to a virtual reality presentation system, a virtual reality presentation device, a virtual reality presentation method, a virtual reality service providing system, a virtual reality service providing server, a virtual reality service providing method, an information processing system, an information processing server, an information processing method, and information. The present invention relates to a processing program and a head mounted display device.

In recent years, various electronic devices have been developed as devices for displaying images in addition to conventional stationary displays such as televisions, and are referred to as mobile phone terminals, tablet terminals, and head mounted displays (HMDs). Electronic devices have already been developed.

  Prior art document 1 discloses a virtual reality presentation device for displaying CG (Computer Graphics) on a real landscape in an HMD.

JP 2006-338163 A

A human being is a living organism that senses events around itself by the five senses, that is, taste, touch, sight, hearing, and smell, and processes the sensed information in the brain to perform future actions. Therefore, by deceiving these human senses using the virtual reality presentation device and causing an illusion, the user can be guided from the real world to the virtual reality world.

However, the narrower the boundary between the artificially created virtual reality world and the real world, the more users have experienced before returning to the real world from the virtual reality world. You will not be able to recognize that the world is a virtual reality world.

For example, a virtual reality world that can interact with the idol X by placing the idol X that the user likes in the virtual space and controlling the behavior of the idol X in the virtual space while being linked to the real world is presented. Let's consider the case.

  The service provides an opportunity to contact the user by placing an idol X in the virtual reality world where the user cannot meet or speak in the real world. Therefore, the virtual reality service providing system basically controls the behavior of the idle X arranged in the virtual space so that the user is pleased. For example, the user can spend time like a dream by taking a behavior that the idol X in the virtual space confesses to the user.

As a matter of course, the idol is a virtual idol X arranged in the virtual space, and the idol X in the real world does not confess to this user. However, a user who enters the highly controlled virtual reality world does not doubt that the real idol X has confessed to him.

Later, in the real world, a friend said, “I was confessed by that idol X and I was going to get along.” In addition, there is a possibility that a situation may occur in which the user pushes against the actual office of the idol X who has confessed but never came to see him. When the user asked, “You told me you would go out at that time!”, The confession to the user was the idol X created in the virtual reality world, not the real idol X. It's clear that I don't know what you're talking about.

The virtual reality presentation device has the greatest significance in attracting users by creating an event in the virtual reality world that cannot be realized by physical or social constraints in the real world. Therefore, the feelings of users entering the virtual reality world created as a wonderful world than the real world are uplifting, and even after returning to the real world, the world that was perceived by the user is virtual There may be cases where you refuse to accept the fact that it was reality. In the above case, since the cruel reality gap that the best experience that Idol X that I had admired confessed was virtual was too large, even if a third party explains that, the user May reject psychologically.

In this way, in order to make the virtual reality world presented to the user even more wonderful, it is necessary to develop technology in a direction that eliminates the boundary with the real world, while the virtual reality world and the real It is said that the more the boundary with the world of the world disappears, the more the gap between the wonderful world of virtual reality and the real world cannot be psychologically accepted, and there is a possibility that it will hinder the life of the real world after that. There was a problem of antinomy.

In view of the above problems, the present invention provides a virtual reality presentation system that ensures that the virtual reality world experienced in the past is accepted when the user cannot psychologically accept the fact that the virtual reality world is not the real world, and This is a virtual reality presentation method.

A virtual reality service providing system according to one aspect of the present invention includes a virtual reality presentation device that presents a virtual reality world related to a virtual reality service to a user, and virtual reality presentation video data used for constructing the virtual reality world. A virtual reality service providing server that generates and transmits at least the virtual reality presenting device, and constructs the virtual reality world in a state where the virtual reality world presented to the user by the virtual reality presenting device is reproducible A presented virtual reality-related data storage database that stores data related to the above as virtual reality-related data.

The virtual reality presentation method according to an aspect of the present invention includes a generation step of generating at least virtual reality presentation video data used for construction of a virtual reality world related to a virtual reality service, and the generated virtual reality presentation video. A presentation step of presenting the virtual reality world to the user by displaying data, and data related to the construction of the virtual reality world in a state where the virtual reality world presented to the user can be reproduced. A stored virtual reality related data storing step for storing as related data.

According to the present invention, after experiencing virtual reality, even if an error occurs in the boundary between virtual reality and reality, the user was watching at that time based on the virtual reality related data stored in the database. By explicitly presenting the virtual reality world to the user again, it is possible to accept the fact that the world believed to be the reality was the virtual reality world.

FIG. 3 is a conceptual diagram schematically showing an overall configuration of a person / organization related to the virtual reality service according to the first embodiment. It is a figure which shows an example of the user registration information which a virtual reality service provision company manages. 1 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 1. FIG. It is a figure explaining the difference between the real world and the virtual reality world. It is a figure explaining the structure of virtual reality content. It is an example of the management file which manages a virtual reality object. It is an example of model data of a virtual reality object. It is a figure explaining the classification | category of a virtual reality object control program. 6 is a block diagram showing a configuration of a modified example of the virtual reality presentation system according to Embodiment 1. FIG. It is an example of virtual reality service history information. It is an example of virtual reality history information. It is an external appearance perspective view from the diagonally forward and diagonally rear of the virtual reality presentation device according to the second embodiment. . FIG. 10 is a cross-sectional view of the virtual reality presentation device according to Embodiment 2 near a display panel. 6 is a block diagram showing a configuration of a virtual reality presentation system according to Embodiment 2. FIG. FIG. 10 is a block diagram illustrating a configuration of a virtual reality presentation system according to Embodiment 3. It is an example of virtual reality service provision start notification information. It is an example of the image | video for virtual reality presentation according to layer produced | generated for every layer. It is a figure explaining the change of a layer order. It is a figure explaining the video for virtual reality presentation for every layer in which the display position of the virtual reality object was changed before and after video correction processing. It is a figure which shows the combination of the virtual reality related data memorize | stored in the database. It is a block diagram which shows the whole virtual reality presentation system structure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the following description, parts denoted by the same reference numerals have substantially the same functions. For the sake of clarification of the invention, explanations of overlapping parts are omitted as appropriate.

<Embodiment 1>
FIG. 1 is a conceptual diagram showing the relationship between persons and organizations related to the virtual reality service according to the first embodiment of the present invention. A virtual reality service use contract is made in advance between a virtual reality service provider that provides the virtual reality service and a user who uses (uses) the virtual reality service. When using a virtual reality service, a user concludes a virtual reality service usage contract with a virtual reality service provider, and pays a fee on a flat rate basis or a pay-as-you-go basis as a price associated with the use of the service.

Here, virtual reality (VR) is computer graphics (CG: Computer).
A virtual reality service is a service that allows the user to experience such a virtual reality world. Humans sense events that occur around them using the five senses, and process the sensed information in the brain to grasp the situation. Therefore, a virtual reality world can be created by displaying images and outputting sounds so as to deceive human senses. Since the virtual reality world is a world that is artificially created by displaying video for virtual reality presentation or outputting audio for virtual reality presentation, there are various forms of virtual reality services.

FIG. 2 is an example of contractor information (user information) managed by the virtual reality service provider. The virtual reality service provider provides a contract that summarizes the personal information (contractor's personal information) of the user who has signed a virtual reality service use contract, the plan that the user has contracted, and the service content included in the plan. Information is stored and managed in a database. Further, the contractor information includes a user ID for uniquely identifying a user, a connection password used for user authentication, and the like.

The virtual reality service provider prepares a plurality of plans in advance, and the fee and the contents of the virtual reality service differ for each plan. For example, in FIG. 2, the user A has a contract for a plan with the content “service that can talk to the idol X”.

The service content of this plan is not a service that says that you can talk to the idol X in the real world, but this idol X is created in the virtual reality world and created in the virtual reality world. In addition, it is a service that can talk to the idol X and the virtual reality world.

Naturally, it is difficult for the idol X in the real world to talk to the user in the real world due to time and physical constraints. However, there are many fans who want to experience the conversation with the idol X. Therefore, the virtual reality service provider creates a virtual reality world in which the idol X appears, and in the state where the user has a sense of reality in the virtual reality world, the idol X and the user in the virtual reality world Provide an opportunity to talk.

In order to provide such a service, it is necessary to create an idle X virtual reality object. The virtual reality object will be described later. Since the idol X has a portrait right, it is illegal for the virtual reality service provider to create and use the virtual reality object of the idol X without permission. Accordingly, the virtual reality service providing company concludes a person data use permission contract with the idol X directly or with the office to which the idol X belongs. After signing the contract, the virtual reality service provider uses its own or external virtual reality content production company to create data that constitutes the virtual reality object of the idle X, and stores it in the database in a usable state. Store it.

User B has a contract for a service content plan “Teacher Service in the Virtual Space World” provided by a virtual reality service provider. The service included in the plan is a service in which a virtual reality world in which a historical person appears is created, and a class is given to the user in a field in which the person left a feat in the historical person.

Because historical figures appearing in the virtual reality world take classes, users can take classes in a more uplifted state compared to teachers in the real world. It becomes possible to learn with higher learning efficiency.

In addition, the user C contracts with a service content plan provided by a virtual reality service provider, such as “create a virtual space world decorated with vases”. The service included in the plan is a service that creates a virtual reality world in which a vase is placed at a predetermined position when the user is at home. Users who do not have time to care for flowers, etc. receive the service, so users can see the scenery where the vase is placed on the shelf, so they do not have to buy flowers in the real world The experience of spending time in a colorful room is also possible.

Further, the user D is contracted with a service content plan such as “Keep pets” provided by the virtual reality service provider. The service included in the plan is a service that is said to create a virtual reality world where a pet is present when the user is at home. When a user who cannot actually keep a pet due to housing rules, care time, etc. receives this service, the user can see the pet moving around in the room, so the pet in the real world Even if you can't keep a pet, you can experience a space with pets. The pet is not a real pet, but a pet created in a virtual reality world. Therefore, the experience of keeping fictional animals such as lions, dinosaurs and other unicorns as pets becomes possible.

FIG. 3 is a block diagram illustrating a configuration of the virtual reality presentation system 1000 according to the first embodiment. The virtual reality presentation system 1000 includes a virtual reality presentation device 100, a virtual reality service providing server 200, a contractor information database 300, a virtual reality content storage unit 400, and a provided virtual reality related data storage database 500. . In the present specification, the virtual reality presentation device 100 will be described as a head mounted display (HMD) device that a user wears on the head. The virtual reality service providing server 200 is connected to a virtual reality presentation device 100 (100a to 100c) worn by a user via a communication network 10 such as the Internet.

The virtual reality presentation device 100 is a head mounted display device worn by a user on the head, and includes at least a display panel that displays a video of a virtual reality object, and a communication unit that communicates with the virtual reality service providing server 200. Prepare. The virtual reality presentation device 100 displays the video including the virtual reality object on the display panel based on the virtual reality presentation video data transmitted from the virtual reality service providing server 200, thereby allowing the user to see the virtual reality world. Present.

FIG. 4 shows a real world where a user wearing the virtual reality presentation device 100 is present, a virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100, and a virtual reality presented to the user. It shows the world, that is, the virtual reality world perceived by the user.

The user is actually standing in a room without a dog and looking at the front sofa. Here, a video of a dog that is a virtual reality object is displayed on the display panel of the virtual reality presentation device 100. The user can see a virtual reality world where a dog sitting on the sofa is a fusion of the scenery of the sofa, which is the real world, and the image of the dog displayed on the display panel.

This dog can only be seen by the user wearing the virtual reality presentation device 100. Thus, while others live in a real world without dogs, this user lives in a virtual reality world created with dogs.

Note that the virtual reality presentation device 100 has an audio reproduction function, and reproduces the dog's cry as the virtual reality object, thereby stimulating vision and hearing and presenting a virtual reality world to the user. Is preferable because it can enhance a sense of reality in the virtual reality world. In this case, in addition to the virtual reality presentation video data including the virtual reality object, virtual reality presentation audio data that is audio data for forming the virtual reality world is transmitted from the virtual reality service providing server 200.

The virtual reality service providing server 200 provides a virtual reality service by displaying a video of a virtual reality object on the display panel of the virtual reality presentation device 100. The virtual reality service providing server 200 may draw the virtual reality object to create virtual reality presentation video data, and transmit the virtual reality presentation video data to the virtual reality presentation device 100. The virtual reality object model data may be transmitted to the virtual reality presentation device 100, and the virtual reality presentation device 100 may draw the model data and display the video of the virtual reality object on the display panel.

The virtual reality content storage database 400 is a database that stores virtual reality content. Here, the virtual reality content is a general term for data used for creating a virtual reality world, and includes at least a virtual reality object (data) and a virtual reality object control program. FIG. 5 shows the structure of the virtual reality content.

The virtual reality object refers to a virtual person or a building that is arranged in the virtual reality world. Virtual reality objects are classified into human objects, animal objects, non-animal objects, and other objects according to the types of objects arranged in the virtual reality world. Each virtual reality object is assigned an object ID (virtual reality object identification number) for uniquely identifying the virtual reality object.

A person object is an object such as a real person, an imaginary / imaginary person, a historical person, a creative person, etc., and is at least operable, and preferably speaks (outputs sound). It has the feature that it is possible.

The animal object is an object such as a real animal, an imaginary / fantasy animal, or a creative animal, and has a feature that it can operate. In addition, a part of the animal object can be configured to speak (output a sound). Since it is a virtual animal placed in the virtual reality world, it is possible to adopt a configuration that speaks like a human.

Non-animal objects are objects such as plants, furniture, buildings, and the like that basically do not move. However, the plant may be configured to partially move in conjunction with the wind blowing in the real world.

The other objects are, for example, video effects, sound effects, and the like, which are objects for giving reality to the virtual reality world. Other objects may be used alone to create a virtual reality world, or may be used in combination with a person object, an animal object, or the like.

Various virtual reality objects are composed of model data, basic audio data, and the like. The model data includes polygon data that defines the appearance shape of the virtual reality object, texture data that is pasted on the polygon surface, and the like.

FIG. 6 is an example of a management file for managing virtual reality objects. Each virtual reality object is assigned an identification number that uniquely identifies it, and is classified according to a plurality of types. For example, type 1 indicates a rough division of the virtual reality object, type 2 indicates whether it is a real thing (person) or an imaginary thing (person), and type 3 is included in the object This shows the gender of the person. For example, video data of a sleeping female lion is stored in the storage address xx010110101 of the management number No. 000012.

FIG. 7 shows an example of model data of an animal object having an object ID number: No000009 which is a kind of virtual reality object. The model video data includes 3D polygon data that defines the appearance shape of the virtual reality object, and texture data that is pasted on the surface of the shape formed by the polygon data. Moreover, it is more preferable that the configuration includes joint point data, skeleton data, and the like for operating the virtual reality object. The joint points are connected by a skeleton, and each joint point is connected to several polygon vertices. The position coordinates and angles of the joint points can be controlled, and the appearance shape of the virtual reality object changes by changing the joint points.

The sound basic data constituting the virtual reality object is data serving as the basis of the sound emitted by the virtual reality object. For example, when the virtual reality object is a person object created based on a real person, the data is voice quality, pitch, voice waveform, phoneme, etc. extracted from the real voice of the real person. These voice basic data are synthesized by voice synthesis processing and presented to the user, so that the user senses that the person object is actually speaking. Note that the basic voice data extraction method (recording method) can be extracted at various angles, and may be extracted in the form of voice pronunciation data and voice expression data, or other classification methods. It may be extracted with. The extraction method is determined by a speech synthesis algorithm to be performed.

The virtual reality object control program included in the virtual reality content is a program related to the control of the virtual reality object described above, and is classified into a plurality of levels of programs as shown in FIG.

The main control program controls the overall flow of the virtual reality service. The virtual reality service is provided by reading out and executing a main control program corresponding to the contract contents of the user when providing the virtual reality service corresponds to the service contents of the virtual reality service shown in FIG. To start.

The sub-control program is appropriately read from the main control program and controls selection, arrangement position, arrangement configuration, and the like of the virtual reality object arranged in the virtual reality world.

The object behavior control program controls the behavior of the objects read by the sub-control program and arranged in the virtual reality world. For example, when the object is a person object, an object behavior control program corresponding to each operation such as raising an arm, changing a facial expression such as laughing or crying, or jumping is prepared. The sub-control program selects an appropriate object behavior control program according to the situation of the virtual reality world, and when the object behavior control program is executed, the virtual reality object placed in the virtual reality world becomes The operation corresponding to the selected object behavior control program is performed.

As an example, it is assumed that the user has made a contract of a plan called “Idle X and 1 hour talk course”. When the virtual reality service providing server 200 determines to start providing the virtual reality service to the user, the virtual reality service providing server 200 reads the main control program corresponding to the plan with which the user has a contract from the virtual reality content storage database 400 and executes it. To do. Thus, the main control program is a program that controls the overall flow of the virtual reality service to be provided.

A plurality of related sub-control programs are read out by the main control program, and a virtual reality object is selected by executing a sub-control program related to virtual reality object selection. Here, the virtual reality object of idle X is selected and read from the virtual reality content storage database 400.

The sub-control program determines an initial arrangement position and an arrangement angle in the virtual space of the idle X that is a virtual reality object read from the virtual reality content storage database 400, and arranges the idle X in the virtual space. To do. The virtual reality service providing server 200 draws an idle X that is a virtual reality object arranged in the virtual space by the sub-control program, generates two-dimensional video data, performs a predetermined encoding process, and then performs virtual processing. It transmits to the reality presentation device 100. The virtual reality presentation device 100 decodes the encoded video data and displays the video on the display panel, whereby the video of idle X is displayed on the display panel.

The sub control program appropriately reads and executes an object behavior control program for controlling the behavior of the virtual reality object based on the state of the user wearing the virtual reality presentation device 100 and the like. By executing the read object behavior control program, the virtual reality object operates, and a video of the virtual reality object whose operation is controlled by the object behavior control program is displayed on the display panel of the virtual reality presentation device 100. Is displayed.

In the sub control program, a program related to the personality of the virtual reality object is provided. The sub-control program related to the personality may be particularly referred to as a personality control program. Different personality control programs are read based on feedback from the virtual reality presentation device 100, the virtual reality object behavior control program is selected according to the personality control program, and the selected virtual reality object behavior control program is executed. As a result, the virtual reality object operates.

For example, as the personality control program for the virtual reality object of the above-mentioned idle X, “normal state 1 to 10”, “anger state 1 to 5”, “joy state 1 to 5”, “sad state 1 to 5”, “fun” It is assumed that 33 types of personality control programs of “states 1 to 5” and “tsundere states 1 to 3” are stored in the virtual reality content storage database 400 as virtual reality content.

It is assumed that the personality control program of “normal state 1” is selected as the personality control program for controlling the personality of the idol X that is the current virtual reality object. In this state, when the user calls “X, good morning!” Via the virtual reality presentation device 100, the personality control program reads out and executes the corresponding virtual reality object behavior program, so that the virtual reality world is read. The idol X in the laughing smiles and utters a voice saying "Good morning."

On the other hand, when the personality control program in the tsundere state 2 is selected, when the user calls “X, good morning!” Via the virtual reality presentation device 100, the personality control program in the tsundere state 2 A different virtual reality object behavior control program is read out, and shows a behavior that emits a voice saying “Oh, I don't greet you!

Thus, the response to the input is determined by the personality control program, and the behavior of the virtual reality object is controlled by the virtual reality object behavior control program. The user wearing the virtual reality presentation device 100 is presented with video and audio of the virtual reality object for which these controls are performed.

The presented virtual reality related data storage database 500 stores data relating to the construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user by the virtual reality presentation device 100 can be reproduced. . More specifically, the presented virtual reality related data storage database 500 stores data related to the video in a state where the video of the virtual reality object displayed on the display panel of the virtual reality presentation device 100 can be reproduced. . The presented virtual reality-related data storage database 500 stores the video displayed on the display panel as data related to the video. Data encoded by a moving image encoding method such as H.264 may be stored. The presented virtual reality related data storage database 500 may store the execution history of the virtual reality object control program as data related to the video. This is because the virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100 can be reproduced by executing the virtual reality object control program again according to the execution history.

As described above, the virtual reality presentation system 1000 according to the first embodiment is used to construct the virtual reality presentation apparatus 100 that presents the virtual reality world related to the virtual reality service to the user and the virtual reality world. A virtual reality service providing server 200 that generates at least virtual reality presentation video data and transmits it to the virtual reality presentation device 100, and the virtual reality world that can be reproduced by the virtual reality presentation device 100 in a state where the virtual reality world can be reproduced. It includes at least a presented virtual reality related data storage database 500 that stores data relating to the construction of the real world as virtual reality related data.

With this configuration, users who have returned to the real world after experiencing the virtual reality world cannot recognize the boundary between the real world and the virtual reality world, When it is not possible to recognize that it is not the real world but the virtual reality world, the user can be appropriately returned to the real world by reproducing the virtual reality world experienced by the user again.

More specifically, in the first embodiment, the virtual reality presentation device 100 takes the form of a head-mounted display device worn by the user on the head. The head-mounted display device includes at least a display panel that displays an image, and allows the user to be drawn into the virtual reality world by deceiving the user's vision. The virtual reality service providing system according to the first embodiment is displayed on the virtual reality service providing server 200 that displays a video of a virtual reality object on the display panel of the head mounted display device 100 and the display panel of the head mounted display device 100. And a provided virtual reality related data storage database 500 that stores data related to the video in a state where the video of the virtual reality object can be reproduced.

According to this configuration, after experiencing virtual reality, even if there is an error in the boundary between virtual reality and reality, the virtual reality that the user was viewing at that time based on the related data stored in the database By explicitly presenting the object video to the user again, it is possible to accept the fact that the world believed to be real was the world of virtual reality.

It is more preferable that the virtual reality presentation system 1000 includes a history information storage server 600 and a virtual reality reproduction processing server 700 as shown in FIG.

The history information storage server 600 stores the virtual reality service used by the user, that is, the history of the virtual reality service provided to the user, as virtual reality service history information. Here, the virtual reality service history information stored in the history information storage server 600 is a so-called log.

The virtual reality content storage database 400 stores virtual reality content related to a plurality of types of virtual reality services, and the virtual reality service providing server 200 stores various data related to virtual reality services provided from the virtual reality content storage database 400. Are read out and subjected to predetermined processing to sequentially generate video data and audio data for virtual reality presentation. The history information storage server 600 summarizes the correspondence between the identification information (user ID) for identifying the user and the virtual reality service identification information for identifying the virtual reality service provided to the user by the virtual reality service providing server 200 in time series. The provided virtual reality service history information is stored.

FIG. 10 shows an example of virtual reality service history information stored in the history information storage server 600. The virtual reality service history information includes user identification information for identifying a user who has been provided with the virtual reality service, virtual reality service identification information for identifying a virtual reality service provided to the user, and the virtual reality service. The date and time information indicating the date and time is grouped in a time series in a state of being associated with each other.

In the virtual reality service providing server 200, user identification information for identifying a user who provides a virtual reality picture service and virtual reality service identification information for identifying a virtual reality service provided to the user are associated with each other in time series. A virtual reality service history information update processing unit for generating and updating the recorded virtual reality service history information is provided.

Only the virtual reality service history information stored in the history information storage server 600 cannot reproduce the virtual reality world presented to the user. This is because there are a plurality of virtual reality objects related to the virtual reality service, and it is impossible to specify which type of virtual reality object is used to build the virtual reality world.

The virtual reality reproduction processing server 700 receives a history disclosure request from the normal personal computer 20 or the virtual reality presentation device 100 via the communication network 10 such as the Internet. The history disclosure request is a packet for requesting disclosure of a virtual reality service log (usage history) used so far by the user, and includes at least a user ID and a password for authentication.

When the history disclosure request is received, the virtual reality reproduction processing server 700 matches the user ID and password pair stored in the contractor information storage database 300 with the user ID and password pair included in the history disclosure request. Perform authentication processing to see if you are doing it.

When the authentication is successful, the virtual reality reproduction processing server 700 reads the virtual reality service history information from the history information storage server 600 and returns it to the transmission source terminal of the history disclosure request. When the presented virtual reality reproduction request including information designating the provided virtual reality service related to the virtual reality world requesting the reproduction is received after the reply, the virtual current reproduction processing server 700 is designated by the request. The virtual reality related data related to the provided virtual reality service is read from the presented virtual reality related data storage database 500, and the virtual reality presentation video displayed on the display panel of the virtual reality presentation device 100 is reproduced to obtain the request source terminal. Send to.

Since the user can directly check the video that he / she saw at that time through a terminal such as the computer 20, he / she has to recognize that the world he / she has experienced is a virtual reality world. There will be no. Therefore, it is possible to recognize that it was a virtual reality more reliably than a case where a third party simply persuades the user.

Among virtual reality services, there is a type of virtual reality service that has story characteristics and allows one virtual reality service to be experienced multiple times. Therefore, it may not be sufficient to specify the virtual reality service that has been provided in order to specify the virtual reality world that requires reproduction. Must be specified.

Therefore, it is further preferable that the history information storage server 600 is configured to store virtual reality history information in which the virtual reality world presented to the user is summarized in time series. In this case, the virtual reality service providing server 200 includes a virtual reality identification number allocation unit (virtual reality identification information allocation unit) that allocates a virtual reality identification number (virtual reality identification information) that identifies the virtual reality world presented to the user. To do.

When providing a virtual reality service to a user, the virtual reality service providing server 200 assigns a virtual reality identification number that uniquely identifies the virtual reality world to the virtual reality world related to the virtual reality service. That is, each virtual reality service that can be provided by the virtual reality service providing server 200 is identified by the virtual reality service identification information uniquely assigned to the virtual reality service, and is presented to the user when the virtual reality service is provided. For each virtual reality world, a virtual reality identification number for identifying the virtual reality world is assigned by the virtual reality identification number assigning unit.

Each virtual reality service can be provided to each user who wants to use the virtual reality service, and the same virtual reality service can be used multiple times. On the other hand, even if they are the same virtual reality service, the virtual reality world related to the virtual reality service is all different. For example, when using the virtual reality service “Let's talk with idol X”, the appearance of idol X, which is a virtual reality object, is displayed in front of the user. As a result, the reaction from the idol X changes, so it may become a virtual reality world that has been associated with the confession to the idol X, or it may become a virtual reality world that quarrels with the idol X. Because.

Therefore, when it is determined that the provision of the virtual reality service to the user is started based on a virtual reality service use request from the user, the virtual reality identification number assigning unit creates the virtual reality created by the provided virtual reality service. A virtual reality identification number that uniquely identifies the virtual reality world is assigned to the world in advance.

The virtual reality related data storage database 500 stores various data used for construction of the virtual reality world as virtual reality related data, but stores the virtual reality related data in a state associated with the virtual reality identification number. Keep it. With this configuration, by designating the virtual reality identification number at a later date, the virtual reality related data associated with the virtual reality identification number can be read and the virtual reality world can be reproduced.

The history information storage server 600 stores virtual reality history information in which correspondences between identification information for identifying a user and virtual reality identification information assigned to the virtual reality world presented to the user are summarized in time series. FIG. 11 shows an example of virtual reality history information. The virtual reality history information includes a virtual reality identification number that identifies the virtual reality world presented to the user, a user ID that identifies the user who receives the presentation of the virtual reality world identified by the virtual reality identification number, A virtual reality service identification number for identifying a virtual reality service related to the virtual reality world, and chapter information indicating which chapter (chapter) of the virtual reality service the virtual reality world is related to; The position information and time information indicating the position and time (time zone) of the user who receives the presentation of the virtual reality world are collected.

In this case as well, the virtual reality world presented to the user cannot be reproduced only with the virtual reality history information stored in the history information storage server 600. In order to reproduce the virtual reality world for the user, the virtual reality identification number included in the virtual reality history information is designated and associated with the virtual reality identification number from the virtual reality related data storage database 500. It is necessary to read virtual reality related data and regenerate virtual reality presentation video and virtual reality presentation audio.

(Embodiment 2)
FIG. 12 is an external perspective view of the virtual reality presentation device 100 according to the second embodiment from obliquely forward and obliquely rearward. A virtual reality presentation device 100 that is a head-mounted display device includes at least a frame 101 and a front panel 102.

The frame 101 is a housing of the virtual reality presentation device 100, and an information processing processor, a RAM (Random Access Memory), a ROM (Read only memory), a battery, and the like are arranged therein. In addition, a dedicated video processor or the like that draws video as necessary is also arranged. The frame 101 is roughly divided into a right frame 101a, a left frame 101b, and a center frame 101c.

The light frames 101a and 101b are members connected to the left and right ears, respectively. Speakers 171a and 171b that convert an analog electric signal into air vibration and output sound are arranged inside one end, and the other end is a center frame. 101c.

In addition, peripheral microphones 132a and 132b that collect surrounding sounds are arranged outside one ends of the right frame 101a and the left frame 101b. In addition, a user microphone 131 that collects a voice uttered by the user is provided below the front end inside the light frame 101a. The sound collected by the user microphone 131 and the peripheral microphones 132a and 132b is converted into a digital signal by AD conversion and output to the information processing processor.

At the ends of the light frames 101a and 101b, inclination sensors 111a and 111b for detecting inclination and acceleration sensors 112a and 112b for detecting acceleration are arranged, respectively. In addition, peripheral cameras 122a and 122b that capture surrounding scenery are respectively arranged outside the right frame 101a and the left frame 101b.

Both ends of the center frame 101c are connected to the right frame 101a and the left frame 101b, respectively, and the front panel 102 is connected to the lower side of the center frame 101c.

Near the center and near both ends of the center frame 101c, tilt sensors 111c, 111d, and 111e that detect tilt and acceleration sensors 112c, 112d, and 112e that detect acceleration are arranged, respectively. In addition, line-of-sight cameras 121a and 122b that capture a scene corresponding to the user's field of view are disposed on the front side of the center frame 101c and on the left and right eyes of the user. The line-of-sight cameras 121a and 121b are disposed on the front side of the center frame 101c at positions separated by a distance corresponding to the distance between the left and right eyes of the user.

The front panel 102 is disposed at a position in front of the user's eyes with the virtual reality presentation device 100 mounted on the head. In the virtual reality presentation device 100 according to the second embodiment, the front panel 102 is a transmissive display, and is a light display panel (first video display unit) that is a display area that displays video in front of the right eye and in front of the left eye, respectively. 151a and a left display panel (second video display unit) 151b are arranged.

The right display panel 151a and the left display panel 151b have a structure in which a light guide plate and a liquid crystal panel (LCD) are overlapped. The liquid crystal panel has a structure in which a transparent electrode (transparent conductive film), a polarizing plate (polarizing film), a liquid crystal layer, a glass substrate, a color filter, and the like are laminated.

FIG. 13 is a cross-sectional view of the vicinity of the light display panel 151 a of the virtual reality presentation device 100. A front panel 102 is disposed below the center frame 101c, and a light source 152 is disposed below the center frame 101c.

Visible light emitted from the light source 152 is irradiated to the front panel 102 side. The front panel 102 has a configuration in which the light guide plate 153 and the LCD 154 are combined. The visible light emitted from the light source 152 is reflected to the user side by the light guide plate 153 and is transmitted through the display-controlled LCD 154 to display an image. Is done.

FIG. 14 is a block diagram illustrating a functional configuration of the virtual reality presentation system 2000 according to the second embodiment. The virtual reality presentation system 2000 includes a virtual reality presentation device 100, a virtual reality service providing server 200, a contractor information storage DB 300, a virtual reality content storage DB 400, and a presented virtual reality related data storage database 500. The virtual reality presentation device 100 and the virtual reality service providing server 200 are connected to each other via a communication network such as the Internet.

The virtual reality presentation device 100 includes a user state detection unit 110, an ambient condition detection unit 120, a voice input unit 130, a display control unit 140, a display unit 150, a voice reproduction control unit 160, and a voice output unit 170. The virtual reality presentation control unit 180 and the communication unit 190 are provided.

The user state detection unit 110 detects the state of a user who is using the virtual reality presentation device 100 that is a virtual reality presentation device. As the user state detection unit 110, the virtual reality presentation device 100 includes an inclination sensor 111 that detects inclination and an acceleration sensor 112 that detects acceleration. The inclination information generated by the inclination sensor 111 and the acceleration information generated by the acceleration sensor 112 are transmitted from the communication unit 190 to the virtual reality service providing server 200 as user state information indicating the state of the user.

The surrounding state detection unit 120 detects the surrounding state of the user who is using the virtual reality presentation device 100. Specifically, the surrounding situation detection unit 120 detects the surrounding situation by acquiring an image of the external environment. As the surrounding state detection unit 120, the virtual reality presentation device 100 includes a line-of-sight camera 121 that captures a scene corresponding to the user's line of sight, and a peripheral camera 122 that captures a range of scenes that cannot be captured by the line-of-sight camera 121, such as the user's sides or the back. And comprising. The video of the scenery in front of the user captured by the line-of-sight camera 121 and the video of the surroundings of the user captured by the peripheral camera 122 are each appropriately encoded such as compression, and then transmitted from the communication unit 190 as the surrounding state information. It is transmitted to the virtual reality service providing server 200.

The voice input unit 130 collects voice, converts it into an analog electric signal, performs AD conversion processing, and converts it into a voice digital signal, thereby inputting the voice. As the voice input unit 130, the virtual reality presentation device 100 includes a user microphone 131 that collects voices uttered by a user and a peripheral microphone 132 that collects peripheral voices.

The user microphone 131 is arranged inside the virtual reality presentation device 100 (user side) so that the user's voice can be collected efficiently, while the peripheral microphone 132 efficiently collects the voice from the user's external environment. It is arranged outside the virtual reality presentation device 100 so that it can. The peripheral microphone 132 is preferably arranged in the vicinity of the position of the ear in the virtual reality presentation device 100 so that the user can collect sound that can be heard when the virtual reality presentation device 100 is not worn. The user microphone 131 and the peripheral microphone 132 collect sound and convert it into an analog electric signal, and perform AD conversion processing to convert it into a sound digital signal, thereby inputting the sound. The input speech is subjected to speech encoding as necessary, and then transmitted to the virtual reality service providing server 200 from the communication unit 190 as user speech information and ambient speech information, respectively.

The display control unit 140 inputs virtual reality presentation video data generated by the virtual reality service providing server 200 among various data received by the communication unit 190, and performs virtual processing based on the input virtual reality presentation video data. Control is performed to display the video of the real object on the display unit 150 as a virtual reality presentation video.

The display unit 150 displays the virtual reality presentation video based on the control from the display control unit 140. The virtual reality presentation video displayed on the display unit 150 is a video such as a still image or a moving image of a virtual reality object for making the user feel virtual reality. As the display unit 150, the virtual reality presentation device 100 is provided with a right display panel 151a and a left display panel 151b, and each displays a virtual reality presentation video with parallax added thereto.

The audio reproduction control unit 160 inputs virtual reality presentation audio data generated by the virtual reality service providing server 200 among various data received by the communication unit 190, and based on the input virtual reality presentation audio data. Then, the audio output unit 170 performs control to output audio generated by the virtual reality object, background music for constructing the virtual reality world, and the like.

  Here, in order to make the user recognize the sound caused by the virtual phenomenon as the sound caused by the real phenomenon, the user is recognized as if the sound is transmitted from the position of the virtual reality object that is the sound generation source. This is preferable in attracting users to the virtual reality world. Therefore, it is preferable that the audio reproduction control unit 160 reproduces the audio data in a form corresponding to the three-dimensional sound such as 5.1Ch, 6.1Ch, or 7.1Ch.

  The audio output unit 170 outputs audio according to the reproduction control from the audio reproduction control unit 160. As the audio output unit 170, the virtual reality presentation device 100 includes speakers 171 a and 171 b arranged near the rear ends of the right frame 101 a and the left frame 101 b, and DA is added to the audio data reproduced by the audio reproduction control unit 160. Conversion is performed to convert the signal into an analog electric signal, which is further converted into air vibration and emitted to the outside.

The virtual reality presentation control unit 180 controls processing for presenting virtual reality to the user by the virtual reality presentation device 100. The virtual reality presentation control unit 180 starts control for presenting virtual reality when the communication unit 190 receives virtual reality service provision start notification information instructing the start of the virtual reality service.

Specifically, based on the virtual reality service provision start notification information, the virtual reality presentation control unit 180 provides user status information, ambient status information, and voice status information to the user status detection unit 110, the ambient status detection unit 120, and the voice input unit 130, respectively. Instruct to start acquisition of audio information and instruct the communication unit 190 to periodically transmit the information to the virtual reality service providing server 200. Further, the virtual reality presentation control unit 180 instructs the display control unit 140 to start control of displaying video on the display unit 150, and starts playback control of audio output from the audio output unit 170 to the audio playback control unit 160. Instruct.

  In accordance with control from the virtual reality presentation control unit 180, the communication unit 190 includes user state information indicating the user state detected by the user state detection unit 110, and surroundings indicating the user's surroundings detected by the surrounding state detection unit 120 The situation information and the voice information indicating the voice input by the voice input unit 130 are transmitted to the virtual reality service providing server 200, respectively. Further, the communication unit 190 outputs the virtual reality presentation video data and the virtual reality presentation audio data received from the virtual reality service providing server 200 to the display control unit 140 and the audio reproduction processing unit 160, respectively.

Next, the virtual reality service providing server 200 will be described. The virtual reality service providing server 200 includes a communication unit 210, an audio analysis processing unit 220, an image analysis processing unit 230, a virtual reality service provision control unit 240, a virtual reality object behavior control unit 250, and a display video generation processing unit. 260 and a reproduction sound generation processing unit 270.

The communication unit 210 communicates with the virtual reality presentation device 100 worn by each user via a communication network such as the Internet. The communication unit 210 receives, from the virtual reality presentation device 100, user status information indicating the user status, ambient status information indicating the user's surroundings, voice information generated by the user and voice information indicating the surrounding sounds. The communication unit 210 outputs audio information to the audio analysis processing unit 220, outputs ambient state information to the image analysis processing unit 230, and outputs user state information to the display video reproduction processing unit 260 and the reproduction audio generation processing unit 270, respectively. To do.

  The voice analysis processing unit 220 receives the voice information received by the communication unit 210 and analyzes the voice included in the voice information to extract the context and emotional elements of the voice uttered by the user. The voice analysis processing unit 220 outputs information about the extracted voice context and emotional elements to the virtual reality service provision control unit 240 and the virtual reality object behavior control unit 250, respectively. In addition, the voice analysis processing unit 220 specifies the voice level of the surrounding voice and the voice level of the user's voice input by the voice input unit 130 of the virtual reality presentation device 100, and outputs them to the playback voice generation processing unit 270.

The image analysis processing unit 230 receives the ambient situation information received by the communication unit 210 and analyzes an image (video) included in the ambient situation information to thereby determine the positional relationship of objects around the user and the brightness of the surroundings. Extract information such as size. The image analysis processing unit 230 outputs information such as the positional relationship and brightness of the extracted objects to the virtual reality object behavior control unit 250 and the display video generation processing unit 260, respectively.

The virtual reality service provision control unit 240 performs processing for starting provision of the virtual reality service and processing for ending. The virtual reality service provision control unit 240 performs scheduling related to the virtual reality service in consideration of the resources of the server 200 and the like, and refers to the contractor information stored in the contractor information storage database 300 to the user. Decide whether to start providing virtual reality services. Further, based on the plan contracted by the user indicated by the contractor information, it is determined whether or not to end the currently provided virtual reality service.

The virtual reality service provision control unit 240 instructs the virtual reality presentation device 100 to start the virtual reality service via the communication unit 210 when determining whether to provide the virtual reality service to the user. Based on the voice information and surrounding situation information received from the virtual reality presentation device 100 based on the instruction, it is determined whether the user can enjoy the virtual reality service, and the virtual reality service is provided based on the determination result. Decide to start.

When the virtual reality service provision control unit 240 determines the start of provision of the virtual reality service, the virtual reality service provision control unit 240 reads the virtual reality content related to the virtual reality service to be provided from the virtual reality content storage database 400, and places the virtual reality object in the virtual space. Deploy.

The virtual space is a virtual space represented by, for example, XYZ orthogonal coordinates or RθZ polar coordinates with the user's position as the origin, and the positional relationship of surrounding objects grasped by the analysis processing in the image analysis processing unit 230. In addition, the virtual reality object is positioned.

  The virtual reality object behavior control unit 250 controls the behavior of the virtual reality object based on the voice context and emotion information extracted by the voice analysis processing unit 220 and the movement of the object extracted by the image analysis processing unit 230. To do. Specifically, control for deforming the shape of the virtual reality object is performed, or sound emitted from the virtual reality object is controlled.

Specifically, the virtual reality object behavior control unit 250 creates a behavior control program for controlling the behavior of the virtual reality object arranged in the virtual space based on the analysis results in the voice analysis processing unit 220 and the image analysis processing unit 230. Then, the virtual reality content storage database 400 is selected, and the selected behavior control program is executed to sequentially change the shape of the virtual reality object.

The display video generation processing unit 260 generates a video displayed on the display unit 150 of the virtual reality presentation device 100. Specifically, the display video generation processing unit 260 determines a display position on the display panel of the virtual reality object controlled by the virtual reality object behavior control unit 250 based on the user state information, and sets the display position to the determined display position. A video in which a virtual reality object is displayed is generated. The display video generation processing unit 260 also determines the brightness of the virtual reality object based on the brightness around the user extracted by the image analysis processing unit 230 or the brightness (illuminance) at the position where the virtual reality object is placed. An optical correction process for correcting (illuminance) is performed, and an image for displaying the virtual reality object after the correction process at the determined display position is generated. The display video generation processing unit 260 converts the generated video into MPEG2 or H.264. The data is output to the communication unit 210 after being encoded by a predetermined encoding method such as H.264 / AVC.

The reproduction sound generation processing unit 270 generates sound output from the sound output unit 170 of the virtual reality presentation device 100. The reproduction sound generation processing unit 270 generates sound output from the virtual reality object controlled by the virtual reality object behavior control unit 250 and output from each channel based on the positional relationship between the user and the virtual reality object. Further, if the playback sound generation processing unit 270 is configured to perform a sound level correction process for correcting the sound level of the sound to be generated in accordance with the surrounding sound level obtained by the sound analysis processing unit 220, the reality is further increased. preferable. The reproduction sound generation processing unit 270 performs the sound level correction processing on the sound emitted from the virtual reality object controlled by the virtual reality object behavior control unit 250, and outputs the sound from the sound output unit 170 of the virtual reality presentation device 100. Generate audio. The reproduction sound generation processing unit 270 encodes the generated sound using a predetermined encoding method such as MPEG3 or CELP, and then outputs the encoded sound to the communication unit 210.

The communication unit 210 inputs various control information (notification information) instructing presentation of virtual reality in the virtual reality presentation device 100 from the virtual reality service provision control unit 240, and transmits it to the virtual reality presentation device 100. In addition, the communication unit 210 transmits the virtual reality presentation video data and audio data respectively input from the display video generation processing unit 260 and the reproduction audio generation processing unit 270 to the virtual reality presentation device 100. In addition, the communication unit 210 outputs the user state information, the surrounding state information, and the voice information received from the virtual reality presentation device 100 to each corresponding unit.

The presented virtual reality related data storage database 500 associates the video data generated by the display video generation processing unit 260 and the audio data generated by the reproduction audio generation processing unit 270 with the user related to the presentation of the virtual reality service. And remember. Since the generated video data and audio data are displayed or output to the user, the user senses the virtual reality world. By accessing and reading the data, it is possible to reproduce the virtual reality world experienced at that time.

<Embodiment 3>
FIG. 15 is a block diagram illustrating a configuration of a virtual reality presentation system 3000 according to the third embodiment. The virtual reality presentation system 3000 includes a virtual reality presentation device 100, a contractor information storage DB 300, a virtual reality content storage DB 400, a presented virtual reality related data storage DB 500, and a virtual reality service providing server 800.

  The virtual reality service providing server 800 includes a communication unit 210, an audio analysis processing unit 220, an image analysis processing unit 230, a virtual reality service provision control unit 240, a virtual reality object behavior control unit 250, and a display video generation processing unit. 260, sound generation processing unit 270, virtual space generation processing unit 810, virtual reality object arrangement processing unit 820, viewpoint control unit 830, light source control unit 840, model data arrangement processing unit 850, and layer order update A processing unit 860 and a layer information generation processing unit 870 are provided.

The communication unit 210 communicates with the virtual reality presentation device 100 via a communication network such as the Internet. The communication unit 210 includes at least a receiving unit that receives user status information indicating a user status, and a transmitting unit that transmits video data for virtual reality presentation.

The communication unit 210 receives user state information indicating a user's state, a surrounding scene image obtained by photographing the user's surroundings, user voice information that is a voice uttered by the user, ambient voice information that is a voice around the user, and the like. To do. In addition, the communication unit 210 receives various types of information used for virtual reality presentation control in the virtual reality presentation device 100 as necessary. Such information includes report information related to the remaining battery capacity, resource consumption, line speed, etc. of the virtual reality presentation device 100.

The user status information and the surrounding scenery image received by the communication unit 210 include user status detection time information (user status detection timing information) indicating the timing when the user status is detected, and the timing when the surrounding scenery image is captured. A surrounding scene image photographing time information (ambient state detection timing information) is included.

The communication unit 210 transmits video encoded data for virtual reality presentation, audio encoded data for virtual reality presentation, layer information, various types of notification information, and the like.

The voice analysis processing unit 220 performs voice analysis processing on the user voice information received by the communication unit 210, thereby acquiring the emotion information indicating the context of the voice uttered by the user and emotion elements. Moreover, the voice analysis process is performed on the surrounding voice information received by the communication unit 210 as necessary, and the situation around the user is specified.

The image analysis processing unit 230 extracts predetermined information by performing an image analysis process on the surrounding scenery image obtained by photographing the scenery around the user received by the communication unit 210. The surrounding scene image is an image taken by a plurality of line-of-sight cameras 121 and peripheral cameras 122 included in the virtual reality presentation device 100.

For example, the image analysis processing unit 230 extracts an optical component from the surrounding scenery image, and extracts information such as the average illuminance around the user and the position and direction of the light source as optical component information.

In addition, the image analysis processing unit 230 compares each object position included in the surrounding scenery image by comparing the position of each object included in the plurality of surrounding scenery images and comparing the adjacent pixels of the same surrounding scenery image. Specify the position and shape. Based on the position and shape of these objects and the like that exist in the real world around the user, the image analysis processing unit 230 models the appearance shape of the objects and the like to generate model data.

Further, the image analysis processing unit 230 detects an instruction operation from the user based on the user's operation included in the series of surrounding scenery images. For example, as an operation indicating interruption of provision of the virtual reality service, a gesture such as raising the index finger and little finger in front of the line-of-sight camera 121 is assigned in advance. The image analysis processing unit 230 identifies the user's operation included in the surrounding scenery image received by the communication unit 210 by performing a matching process with a predetermined operation video, and indicates operation information indicating an instruction operation from the user To extract.

The virtual reality service provision control unit 240 controls provision of the virtual reality service. Specifically, the virtual reality service provision control unit 240 reads out and executes a control program related to the virtual reality service to be provided from the virtual reality content stored in the virtual reality content storage DB 400, thereby executing the virtual reality service. Start offering.

The virtual reality service provision control unit 240 requests that the voice context extracted by the voice analysis processing unit 220 or the operation information extracted by the image analysis processing unit 230 start provision of the virtual reality service. If there is, the virtual reality service provision start process is performed for the user.

Whether or not the virtual reality service provision control unit 240 can provide the requested virtual reality service to the user with reference to the contractor information of the user stored in the contractor information storage DB 300 That is, it is determined whether the user has a right to use the virtual reality service.

In the contractor information storage DB 300, a virtual reality service that can be used by each user is recorded as contractor information. Further, in the contractor information, a user ID for identifying each user, an authentication password, and other personal information of the user may be recorded together. The virtual reality service provision control unit 240 matches the combination of the user ID and password stored in the contractor information with the user ID and password indicated by the information extracted by the voice analysis processing unit 220 and the image analysis processing unit 230. It is even better if it is configured to determine whether or not the requested virtual reality service can be provided by performing authentication processing or the like.

As a result of the determination, if the requested virtual reality service can be provided, the virtual reality service provision control unit 240 generates virtual reality service provision start notification information for notifying the start of provision of the virtual reality service. . The communication unit 210 transmits the virtual reality service provision start notification information to the virtual reality presentation device 100.

FIG. 16 is a diagram illustrating an example of the virtual reality service provision start notification information. The virtual reality service provision start notification information includes the transmission source address that is the address of the server 800, the transmission destination address that is the address of the virtual reality presentation device 100 that is the transmission destination, and the information is the virtual reality service provision start notification information. And a number assigned to each packet in order to distinguish communication generated in relation to the virtual reality service between the virtual reality presentation device 100 and the virtual reality service providing server 800 from other communication. It includes a communication identification number, virtual reality service identification information for identifying a virtual reality service to be provided, virtual reality control parameter information used in control and processing necessary for provision of the virtual reality service, and the like.

As virtual reality control parameter information, detected user state specifying information for specifying what kind of user state is detected as the user state, and user state transmission indicating the transmission frequency of the user state information indicating the detected user state The frequency, the transmission frequency of the surrounding scenery image that is an image of the surrounding scenery of the photographed user, the surrounding scenery image encoding format indicating the image compression encoding format of the surrounding scenery image, and the video frame rate indicating the frame rate of the video to be displayed The virtual reality presentation video data encoding format indicating the encoding format of the virtual reality presentation video data appropriately transmitted from the virtual reality service providing server 800 is included. The virtual reality service provision control unit 240 sets these parameters based on report information such as the type of virtual reality service to be provided, the processing capability and resources of the virtual reality presentation device 100 worn by the user, and the communication line speed. The virtual reality service provision start notification information is generated using the determined parameter value as virtual reality control parameter information.

In addition, the virtual reality service provision control unit 240 controls the virtual reality service provision end notification information for notifying the end of provision of the currently provided virtual reality service or the control parameter change notification in the virtual reality service as necessary. Various information such as parameter change notification information is generated. These pieces of information generated by the virtual reality service provision control unit 240 are transmitted from the communication unit 210 to the virtual reality presentation device 100.

The function of the virtual reality identification number assigning unit described above may be set as one function of the virtual reality service provision control unit 240. Further, the communication identification number described above may be used as the virtual reality identification number. Since the virtual reality identification number is included in each packet transmitted and received in order to construct the virtual reality world related to the virtual reality service between the virtual reality presentation device 100 and the virtual reality service providing server 700, the description will be given later. The virtual reality related data acquisition unit 880 acquires each data used for construction of each virtual reality world for each virtual reality identification number as virtual reality related data, and stores it in the presented virtual reality related data storage DB 500.

The virtual reality service provision control unit 240 issues a virtual space generation instruction and a virtual reality object placement instruction to the virtual space generation processing unit 810 and the virtual reality object placement processing unit 820, respectively, when provision of the virtual reality service is started. .

In addition, the virtual reality service provision control unit 240 outputs a virtual reality presentation video data generation instruction and a virtual reality presentation audio data generation instruction to the display video generation processing unit 260 and the reproduction audio generation processing unit 270, respectively.

In addition, the virtual reality service provision control unit 240 instructs the virtual reality related data acquisition unit 880 to acquire data used to construct a virtual reality world related to the virtual reality service provided to the user.

The virtual reality content storage DB 400 stores virtual reality content that is data for realizing a virtual reality service. The virtual reality content includes a virtual reality object, a virtual reality service control program that controls the progress of the virtual reality service, a virtual reality object behavior control program that controls the behavior of the virtual reality object, and audio data for virtual reality presentation. It stores voice data for virtual reality presentation.

The virtual reality service provision control unit 240 performs various controls such as the start and end of the virtual reality service by executing the virtual reality service control program stored in the virtual reality content storage DB 400.

The virtual space generation processing unit 810 generates a virtual space used for the virtual reality service based on a virtual space generation instruction from the virtual reality service provision control unit 240. The virtual space is a coordinate space having a predetermined area, and is an information processing space created artificially so as to correspond to a space in the real world.

The virtual reality service provision control unit 240 issues a virtual space generation instruction corresponding to the virtual reality service to be started to the virtual space generation processing unit 810, and the virtual space generation processing unit 810 indicates the virtual space generation instruction. Generate a virtual space that is as large as possible.

Based on the virtual reality object placement instruction from the virtual reality service provision control unit 240, the virtual reality object placement processing unit 820 reads out the virtual reality object related to the virtual reality service from the virtual reality content storage DB 400, and generates a virtual space generation processing unit. 810 is arranged in the generated virtual space.

Here, when there are a plurality of virtual reality objects to be arranged, the virtual reality object arrangement processing unit 820 arranges the read virtual reality objects in virtual spaces of different layers.

That is, the virtual space generation processing unit 810 generates a virtual space of a plurality of layers having a common origin, and the virtual reality object placement processing unit 820 differs from the plurality of virtual reality objects read from the virtual reality content storage DB 400. Place in the virtual space of the layer.

FIG. 17 shows a case where the plurality of virtual reality objects are arranged in different layers of the virtual space. Three virtual reality objects, an elephant, a dog, and a lion, are arranged in three different layers of virtual space, each having a common origin.

The viewpoint control unit 830, based on the user state information received by the communication unit 210, in the virtual space generated by the virtual space generation processing unit 810, the viewpoint position coordinates and the line-of-sight direction corresponding to the user's eye position and line-of-sight direction And control. The viewpoint control unit 830 determines the viewpoint position coordinates corresponding to the position of the user's eyes and the line-of-sight direction from the change in the inclination of the user head and the change in the acceleration specified based on the inclination information and the acceleration information included in the user state information. And the line-of-sight direction are calculated and updated. Here, since the user state information includes user state detection time information that is a time stamp indicating the timing at which the corresponding user state is detected, the viewpoint control unit 830 determines the current viewpoint position coordinates and the line-of-sight direction. Manages the viewpoint position coordinates and the line-of-sight direction set based on the user state detected at which time.

The light source control unit 840 updates the position coordinates of the light source and the illuminance or luminance of the light source in the virtual space generated by the virtual space generation processing unit 810 based on the optical component information extracted by the image analysis processing unit 230. For example, the light source control unit 840 sets the illuminance or luminance of the light source at infinity based on the average luminance in the surrounding scene image obtained by the image analysis processing unit 230. Further, a local light source is newly arranged at a corresponding position in the virtual space based on the optical component information extracted based on the position of the shadow appearing in the surrounding scene image in the image analysis processing unit 230. In addition, the light source control unit 840 may set the color of the light source (color difference information) based on the optical component information extracted by the image analysis processing unit 230.

The model data arrangement processing unit 850 arranges model data such as an object existing in the real world around the user generated by the image analysis processing unit 230 in the virtual space. The model data arrangement processing unit 850 identifies from which position to which direction the surrounding landscape image is captured based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 830, and performs image analysis. The model data of each object generated by the processing unit 230 is arranged in the virtual space.

Here, it is preferable that the virtual space generation processing unit 810 generates a virtual space of a model data dedicated layer in which model data of an object existing in the real world is arranged. The model data arrangement processing unit 850 arranges the model data of the real world object generated by the image analysis processing unit 230 in the virtual space of the model data dedicated layer.

The virtual reality object behavior control unit 250 controls the behavior of the virtual reality object arranged in the virtual space. Some virtual reality objects arranged in each layer of the virtual space by the virtual reality object arrangement processing unit 820 are configured to be operable like a virtual reality object of a human being or an animal, for example. . The virtual reality object behavior control unit 250 reads and executes the behavior control program of the virtual reality object from the virtual reality content storage DB 400, thereby changing the appearance shape of the virtual reality object arranged in the virtual space. Or moving a virtual reality object in the virtual space.

The display video generation processing unit 260 generates a virtual reality presentation video. The display video generation processing unit 260 includes a drawing processing unit 261 and a video encoding processing unit 262.

The drawing processing unit 261 generates a virtual reality presentation video by drawing a virtual reality object arranged in the virtual space based on the virtual reality presentation video generation instruction from the virtual reality service provision control unit 240. . Here, the drawing processing unit 261 generates a virtual reality presentation video for each layer by performing drawing processing for each virtual reality layer.

The drawing processing unit 261 selects one of the layers in the virtual space and the model data dedicated layer, and in the virtual space of the selected layer based on the current viewpoint position coordinates and the line-of-sight direction set by the viewpoint control unit 830, respectively. Projection processing is performed for projecting the virtual reality object and the model of the object in the real world onto the image plane arranged forward in the line-of-sight direction from the viewpoint position coordinates. The drawing processing unit 261 deletes the model portion of the object in the real world from the two-dimensional projection image obtained by the projection processing, thereby obtaining the two-dimensional projection image of the virtual reality object arranged in the selected layer. get.

  The drawing processing unit 261 performs clipping processing that cuts the two-dimensional projection image obtained by the projection processing into a predetermined size. Here, the drawing processing unit 261 is characterized by cutting out in a range exceeding the resolution of the display panel of the virtual reality presentation device 100. For example, when the display panel of the virtual reality presentation device 100 has a resolution of 1080 pixels × 1920 pixels, the drawing processing unit 261 performs a clipping process of cutting the two-dimensional projection image by 1188 × 2112, which is 10% higher. It should be noted that how much the size is to be cut out may be determined based on the magnitude of change in the user state. This is because when the user is stationary, the subsequent correction process is not necessary, and it is not necessary to cut it out into large parts.

  Next, the drawing processing unit 261 performs a texture mapping process for pasting the texture mapping data of the virtual reality object to the two-dimensional projection image after the clipping process.

Next, the drawing processing unit 261 performs a shading process for applying a shadow to the two-dimensional projection image based on the light source set by the light source control unit 840. The drawing processing unit 261 performs anti-aliasing processing as necessary to obtain a virtual reality presentation video for the layer.

Here, the drawing processing unit 261 assigns information indicating the time at which the virtual reality presentation video of the layer generated by drawing is a video drawn based on the user state, and the virtual processing of the layer is performed. Reality-presenting video is generated. The viewpoint position coordinates and the line-of-sight direction used for the drawing process are updated and set by the viewpoint control unit 830. The time when the user state used when setting the currently used viewpoint position coordinates and the line-of-sight direction is detected is set. The user status detection time information, which is the time stamp shown, is given, and the virtual reality presentation video of the layer is generated.

The drawing processing unit 261 generates a virtual reality presentation video for each layer by drawing the virtual space of each layer generated by the virtual space generation processing unit 810 by parallel processing.

The video encoding processing unit 262 encodes the virtual reality presentation video of each layer generated by the drawing processing unit 261 to generate layered virtual reality presentation encoded video data. The layered virtual reality presentation encoded video data is transmitted from the communication unit 210 to the virtual reality presentation device 100.

The layer order update processing unit 860 updates the order of each layer in the virtual space generated by the virtual space generation processing unit 810. Since the rendering processing unit 261 performs rendering for each layer and generates virtual reality presentation videos for each layer, it is necessary to finally synthesize a plurality of virtual reality presentation videos for each layer in the virtual reality presentation device 100. is there. For this reason, the layer order update processing unit 860 sets the order of the layers in each virtual space, and updates the order of the layers.

The layer order is determined by the relationship between the position of the user and the position of the virtual reality object. That is, the layer order is determined in order from the layer where the virtual reality object closer to the user is arranged. The position of the user changes as appropriate, and the position of the virtual reality object also changes due to behavior control performed by the virtual reality object behavior control unit 250. Therefore, the layer order update processing unit 860 includes the viewpoint position coordinates (or user position coordinates) of the user controlled by the viewpoint control unit 830 based on the user state information received by the communication unit 210 and each layer in the virtual space. The layer order is updated based on the position coordinates of each virtual reality object arranged.

For example, when virtual reality objects A to C are arranged in layers 1 to 3 of the virtual space as shown in FIG. 18, in the state 1, the virtual reality objects A, B, and C are in order from the closest to the user. . Therefore, the layer order update processing unit 860 assumes the order of layers 1, 2, and 3. Here, when the positional relationship of the state 2 is obtained due to the user moving in the real world, the virtual reality objects are in the order of B, C, and A from the side closer to the user. Therefore, the layer order update processing unit 860 manages the layer order by changing the layer order to layers 2, 3, and 1.

The layer information generation processing unit 870 generates layer information indicating the current layer order based on the update processing in the layer order update processing unit 860. The layer information generation processing unit 870 includes layer order information indicating the layer order, positional relationship information indicating the positional relationship between the user and each layer, the type and number of virtual reality objects included in each layer, and the importance of each layer. It is preferable to generate layer information including information such as degree. The positional relationship information is information indicating a distance and a vector from the user position to the virtual reality object included in each layer. The layer information is transmitted from the communication unit 210 to the virtual reality presentation device 100 together with the encoded video data for virtual reality presentation for each layer.

Based on the virtual reality presentation sound generation instruction from the virtual reality service provision control unit 240, the reproduction sound generation processing unit 270 reads the virtual reality presentation sound data related to the virtual reality service provided from the virtual reality content storage DB 400, Audio data for virtual reality presentation for each channel is generated.

For example, the playback audio generation processing unit 270 uses the virtual reality content storage DB 400 based on the relative relationship between the viewpoint position set by the viewpoint control unit 830 and the position of the virtual reality object arranged by the virtual reality object arrangement processing unit 820. The virtual reality presentation audio data is generated by distributing the read virtual reality presentation audio data to a plurality of channels.

Here, the reproduction sound generation processing unit 270 reads a plurality of virtual reality presentation sound data from the virtual reality content storage DB 400, distributes the sound data to the sound data of each channel, and synthesizes them, thereby synthesizing the virtual reality presentation sound for each channel. Generate data. Here, as a channel, it is preferable to provide a multi-channel audio output unit 530 such as 5.1Ch, 6.1Ch, 7.1Ch on the virtual reality presentation device 100 side so as to realize three-dimensional sound.

In addition, the reproduction sound generation processing unit 270 extracts the noise level of the user's surrounding environment from the surrounding sound information analyzed by the sound analysis processing unit 220, and reads the virtual reality presentation sound read based on the extracted noise level. It is even better if the data is amplified.

In addition, the reproduction sound generation processing unit 270 generates sound data for virtual reality presentation for each channel by encoding the generated sound data for virtual reality presentation for each channel in a predetermined encoding format. The channel-specific virtual reality presentation speech encoded data is transmitted from the communication unit 210 to the virtual reality presentation device 100.

Further, the reproduction voice generation processing unit 270 may generate the virtual reality presentation voice encoded data by generating the artificial voice by the voice synthesis process and encoding the voice data. In this case, the playback speech generation processing unit 270 includes a response speech context generation processing unit, a speech synthesis processing unit, and a speech encoding processing unit.

The response voice context generation processing unit is extracted by the user analysis information received by the communication unit 210, information about the voice context and emotion elements extracted by the voice analysis processing unit 220, and image analysis processing by the image analysis processing unit 230. Based on various kinds of information, a voice context of response voice generated by the virtual reality object currently arranged in the virtual space is generated. The speech synthesis processing unit joins the basic speech data (phoneme data) associated with the virtual reality object stored in the virtual reality content storage database 400 in accordance with the generated speech context, and then performs a predetermined filtering process. Is performed to generate response voice data. The response context generation processing unit further generates information related to the emotional element related to the response voice based on the input user state information, voice context, and emotional element information, and the voice synthesis processing unit It is further preferable that the synthesized voice data in which the pitch, the voice level, the conversation speed, and the like are generated by executing the filtering process with the filter coefficient based on the information related to the above-described information. The speech encoding processing unit uses the generated synthesized speech data as virtual reality presentation speech data, and generates speech encoding data for virtual reality presentation by performing speech encoding in an encoding format such as CELP. It is good also as the said structure.

The virtual reality related data acquisition unit 880 acquires various data used for realizing the virtual reality world presented to the user wearing the virtual reality presentation device 100 as virtual reality related data.

  For example, the virtual reality related data acquisition unit 880 monitors various types of information transmitted from the communication unit 210 to the virtual reality presentation device 100, and the virtual reality presentation video data and the virtual reality presentation audio data. Data transmitted to the reality presentation device 100 is acquired.

The virtual reality related data acquisition unit 880 stores the acquired virtual reality related data in the presented virtual reality related data storage DB 500. The presented virtual reality related data storage DB 500 stores and stores various types of information used to realize the virtual reality world presented to the user as presented virtual reality related data. For example, when the virtual reality service presented to the user is a virtual reality presentation service of “Idle X and 1 hour date”, the virtual reality related data storage DB 500 stores the video of the virtual reality object of the idle X and the idle Data such as sound emitted from the X virtual reality object is stored and stored as virtual reality related data. The presented virtual reality related data storage DB 500 stores and stores the virtual reality related data acquired by the virtual reality related data acquisition unit 880 as data used to reproduce the world of the virtual reality that has been presented.

The virtual reality presentation video data and the virtual reality presentation audio data acquired by the virtual reality related data acquisition unit 880 are data displayed and output by the virtual reality presentation device, respectively, for presenting the virtual reality world. Used. Therefore, by storing and storing these data, it is possible to reproduce the same virtual reality world by reading and reproducing these data from the presented virtual reality related data storage DB 500 at a later date. .

The virtual reality related data acquisition unit 880 stores the virtual reality related data in the database 500 in a state in which each virtual reality related data related to the acquired virtual reality world is associated with the virtual reality identification information.

The virtual reality presentation device 100 includes a user state detection unit 110, an ambient condition detection unit 120, a voice input unit 130, a display control unit 140, a display unit 150, a voice reproduction control unit 160, and a voice output unit 170. The virtual reality presentation control unit 180 and the communication unit 190 are provided.

The user state detection unit 110 includes an inclination sensor 111 that detects the inclination of the user's head and an acceleration sensor 112 that detects the acceleration of the user's head. The user state detection unit 110 includes an orientation sensor that detects the user's orientation, a position sensor that detects the user's position, a biological reaction sensor that detects biological reactions such as the user's body temperature, blood pressure, pulse, and brain waves, and the user. It may be configured with an internal camera that shoots the facial expression. Detection results of these sensors and the like are collected as user status information together with user status detection time information that is a time stamp indicating the detection time, and transmitted from the communication unit 190 to the virtual reality service providing server 800. A part of the user status information is used for video correction processing in the display control unit 140.

The ambient condition detection unit 120 includes a line-of-sight camera 121 that captures the front of the user corresponding to the user's line of sight, and a peripheral camera 122 that captures the user's periphery such as both sides and the back of the user. The line-of-sight camera 121 is arranged at the position of the user's left eye and right eye, respectively. Ambient scene images captured and acquired by these cameras are collected as ambient status information together with ambient status detection time information that is a time stamp indicating the shooting time, and transmitted from the communication unit 190 to the virtual reality service providing server 800. .

The voice input unit 130 is installed near the user's mouth, collects a voice uttered by the user and inputs user voice information, and collects a voice from an external environment around the user and collects the ambient sound. And a peripheral microphone 132 for inputting voice information. Each piece of audio information input by the audio input unit 130 is transmitted from the communication unit 190 to the virtual reality service providing server 800.

The communication unit 190 communicates with the virtual reality service providing server 800 via a communication network such as the Internet. The communication unit 190 wirelessly transmits user status information detected by the user status detection unit 110, ambient status information detected by the ambient status detection unit 120, and other audio information input by the audio input unit 130. And a wireless reception unit for wirelessly receiving video encoded data for virtual reality presentation and layer information, audio encoded data for virtual reality presentation, various notification information, and the like transmitted from the virtual reality service providing server 800.

The display control unit 140 includes a video decoding processing unit 141, a video correction processing unit 142, a video composition processing unit 143, and a display area selection unit 144.

The video decoding processing unit 141 performs a decoding process on the virtual reality presentation video encoded data for each layer received by the communication unit 190, and acquires virtual reality presentation video data for each layer.

The video correction processing unit 142 corrects the position of the decrypted virtual reality presentation video data for each layer, and the positional relationship included in the user status detected by the user status detection unit 110 and the layer information received by the communication unit 190 Based on information.

Each virtual reality presentation video data generated by the display video generation processing unit 260 is generated based on the viewpoint position and the line-of-sight direction controlled by the viewpoint control unit 830 based on the user state. Here, the user's viewpoint position and line-of-sight direction change during the communication lag between the virtual reality presentation device 100 and the virtual reality service providing server 800, and the time required for drawing processing, encoding processing, decoding processing, and the like. There is a possibility. For this reason, the video correction processing unit 142 performs a predetermined correction on the generated virtual reality presentation video, thereby correcting the video so as to be a video suitable for the current user state.

For example, as shown in FIG. 19, there are two virtual reality presentation images of layer 1 and layer 2, and relative position vectors (x, x) up to the virtual reality object included in each layer indicated by the positional relationship information of each layer. y, z) are (+0.3, +2.2, −0.4) and (+2.1, +21.4, +0.6), respectively. Here, when the user is moving in the −X direction, the viewpoint position and the line-of-sight direction are different between when the video is generated and at the present time. Therefore, the video correction processing unit 142 determines the position of the virtual reality object included in the video based on the user status detected between the user status detection time information given to the virtual reality presentation video and the current time. Correct the size. Since how the video of the virtual reality object is corrected according to changes in the viewpoint position and the line-of-sight direction is determined by the relative positional relationship between the viewpoint position and line-of-sight direction and the virtual reality object, the video correction processing unit 142 Performs a correction process after calculating a correction amount for the virtual reality presentation video for each layer.

The video composition processing unit 143 performs a process of synthesizing the virtual reality presentation video of each layer after the correction processing by superimposing according to the layer order information included in the layer information. The video of the virtual reality object included in the rear layer is hidden by the virtual reality object included in the front layer.

The display area selection unit 144 selects an area to be displayed on the display panel 151 from the virtual reality presentation video after the synthesis process. Since each virtual reality presentation video is a video having a resolution higher than the resolution of the display panel 151 so as to be compatible with the correction processing, the virtual reality presentation video is displayed on the display panel 151 from the virtual reality presentation video after the correction processing and the synthesis processing. Select the range to display.

The display unit 150 displays a virtual reality presentation video according to the control from the display control unit 140. The display unit 150 is two display panels 151a and 151b that are specifically arranged in front of the left and right eyes of the user, and each virtual reality presentation selected by the display region selection unit 144 of the display control unit 140 Display the video. The viewpoint control unit 830 sets two viewpoint position coordinates and line-of-sight directions corresponding to the user's left eye and right eye, and the display image generation processing unit 260 draws from each of these two viewpoints for virtual reality presentation for the right eye. The virtual reality presentation video for the video and the left eye is generated. The generated virtual reality presentation video is displayed on the right display panel 151a and the left display panel 151b through correction processing and layer synthesis processing.

The reproduction audio control unit 160 performs a decoding process on the virtual reality presentation audio encoded data received by the communication unit 190, and distributes the audio data after the decoding process to each channel as necessary. Or, an amplification process for adjusting the sound level is performed.

The audio output unit 170 outputs the virtual reality presentation audio of each channel in accordance with the control from the reproduction audio control unit 160. Specifically, the audio output unit 170 includes a plurality of speakers 171a171b, performs DA conversion processing on the virtual reality presentation audio data reproduced by the reproduction audio control unit 160, converts the audio data into an analog electric signal, and further converts the audio data. Sound is output by converting it into vibration and emitting it to the outside.

The virtual reality presentation control unit 180 controls each process for virtual reality presentation on the virtual reality presentation device 100 side. Based on various notification information and parameter change information received by the communication unit 190, the virtual reality presentation control unit 180 encodes / decodes data in transmission / reception, a detection cycle of a user state and surrounding conditions, a frame rate of video display, and the like. Set the format. For example, the virtual reality presentation control unit 180 sets the type of user state to be detected by the user state detection unit 110 based on the detected user state designation information of the virtual reality control parameter information included in the virtual reality service provision start information, Based on the user state detection frequency, a user state detection frequency to be detected by the user state detection unit 110 is set.

Further, the virtual reality presentation control unit 180 monitors the state of the virtual reality presentation device 100 such as the remaining battery level, temperature, resource consumption, and wireless communication speed, and collects these information in a report to provide a virtual reality service providing server 800. Send to. The report is sent to the virtual reality service provision control unit 240 and used for various controls.

The communication unit 190 transmits user state information, ambient state information, audio information, and various control information generated in the device 100 to the virtual reality service providing server 800, and the virtual reality presentation transmitted from the virtual reality service providing server 800. Video encoded data, virtual reality presentation audio encoded data, various control information and notification information are received.

As described above, in the virtual reality presentation system according to the third embodiment, related data related to the virtual reality world is stored and stored in a state where the virtual reality world presented to the user can be reproduced. Therefore, when it is no longer possible to determine whether the world that you have experienced is the virtual reality world or the real world, it is possible to reproduce the same virtual reality world based on the relevant stored data. The user can appropriately determine that it was a virtual reality world.

In the above description, the case where the virtual reality related data acquisition unit 880 acquires the virtual reality presentation video data and the virtual reality presentation audio data transmitted to the virtual reality presentation device 100 and stores them in the database 500 has been described. However, it is even better if the configuration is such that the ambient state information is acquired together and stored in the database 500.

The user senses the virtual reality world by simultaneously viewing the virtual reality presentation video and the outside scene. Here, when the virtual reality world is reproduced at a later date, the scenery of the outside world may be different. For example, if the user is receiving a virtual reality service called “Idol X and One Hour Date” while in the park, the user will still have the experience of dating the Idol X and the park for an hour. On the other hand, when reproducing at home at a later date, the scenery of the outside world is the scenery of the home, so even if the same video and audio are reproduced, the impression of the virtual reality world felt at that time is different.

Therefore, the surrounding scene image which is the surrounding situation information detected by the surrounding situation detection unit 120 is stored together, and at the time of reproduction, the gaze scene image photographed by the gaze camera 121 and the virtual reality presentation video are synthesized. It is possible to reproduce a world similar to the virtual reality world at that time. This is because the gaze scenery image includes a park scenery that is equivalent to the scenery of the outside world that the user was viewing at that time, and the virtual reality object image of the idol X that is the virtual reality presentation video is included in the image. By multiplexing and synthesizing the video, the scene that was perceived by the user at that time is reproduced.

It is further preferable that the virtual reality related data acquisition unit 880 further acquires voice information input by the virtual reality presentation device 100 and stores it in the database 500. Since it is possible to reproduce what kind of words the user himself has spoken and what kind of noise or the like was heard in the surroundings, the degree of reproduction of the virtual reality world can be increased.

In the above description, the case where the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted from the communication unit 210 is stored as virtual reality related data has been described. Therefore, there arises a problem that a sufficient space must be secured in the database 500. The virtual reality presentation video is a video controlled based on the user state at the time, but the user state at the time of reproduction is different from the user state at the time, so the virtual reality object is displayed in the display panel at the time of reproduction. Will move without permission.

Therefore, the virtual reality related data acquisition unit 880 can be configured to acquire other information as virtual reality related data. FIG. 20 is a diagram illustrating an example of a combination of virtual reality related data acquired by the virtual reality related data acquisition unit 880 so that it can be reproduced at a later date.

In the pattern 01, the virtual reality related data acquisition unit 880 sequentially acquires and collects the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted to the virtual reality presentation device 100 as described above. And stored in the presented virtual reality related data storage DB 500. In this case, the reproducibility when reproducing the virtual reality world presented to the user based on the acquired virtual reality related data is 5 in the 10-level evaluation. This is because the virtual reality world sensed by the user has elements of ambient noise and surrounding scenery, but these elements cannot be reproduced only by such information.

In the pattern 02, the virtual reality related data acquisition unit 880 receives the line of sight received by the communication unit 210 in addition to the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted to the virtual reality presentation device 100. Scenery images are sequentially acquired and collected and stored in the presented virtual reality related data storage DB 500. By acquiring a gaze-scene image and storing it in time series, the virtual reality-presentation video obtained by decoding the virtual reality-presentation video-encoded data is multiplexed and synthesized with the gaze-scene image. Can be reproduced at a high level. Therefore, the reproducibility in this case is set to 6 which is higher.

In the pattern 03, the virtual reality related data acquisition unit 880 receives the virtual reality presentation video encoded data and the virtual reality presentation audio encoded data transmitted from the communication unit 210 to the virtual reality presentation device 100, and the communication unit 210 receives them. The acquired line-of-sight image and user audio information are sequentially acquired and collected, and stored in the presented virtual reality related data storage DB 500. By acquiring and storing user voice information together, it is possible to reproduce what kind of words the user was talking to the virtual reality object at that time, so that the situation at that time can be reproduced at a higher level . Therefore, the reproducibility in this case is set to 6 which is even higher. Note that the virtual reality-related data acquisition unit 880 further increases the reproducibility when it is configured to further acquire the peripheral audio information and store it in the database 500.

In the pattern 04, the virtual reality related data acquisition unit 880 controls the user status information received by the communication unit 210, the surrounding state information, the voice information (user voice information and surrounding voice information), and the control program execution history. The program execution history information is sequentially acquired / collected and stored in the presented virtual reality related data storage DB 500.

The information necessary for reproducing the virtual reality world is an output from the virtual reality service providing server 800. The virtual reality presentation video and the virtual reality presentation audio, which are the outputs, are input to the server 800. And generated by a program executed by the input. Therefore, by sequentially acquiring and storing the input and the history of the executed program, it is possible to obtain the output again by simulating based on this information, and reproduce the virtual reality world. be able to.

Here, the control program execution history information is information that summarizes the relationship between the types of control programs and the times that the virtual reality service provision control unit 240 reads from the virtual reality content storage DB 400 and executes. The virtual reality related data acquisition unit 880 monitors the virtual reality service provision control unit 240, and displays control program execution history information that summarizes the types and times of control programs executed by the virtual reality service provision control unit 240. get. The virtual reality related data acquisition unit 880 is more preferably configured to acquire the control program execution history information further together with the type and time of the virtual reality object behavior control program executed by the virtual reality object behavior control unit 250. It is. The reproducibility in this pattern 04 is 5.

In the pattern 05, the virtual reality related data acquisition unit 880 includes viewpoint information on the viewpoint position coordinates and the line-of-sight direction controlled by the viewpoint control unit 830, the position coordinates of the light source controlled by the light source control unit 840, the light source illuminance, and the light source Light source information relating to color, model data information relating to model data of an object in the real world arranged by the model data arrangement processing unit 850, virtual reality object history information indicating a history of virtual reality objects arranged in the virtual space, Are sequentially acquired and stored in the database 500.

The user status information and ambient status information received by the communication unit 210 are information used for obtaining coordinate information, light source information, and the like. Therefore, coordinate information and light source information, which are results obtained by processing these pieces of information, instead of user state information and surrounding state information, are acquired as virtual reality related data. With this configuration, the amount of information to be stored can be further compressed. Coordinate information is history information that summarizes temporal transitions of viewpoint position coordinates and line-of-sight directions in virtual space, and light source information summarizes temporal transitions of information related to light source position coordinates, light source illuminance, light source color, etc. in virtual space. It is history information.

The virtual reality object history information includes identification information (identification number) for identifying a virtual reality object arranged in the virtual space, position coordinates of the virtual reality object in the virtual space, information on the behavior of the virtual reality object, and the like. The history information is summarized in time series. The appearance of the virtual reality object changes by changing points such as joint data and skeleton data or changing the type of texture data pasted on the surface by the virtual reality object behavior control program. The information related to the behavior of the virtual reality object includes, for example, joint data that defines the position coordinates and angles of joint points that are movable parts in the virtual reality object, parameter values that specify the type and expression of the texture attached to the surface, etc. It is. The virtual reality related data acquisition unit 880 monitors a virtual reality object placed in the virtual space by the virtual reality object placement processing unit 820, and monitors temporal changes in joint data, skeleton data, and texture data in the virtual reality object. Thus, virtual reality object history information is acquired.

According to the virtual reality object history information, the virtual reality object is read from the virtual reality content storage DB 400, and the arrangement and behavior are controlled again in the virtual space, projected from the viewpoint position indicated by the viewpoint information onto the image plane, and shaded based on the light source information By performing the above, it is possible to generate the video generated at that time again. Therefore, the virtual reality world at that time can be reproduced. In this case, the reproducibility is 3.

In the pattern 06, the virtual reality related data acquisition unit 880 acquires viewpoint information, light source information, virtual reality object history information, line-of-sight scene images, and user audio information, and stores them in the database 500. Here, the model data of the pattern 05 is data used for hidden surface processing at that time, but since the environment around the user has changed at the time of reproduction, this processing is a process of regenerating the virtual reality presentation video. However, since it makes little sense, it is not acquired as virtual reality related data. Instead, by storing the line-of-sight scene image and the user voice information, it is possible to reproduce the scene at that time and the voice generated by itself. In this case, since the reproducibility of the virtual reality world is relatively high, the reproducibility is set to 8.

In the pattern 07, the virtual reality related data acquisition unit 880 acquires the light source information, the virtual reality object history information, and the reproduction sound generation history information and stores them in the database 500. The virtual reality presentation audio is generated by the reproduction audio generation processing unit 270 appropriately reading out relevant audio data from the virtual reality content storage DB 400. If the history of which audio data is read at which time is known, the virtual reality presentation audio is generated. Can be reproduced. Therefore, the virtual reality related data acquisition unit 880 monitors the playback audio generation processing unit 270 and acquires the history of audio data read by the playback audio generation processing unit 270 as playback audio generation history information.

Since the viewpoint information is not acquired compared to the pattern 05, the virtual reality object cannot be observed from the same angle or distance as that time, but the virtual reality object can be observed from the third party's position. The world of virtual reality can be reproduced. However, the reproducibility in this case is relatively low, and the reproducibility is 4.

In the pattern 08, the virtual reality related data acquisition unit 880 acquires access history information to the virtual reality content storage DB 400 as virtual reality related data and stores it in the database 500.

Since the output from the virtual reality service providing server 800 to the virtual reality presentation device 100 is obtained by executing various data and programs read from the virtual reality content storage DB 400, the access history to the virtual reality content storage DB 400 is suppressed. By doing so, it is possible to reproduce the virtual reality world.

In the pattern 09, the virtual reality related data acquisition unit 880 has access history information to the virtual reality content storage DB 400, user status information received by the communication unit 210, and a voice context that is a voice analysis result in the voice analysis processing unit 220. The information and the image analysis result in the image analysis processing unit 230 are acquired as virtual reality related data and stored in the database 500.

As described above, the virtual reality related data acquisition unit 880 acquires data used for constructing the virtual reality world presented to the user by the virtual reality presentation device 100 as virtual reality related data, and the presented virtual reality related data The storage DB 500 stores the virtual reality related data. The reproducibility (reproduction level) of the virtual reality world to be reproduced changes depending on what kind of data is acquired and stored as virtual reality related data.

In general, the greater the amount of virtual reality-related data to be stored, the higher the reproducibility of the virtual reality world. On the other hand, storing a large amount of data that cannot be used reduces the efficiency of database use. Therefore, the selection is appropriately made in consideration of the relationship between the reproducibility of the virtual reality world and the data amount.

Here, in order to reproduce the virtual reality world, a video of the virtual reality object represented in the virtual reality world is required. Therefore, at least the virtual reality presentation video data for which the virtual reality object has been drawn or information for deriving the virtual reality object arranged in the virtual space is required. By storing these pieces of information, it is possible to reproduce a virtual reality world with a level of reproducibility of 1.

Here, the virtual reality object operates in the virtual reality world. Therefore, even if the virtual reality object represented in the presented virtual reality world can be identified, there is a difference from the virtual reality world felt by the user at that time alone. Therefore, it is preferable to store the virtual reality presentation video data or to further store information for deriving the motion history of the virtual reality object because the reproducibility is increased.

In the virtual reality world, the virtual reality object emits sound. Users who are impressed with this voice may not believe it was virtual unless they listen to it again. For this reason, it is preferable to further store information for guiding the virtual reality presentation audio data or the audio generated by the virtual reality object because the reproducibility increases.

In addition, as described above, the user senses the virtual reality world by detecting the virtual reality presentation video and the virtual reality presentation sound, and the scenery and sound from the real world together. Sense. Therefore, it is preferable to memorize the scenery in the real world at that time and the surrounding sound in the real world in order to reproduce the virtual reality world sensed by the user at a high level.

Thus, as an example, the virtual reality related data acquisition unit 880 uses the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server 800 to the virtual reality presentation device 100 as virtual reality. The acquired virtual reality related data storage database 500 acquired as related data stores the acquired virtual reality presentation video data and virtual reality presentation audio data.

More preferably, the virtual reality related data acquisition unit 880 acquires, as virtual reality related data, forward scenery images sequentially transmitted from the virtual reality presentation device 100 to the virtual reality service providing server 800, and presents the virtual reality related data storage database that has been presented. 500 stores the acquired forward scene image as virtual reality related data. It is further preferable that the virtual reality related data acquisition unit 880 acquires the surrounding scenery image together and the presented virtual reality related data storage database 500 stores the surrounding scenery image.

  More preferably, the virtual reality related data acquisition unit 880 sequentially acquires user status information transmitted from the virtual reality presentation device 100 to the virtual reality service providing server 800, and the presented virtual reality related data storage database 500 stores the acquisition. The user state history information in which the obtained user state information is collected in time series is stored. If the user status information includes an image taken with an internal camera that captured the user's facial expression, the user's facial expression is obsessed with the virtual reality world. Can be reviewed more objectively.

More preferably, the virtual reality related data acquisition unit 880 acquires various audio data such as user audio data and peripheral audio data input by the virtual reality presentation device 100, and the presented virtual reality related data storage database 500 Voice history information in which the acquired voice data is collected in time series is stored. Since users who are in the virtual reality world have a two-way conversation with the virtual reality object, it is possible to store voices uttered from the user side and voices from the external environment together. , Increase the reproducibility of the virtual reality world.

As another form, the virtual reality related data acquisition unit 880 sequentially acquires and presents virtual reality object identification information for identifying virtual reality objects arranged in at least the virtual space generated by the virtual space generation processing unit 810. The virtual reality related data storage database 500 stores virtual reality object history information in which virtual reality object identification information for identifying virtual reality objects arranged in the acquired virtual space is collected in time series. According to this configuration, according to the virtual reality object history information, the virtual reality world identified by the virtual reality object identification information is again read from the virtual reality content storage database 400 and arranged in the virtual space. Can be reproduced at a certain level.

Here, it is even better if the virtual reality related data acquisition unit 880 acquires the position coordinates of the virtual reality object in the virtual space and information related to the behavior of the virtual reality object as virtual reality related data. The presented virtual reality related data storage database 500 includes virtual reality object history information in which position coordinates of the acquired virtual reality object in the virtual space and information related to the behavior of the virtual reality object are further collected in time series. Remember. With this configuration, when multiple virtual reality objects are placed in the virtual space at the same time, the relative positional relationship of the virtual reality objects is an important factor in reproducing the virtual reality world. It is possible to reproduce the relative positional relationship. Further, when the virtual reality object is an operable object, the virtual reality world at that time can be reproduced by reproducing the motion.

In addition, the virtual reality related data acquisition unit 880 acquires, as virtual reality related data, an execution history of a behavior control program that controls at least the behavior of the virtual reality object arranged in the virtual space generated by the virtual space generation processing unit 810. The presented virtual reality related data storage database 500 may be configured to store behavior control program execution history information in which the acquired joint control program execution history is collected in time series.

More preferably, the virtual reality related data acquisition unit 880 is a voice context sequentially extracted by the voice analysis processing unit 220 (voice context extraction unit) from the user voice input by the user microphone 131 of the virtual reality presentation device 100. Information related to emotional elements is acquired as virtual reality related data, and the presented virtual reality related data storage database 500 stores voice context history information in which acquired voice contexts and emotional elements are sequentially collected in time series.

Further, the virtual reality related data acquisition unit 880 acquires an access history for the virtual reality content storage database 500 as virtual reality related data, and the presented virtual reality related data storage database 500 is an access history for the virtual reality content storage database 500. May be configured to store access history information gathered in time series.

The presented virtual reality related data storage database 500 stores the above-described various data in association with each virtual reality identification information for identifying the virtual reality world, so that the data related to the virtual reality world to be reproduced can be read collectively. preferable.

FIG. 21 is a block diagram of the entire system including the virtual reality reproduction request processing server 900 that reproduces the presented virtual reality world. The virtual reality reproduction request processing server 900 includes a communication unit 910, a reproduction request processing unit 920, a virtual reality related data reading unit 930, an audio analysis processing unit 220, an image analysis processing unit 230, and a virtual reality service provision control unit. 240, a virtual reality object behavior control unit 250, a display video generation processing unit 260, a reproduction audio generation processing unit 270, a virtual space generation processing unit 810, a virtual reality object arrangement processing unit 820, and a viewpoint control unit 830 A light source control unit 840.

The communication unit 910 communicates with the virtual reality presentation device 100n and the PC 20 that request reproduction of virtual reality. The communication unit 910 includes a reproduction request receiving unit 911 that receives a virtual reality reproduction request that requests reproduction of the virtual reality world, and a reproduction virtual reality that transmits virtual reality presentation data for constructing the virtual reality world to be reproduced. A data transmission unit 912 for presentation.

The reproduction request processing unit 920 determines whether to execute processing for reproducing the virtual reality world based on the reproduction request received by the reproduction request receiving unit 911. The reproduction request includes at least the user ID for identifying the user, the user password, and the presented virtual reality identification information for specifying the virtual reality world for which reproduction is requested. The reproduction request processing unit 920 performs authentication processing by referring to the contractor information stored in the contractor information storage DB 300 as to whether the set of the user ID and the password included in the reproduction request matches. If the reproduction request is appropriate, the reproduction request processing unit 920 causes the virtual reality related data reading unit 930 to store virtual reality related data related to the virtual reality world specified by the virtual reality identification information included in the reproduction request. A read instruction is issued.

Based on the instruction, the virtual reality related data reading unit 930 sequentially reads each virtual reality related data identified by the virtual reality identification information from the presented virtual reality related data storage DB 500 and outputs it to the corresponding units.

For example, when the virtual reality presentation video data and the virtual reality presentation audio data are stored as they are in the presented virtual reality related data storage DB 500 as virtual reality related data, the virtual reality related data reading unit 930 may The data is read out and output to the reproduced virtual reality presentation data transmission unit 912. The reproduction virtual reality presentation data transmission unit 912 transmits the virtual reality presentation video data and the virtual reality presentation audio data read by the virtual reality related data reading unit 930 to the reproduction request source apparatus.

In addition, when the gaze scene image is included in the virtual reality related data, the gaze scene image may be combined and transmitted to the requesting apparatus, or a video composition processing unit may be included in the virtual reality reproduction request processing server 900. May be provided, and the combined virtual reality presentation video data obtained by synthesizing the virtual reality presentation video data and the surrounding scene image may be transmitted to the reproduction request source apparatus.

When virtual reality related data is stored in the above format, other functional blocks are not necessary. However, when virtual reality related data is stored in the form of control program execution history information, a virtual The reality reproduction processing server 900 needs to regenerate a virtual reality presentation video or the like. In this case, the virtual reality related data reading unit 930 outputs the read control program execution history information to the virtual reality service provision control unit 240. The virtual reality service provision control unit 240 reads the corresponding control program from the virtual reality content storage DB 400 and executes it at each timing indicated by the control program execution history information, thereby generating the display video generation processing unit 260 and the reproduction audio generation. The virtual reality presentation video data and the virtual reality presentation audio data related to the virtual reality world already presented by the processing unit 270 are regenerated. The reproduction virtual reality presentation data transmission unit 912 transmits the regenerated virtual reality presentation video data and virtual reality presentation audio data to the reproduction request source apparatus.

When the virtual reality related data is stored in the format of access history information to the virtual reality content storage DB 400, the virtual reality related data reading unit 930 uses the read access history information as the virtual reality service provision control unit 240, The virtual reality object behavior control unit 250, the virtual reality object arrangement processing unit 820, the playback sound generation processing unit 270, and the like are output to each unit that reads corresponding data and programs from the virtual reality content storage DB 400. Each of these units reads and processes the specified data and programs at the timing indicated by the access history information, thereby processing the virtual reality presentation video data and virtual reality presentation related to the presented virtual reality world. The audio data is regenerated by the display video generation processing unit 260 and the reproduction audio generation processing unit 270. The reproduction virtual reality presentation data transmission unit 912 transmits the regenerated virtual reality presentation video data and virtual reality presentation audio data to the reproduction request source apparatus.

As described above in each embodiment, according to the present invention, a user who has experienced a virtual reality world can request to reproduce the same virtual reality world again.

By changing the image displayed according to the change in the user's viewpoint position and line-of-sight direction, the object or person on the image displayed on the display panel is presented to the user so as to actually exist in the real world. A virtual reality world is built. Here, the virtual reality presentation device is not limited to the transmissive head-mounted display device. For example, the virtual reality world can be displayed to the user by synthesizing and displaying the virtual reality presentation video in which the virtual reality object is drawn on the video captured by the line-of-sight cameras respectively disposed at the positions of the left and right eyes of the user. It can be set as the structure which presents.

Note that the above-described voice analysis processing unit and image analysis processing unit have a configuration in which a dedicated voice analysis server and image analysis server are separately provided to communicate with the virtual reality service providing server. In addition, each processing unit described above may be included in one information processing server, or a plurality of information processing servers may be linked.

Further, in the above description, the case where the reproduction sound generation processing unit included in the virtual reality service providing server generates sound data for virtual reality presentation has been described. However, virtual reality presentation in a state that can be reproduced in advance in the virtual reality content storage DB 400 has been described. When the audio data is stored, the reproduction audio generation processing unit may be configured to read out the virtual reality presentation audio data and transmit it to the virtual reality presentation device.

The line-of-sight camera and the peripheral camera may be configured to capture a still image and transmit it to the virtual reality service providing server, or may be configured to capture a moving image and transmit it to the virtual reality service providing server.

  Note that the virtual reality related data acquisition unit 880 may be configured to store the data after reducing the resolution of the video data for virtual reality presentation transmitted to the virtual reality presentation device 100 to reduce the data.

Note that it is better if the above-described virtual reality content includes special video effect data such as snow, afterglow, rain, and thunder. The display video generation processing unit is more preferably configured to generate a composite video in which a layer of the special video effect is superimposed on a video in which a virtual reality object is drawn as a virtual reality presentation video.

Note that the above-described method of taking virtual reality related data is an example, and a company that provides a virtual reality service can appropriately select and store the data. In general, in order to be able to reproduce the virtual world with high reproducibility, the amount of virtual reality-related data increases, so it is necessary to prepare a database with a large capacity. For this reason, the data amount of the virtual reality related data may be suppressed even if the reproducibility is lowered. Further, the virtual reality reproduction request processing server may include an erasing unit that erases the virtual reality related data stored in the presented virtual reality related data storage DB 500 when a certain period of time has passed.

Further, the virtual reality reproduction request processing server and the virtual reality service providing server described above may be configured integrally because there are many common processing units.

Each processing unit described above can be realized by causing an information processing apparatus such as a CPU to execute a predetermined program on hardware.

Moreover, it is also possible to combine the structure of each embodiment mentioned above suitably. Further, a part of the virtual reality service providing system may be arranged in the home in the form of a home server.

Virtual reality may also be referred to as augmented reality. Further, a part of the video presented to the user may be configured to display an area determined based on the user state from an image generated in advance as a background image instead of a video drawn in real time.

The present invention can take the following forms.
(Appendix 1) (Basic system design)
A virtual reality presentation device that presents a virtual reality world related to a virtual reality service to a user;
A virtual reality service providing server that generates at least virtual reality presentation video data to be used to construct the virtual reality world and transmits the virtual reality presentation video data to the virtual reality presentation device;
A presented virtual reality-related data storage database that stores data related to construction of the virtual reality world as virtual reality-related data in a state in which the virtual reality presentation device can reproduce the virtual reality world presented to the user; ,
A virtual reality presentation system comprising:
(Appendix 2) (Present Virtual Reality History Information)
The virtual reality service providing server comprises virtual reality identification information assigning means for assigning virtual reality identification information for identifying the virtual reality world to be presented,
A virtual reality service for storing provided virtual reality history information in which correspondences between identification information for identifying a user and the virtual reality identification information assigned to the virtual reality world presented to the user are summarized in time series Further comprising a history information storage database;
The virtual reality presentation system according to attachment 1.
(Appendix 3) (Virtual reality service history information provided)
Virtual reality service history information storing provided virtual reality service history information in which correspondences between identification information for identifying a user and virtual reality service identification information for identifying the virtual reality service provided to the user are summarized in time series Further comprising a storage database;
The virtual reality presentation system according to attachment 1.
(Supplementary Note 4) (Virtual Reality Object)
A virtual reality content storage database for storing virtual reality content used to construct the virtual reality world including at least the virtual reality object related to the virtual reality service;
The virtual reality service providing server includes:
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object related to the virtual reality service in the virtual space;
Video generation means for sequentially generating the virtual reality presentation video data by rendering the virtual reality object arranged in the virtual space in real time;
Comprising at least
The virtual reality presentation device presents the virtual reality world to a user by displaying a video including the virtual reality object based on the generated virtual reality presentation video data.
The virtual reality presentation system according to appendices 1 to 3.
(Supplementary note 5) (Drawing based on user status)
The virtual reality presentation device includes:
User status detection means for detecting the user status;
Wireless transmission means for wirelessly transmitting user status information indicating the detected user status;
Wireless receiving means for wirelessly receiving the virtual reality presentation video data;
Display means for displaying a virtual reality presentation video based on the wirelessly received virtual reality presentation video data;
Comprising
The virtual reality service providing server includes:
Receiving means for receiving the user status information;
Transmission means for transmitting the virtual reality presentation video data sequentially generated by the video generation means;
Further comprising
The video generation means renders the virtual reality presentation video in real time by drawing the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction specified based on the user state information. Generate data sequentially,
The virtual reality presentation system according to attachment 4.
(Appendix 6) (Sound for virtual reality presentation)
The virtual reality content storage database stores audio data related to the virtual reality service as the virtual reality content,
The virtual reality service providing server further includes voice generation processing means for generating virtual reality presentation voice data based on voice data related to the virtual reality service stored in the virtual reality content storage database,
The virtual reality presentation device further includes voice output means for converting the virtual reality presentation voice data into an analog electrical signal and outputting the virtual reality presentation voice to the outside.
The virtual reality presentation system according to attachment 5.
(Supplementary note 7) (Landscape image)
The virtual reality presentation device includes:
Voice input means for collecting user voice data by collecting voice uttered by the user;
Image photographing means for photographing at least the front direction of the user and acquiring a front scenery image;
Further comprising
The virtual reality service providing server includes:
Voice context extraction means for extracting a voice context included in the user voice data;
Light source control means for extracting optical information relating to illuminance or luminance from the front scenery image, and updating the position coordinates of the light source and the illuminance or luminance of the light source;
Viewpoint control means for updating the viewpoint position coordinates and the line-of-sight direction based on the user state information;
Further comprising
The video generation unit is configured to store the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit, and the light source position coordinates and the illuminance or luminance of the light source updated by the light source control unit. Sequentially generating the virtual reality presentation video data by rendering the arranged virtual reality objects in real time;
The virtual reality presentation system according to attachment 6.
(Appendix 8) (Shading)
The video generation unit generates a projection image by projecting the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit, and the light source control unit Generating the virtual reality presentation video by calculating the luminance or illuminance of the virtual reality object in the projection image based on the position coordinates of the light source and the illuminance or luminance of the light source updated by
The virtual reality presentation system according to appendix 7.
(Supplementary note 9) (Virtual reality object behavior control)
The virtual reality content storage database further stores a virtual reality object behavior control program for controlling the behavior of the virtual reality object as the virtual reality content,
The virtual reality service providing server includes:
A virtual reality object behavior control means for controlling the behavior of the virtual reality object placed in the virtual space by executing the virtual reality object behavior control program read from the virtual reality content storage database;
The virtual reality presentation system according to appendix 7 or 8.
(Supplementary Note 10) (Type of Virtual Reality Related Data: Virtual Reality Presentation Data)
The presented virtual reality related data storage database stores the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server to the virtual reality presentation device.
The virtual reality presentation system according to attachment 9.
(Supplementary Note 11) (Type of Virtual Reality Related Data: Ambient Scene Image)
The presented virtual reality related data storage database stores the front scene image transmitted from the virtual reality presentation device to a virtual reality service providing server,
The virtual reality presentation system according to attachment 10.
(Supplementary Note 12) (Type of Virtual Reality Related Data: User Status Information)
The presented virtual reality related data storage database stores user status history information in which the user status information transmitted from the virtual reality presentation device to the virtual reality service providing server is summarized in time series.
The virtual reality presentation system according to appendix 10 or 11.
(Supplementary note 13) (Type of virtual reality related data: user voice data)
The presented virtual reality related data storage database stores user voice history information in which the user voice data input by the virtual reality presentation device is summarized in time series.
The virtual reality presentation system according to any one of appendices 10 to 12.
(Supplementary Note 14) (Type of Virtual Reality Related Data: Virtual Reality Object History)
The presented virtual reality related data storage database stores virtual reality object history information in which virtual reality object identification information for identifying at least the virtual reality object arranged in the virtual space is collected in time series.
The virtual reality presentation system according to attachment 9.
(Supplementary Note 15) (Type of Virtual Reality Related Data: Virtual Reality Object History)
The presented virtual reality related data storage database includes the virtual reality object history information in which the position coordinates of the virtual reality object in the virtual space and the information related to the behavior of the virtual reality object are further summarized in time series. Remember,
The virtual reality presentation system according to attachment 14.
(Supplementary Note 16) (Type of Virtual Reality Related Data: Control Program Execution History)
The presented virtual reality related data storage database stores behavior control program execution history information in which the execution history of a behavior control program for controlling the behavior of the virtual reality object arranged in the virtual space is summarized in time series. ,
The virtual reality presentation system according to attachment 9.
(Supplementary Note 17) (Type of Virtual Reality Related Data: Control Program Execution History)
The presented virtual reality related data storage database stores audio context history information in which the audio contexts extracted by the audio context extraction unit are summarized in time series.
The virtual reality presentation system according to attachment 16.
(Appendix 18) (Method)
A generation step of generating at least virtual reality presentation video data used to construct a virtual reality world related to a virtual reality service;
A presentation step of presenting the virtual reality world to the user by displaying the generated virtual reality presentation video data;
Presented virtual reality related data storage step for storing data related to construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user can be reproduced;
A virtual reality presentation method comprising:

  The present invention can be widely used for providing a virtual reality world that is an extension of the real world.

10 communication network 20 PC
DESCRIPTION OF SYMBOLS 100 Virtual reality presentation apparatus 110 User state detection part 111 Inclination sensor 112 Acceleration sensor 120 Ambient condition detection part 121 Gaze camera 122 Peripheral camera 130 Audio | voice input part 131 User microphone 132 Peripheral microphone 140 Display control part 141 Video decoding process part 142 Video correction | amendment Processing unit 143 Video composition processing unit 144 Display area selection unit 150 Display unit 151a Right display panel 151b Left display panel 160 Audio reproduction control unit 170 Audio output unit 171a Right speaker 171b Left speaker 180 Virtual reality presentation control unit 190 Communication unit 200 Virtual reality Service providing server 210 Communication unit 220 Audio analysis processing unit 230 Image analysis processing unit 240 Virtual reality service providing server 250 Virtual reality object behavior control unit 260 Display video generation processing unit 26 Drawing processing unit 262 image decoding processing unit 270 reproducing the sound generation processing unit 300 subscriber information storage DB 400 virtual reality contents storage DB
500 Presented virtual reality related data storage DB
600 history information storage DB 700 virtual reality reproduction request processing server 800 virtual reality service providing server 810 virtual space generation processing unit 820 virtual reality object arrangement processing unit 830 viewpoint control unit 840 light source control unit 850 model data arrangement processing unit 860 layer order update processing Unit 870 layer information generation processing unit 900 virtual reality reproduction request processing server 910 communication unit 911 reproduction request receiving unit 912 reproduction virtual reality presentation data transmission unit 920 reproduction request processing unit 930 virtual reality related data reading unit

Claims (15)

A virtual reality presentation device that presents a virtual reality world related to a virtual reality service to a user;
A virtual reality service providing server that generates at least virtual reality presentation video data to be used to construct the virtual reality world and transmits the virtual reality presentation video data to the virtual reality presentation device;
A presented virtual reality-related data storage database that stores data related to construction of the virtual reality world as virtual reality-related data in a state in which the virtual reality presentation device can reproduce the virtual reality world presented to the user; ,
A virtual reality presentation system comprising: A virtual reality content storage database for storing virtual reality content used to construct the virtual reality world including at least the virtual reality object related to the virtual reality service;
The virtual reality service providing server includes:
Virtual space generation means for generating a virtual space;
Virtual reality object placement means for placing the virtual reality object related to the virtual reality service in the virtual space;
Video generation means for sequentially generating the virtual reality presentation video data by rendering the virtual reality object arranged in the virtual space in real time;
Comprising at least
The virtual reality presentation device presents the virtual reality world to the user by displaying a virtual reality presentation video including the virtual reality object based on the generated virtual reality presentation video data.
The virtual reality presentation system according to claim. The virtual reality presentation device includes:
User status detection means for detecting the user status;
Wireless transmission means for wirelessly transmitting user status information indicating the detected user status;
Wireless receiving means for wirelessly receiving the virtual reality presentation video data;
Display means for displaying the virtual reality presentation video based on the wirelessly received virtual reality presentation video data;
Comprising
The virtual reality service providing server includes:
Receiving means for receiving the user status information;
Transmission means for transmitting the virtual reality presentation video data sequentially generated by the video generation means;
Further comprising
The video generation means renders the virtual reality presentation video in real time by drawing the virtual reality object arranged in the virtual space based on the viewpoint position coordinates and the line-of-sight direction specified based on the user state information. Generate data sequentially,
The virtual reality presentation system according to claim 2. The virtual reality content storage database stores audio data related to the virtual reality service as the virtual reality content,
The virtual reality service providing server further includes voice generation processing means for generating virtual reality presentation voice data based on voice data related to the virtual reality service stored in the virtual reality content storage database,
The virtual reality presentation device further includes voice output means for converting the virtual reality presentation voice data into an analog electrical signal and outputting the virtual reality presentation voice to the outside.
The virtual reality presentation system according to claim 3. The virtual reality presentation device includes:
Voice input means for collecting user voice data by collecting voice uttered by the user;
Image photographing means for photographing at least the front direction of the user and acquiring a front scenery image;
Further comprising
The virtual reality service providing server includes:
Voice context extraction means for extracting a voice context included in the user voice data;
Light source control means for extracting optical information relating to illuminance or luminance from the front scenery image, and updating the position coordinates of the light source and the illuminance or luminance of the light source;
Viewpoint control means for updating the viewpoint position coordinates and the line-of-sight direction based on the user state information;
Further comprising
The video generation unit is configured to store the virtual space based on the viewpoint position coordinates and the line-of-sight direction updated by the viewpoint control unit, and the light source position coordinates and the illuminance or luminance of the light source updated by the light source control unit. Sequentially generating the virtual reality presentation video data by rendering the arranged virtual reality objects in real time;
The virtual reality presentation system according to claim 4. The virtual reality content storage database further stores a virtual reality object behavior control program for controlling the behavior of the virtual reality object as the virtual reality content,
The virtual reality service providing server includes:
A virtual reality object behavior control means for controlling the behavior of the virtual reality object placed in the virtual space by executing the virtual reality object behavior control program read from the virtual reality content storage database;
The virtual reality presentation system according to claim 5. The presented virtual reality related data storage database stores the virtual reality presentation video data and the virtual reality presentation audio data transmitted from the virtual reality service providing server to the virtual reality presentation device.
The virtual reality presentation system according to claim 6. The presented virtual reality related data storage database stores the front scene image transmitted from the virtual reality presentation device to a virtual reality service providing server,
The virtual reality presentation system according to claim 7. The presented virtual reality related data storage database stores user status history information in which the user status information transmitted from the virtual reality presentation device to the virtual reality service providing server is summarized in time series.
The virtual reality presentation system according to claim 7 or 8. The presented virtual reality related data storage database stores user voice history information in which the user voice data input by the virtual reality presentation device is summarized in time series.
The virtual reality presentation system according to any one of claims 7 to 9. The presented virtual reality related data storage database stores virtual reality object history information in which virtual reality object identification information for identifying at least the virtual reality object arranged in the virtual space is collected in time series.
The virtual reality presentation system according to claim 6. The presented virtual reality related data storage database includes the virtual reality object history information in which the position coordinates of the virtual reality object in the virtual space and the information related to the behavior of the virtual reality object are further summarized in time series. Remember,
The virtual reality presentation system according to claim 11. (Type of virtual reality related data: control program execution history)
The presented virtual reality related data storage database stores access history information in which at least the access history from the virtual reality service providing server for the virtual reality content storage database is summarized in time series.
The virtual reality presentation system according to claim 6. The presented virtual reality related data storage database further stores an execution history of the control program read from the virtual reality content storage database.
The virtual reality presentation system according to claim 13. A generation step of generating at least virtual reality presentation video data used to construct a virtual reality world related to a virtual reality service;
A presentation step of presenting the virtual reality world to the user by displaying the generated virtual reality presentation video data;
Presented virtual reality related data storage step for storing data related to construction of the virtual reality world as virtual reality related data in a state where the virtual reality world presented to the user can be reproduced;
A virtual reality presentation method comprising: