JP7138392B1 - Virtual space sensory system - Google Patents

Abstract

To provide a virtual space bodily sensation system that allows a plurality of users who have experienced different environments in a virtual space to mutually experience the same environment. The avatar action control unit 32 assigns the first characteristic to the first avatar and the second characteristic to the second avatar before the trigger event, and assigns the first characteristic to the first avatar and the second avatar after the trigger event. Give to avatar. The avatar environment determining unit 33 determines the environment of the first avatar based on the motion of the first avatar, and determines the environment of the second avatar based on the motion of the second avatar.

Description

The present invention provides a virtual space sensory system for recognizing that the user exists in a virtual space generated by a server or the like, and a complex space for recognizing a complex space in which virtual objects exist in the real space. It relates to a spatial sensation system.

Conventionally, a virtual space is generated by a server or the like, and a user recognizes an image of the virtual space through a head-mounted display (hereinafter sometimes referred to as "HMD"), and the user himself/herself exists in the virtual space. There is a virtual space sensory system that makes you realize that you are there.

This type of virtual space experience system recognizes the user's motion in the real space (for example, movement of coordinates, change of posture including orientation, etc.) by a motion capture device or the like, and responds to the user according to the recognized motion. There is an avatar that operates in a virtual space (see Patent Document 1, for example).

The virtual space bodily sensation system described in Patent Literature 1 simultaneously generates a plurality of avatars in the same virtual space so as to correspond to each of a plurality of users. In this virtual space bodily sensation system, the image of the virtual space recognized by each user is determined based on the coordinates of the avatar corresponding to the user. That is, even when the same virtual space is being experienced at the same time, the image of the virtual space being visually recognized differs for each user. As a result, the environment of the virtual space experienced is also different.

As a virtual space experience system, in addition to the virtual space experience system described in Patent Document 1, a system that allows each of a plurality of users simultaneously using the virtual space experience system to experience different virtual spaces is also proposed. It is

For example, an avatar corresponding to one user is made to exist in a first virtual space (for example, a virtual space of the sea) while maintaining a state in which the users can communicate with each other, and the user experiences the first virtual space. In addition to making the avatar corresponding to the other user exist in a second virtual space (for example, a virtual space of a mountain), the user is allowed to experience the second virtual space.

In recent years, a complex space has been developed in which a virtual object exists in the real space and a user existing in the real space can influence the virtual object (for example, move the object). A complex spatial experience system has been proposed that generates a complex space and allows the user to experience the complex space.

When such a complex spatial sensation system is used by multiple users at the same time, the virtual objects that exist in the complex space do not necessarily have to be common to all users, and may be different for each user. . That is, even when using the same complex space sensation system, each user may be configured to experience a complex space with different existing virtual objects.

JP 2010-257461 A

By the way, when a plurality of users are experiencing a virtual space or a composite space at the same time, and the environment of the virtual space or the composite space experienced by each user is different, one user experiences the experience of the other user. There are times when you want to experience the environment where you are. Conversely, one user may want the other user to experience the environment he or she is experiencing.

SUMMARY OF THE INVENTION The present invention has been made in view of the above points. An object is to provide a system and a complex spatial sensation system.

The virtual space experience system of the first invention is
a virtual space generation unit that generates a virtual space corresponding to a real space in which a plurality of users including a first user and a second user exist;
an avatar generation unit that generates a plurality of avatars including a first avatar corresponding to the first user and a second avatar corresponding to the second user in the same virtual space;
a user action recognition unit that recognizes actions of the first user and the second user in the physical space;
Avatar action of controlling the action of the first avatar in the virtual space based on the action of the first user, and controlling the action of the second avatar in the virtual space based on the action of the second user. a control unit;
an avatar environment determination unit that determines environments of the first avatar and the second avatar in the virtual space;
a user environment determination unit that determines an environment to be recognized by the one user based on the environment of the one avatar and determines an environment to be recognized by the second user based on the environment of the second avatar;
a trigger event recognition unit that recognizes a predetermined trigger event;
A virtual space experience system comprising an environment output device that allows the first user and the second user to recognize the environment of the virtual space,
Before the predetermined trigger event is recognized,
The avatar action control unit controls a first property that is a property that defines the action of the avatar corresponding to the user in response to the user's action, and a property that is different from the first property and is the property that defines the action of the user's action One characteristic of a second characteristic, which is a characteristic that defines the behavior of the avatar corresponding to the user, is given to the first avatar, and then the behavior of the first avatar is controlled, and the first characteristic and the first characteristic are controlled. Giving the other of the two characteristics to the second avatar, and controlling the behavior of the second avatar,
The avatar environment determining unit determines the environment of the first avatar based on the behavior of the first avatar to which the one property is assigned, and the behavior of the second avatar to which the other property is assigned. determining the environment of the second avatar based on
After the predetermined trigger event is recognized,
The avatar action control unit imparts one of the first property and the second property to the first avatar and the second avatar, and then the action of the first avatar and the action of the second avatar. to control the
The avatar environment determining unit determines the environment of the first avatar based on the behavior of the first avatar to which the one property is assigned, and the behavior of the second avatar to which the one property is assigned. and determining the environment of the second avatar.

Here, the avatar's "environment" refers to what affects the avatar in the virtual space. For example, it is the state of an object existing in the virtual space, physical laws in the virtual space (speed of time flow, magnitude of gravity, etc.), and the like. In addition, the “environment to be recognized” by the user refers to the environment of the virtual space that is recognized by the user through the five senses. For example, the user's environment is images, sounds, etc. of the virtual space that the user recognizes.

In the virtual space experience system of the first invention, the motion of each avatar is controlled based on the corresponding motion of the user. Also, the environment of the virtual space to be recognized by each user is determined based on the environment of the corresponding avatar in the virtual space.

Here, when avatars have different characteristics (for example, size, function, etc.), even if each user performs the same action in the real space, the environment during or after the action of the corresponding avatar may differ. . As a result, the environment experienced by each user may also be different.

For example, if each avatar has a different size, even if each user walks the same number of steps in the real space, the moving distance of the corresponding avatar in the virtual space (that is, the environment experienced by each user after the action) will be different. become.

Then, when the characteristics of each avatar can be different (that is, when the environment experienced by each user can be different), for example, when recognizing an avatar of a user other than oneself, the user who has recognized the avatar is a user other than himself. There may be a desire to experience the environment, or a user who has recognized that desire to allow other users to experience his/her own environment.

Therefore, in the virtual space bodily sensation system of the first invention, one of the first characteristic and the second characteristic is imparted to the first avatar before the predetermined trigger event is recognized, and the first avatar moves. Based on, the environment of the first avatar is determined, the other of the first property and the second property is given to the second avatar, and based on the behavior of the second avatar, the second avatar Determine your environment.

On the other hand, after the predetermined trigger event is recognized, one of the first characteristic and the second characteristic is given to the first avatar, and the environment of the first avatar is determined based on the action of the first avatar. is determined, one of the characteristics is also given to the second avatar, and the environment of the second avatar is determined based on the behavior of the second avatar.

Here, the first characteristic and the second characteristic are characteristics that define the behavior of the avatar corresponding to the user with respect to the behavior of the user. Specifically, it is the size of the avatar in the virtual space and the functions of the avatar (for example, movable speed, movable range, etc.).

Thus, before the predetermined trigger event is recognized, the characteristics of the first avatar corresponding to the first user and the characteristics of the second avatar corresponding to the second user are different. Therefore, when the avatar moves in response to the user's motion, the environment of the avatar becomes independent as in the past. As a result, the environment experienced by the user becomes independent as in the past.

On the other hand, after the predetermined trigger event is recognized, the characteristics of the first avatar corresponding to the first user and the characteristics of the second avatar corresponding to the second user are both the same. Therefore, when the avatar moves in response to the user's motion, the environment of the avatar matches or is similar. Consequently, the environment experienced by the user will also match or be similar.

Therefore, according to the virtual space bodily sensation system of the first invention, after the predetermined trigger event is recognized, the process of synchronizing the environments of the avatars is executed so that the environments of the virtual space recognized by each user are the same. As a result, the first user and the second user, who have experienced different environments until then, can now experience the same environment.

Further, in the virtual space bodily sensation system of the first invention,
Preferably, the predetermined trigger event is a predetermined action performed by one of the first avatar and the second avatar on the other of the first avatar and the second avatar.

Here, the "predetermined action" may be any action performed by one of the first avatar and the second avatar with reference to the other avatar.

For example, the predetermined action is an action in which one avatar touches a second avatar, an action in which one avatar moves within a predetermined range relative to the other avatar, and one avatar moves in a complex space. When operating an object existing in a virtual space existing in a virtual space, the other avatar is selected as the target (for example, the other avatar is photographed with a camera-type object, etc.).

In this way, when a predetermined action performed by one avatar with respect to the other avatar is set as a predetermined trigger event, the user corresponding to one avatar will be triggered by the other avatar as a result of executing the action. It becomes possible to intuitively understand that the second user can also experience the environment of the second user.

In addition, the virtual space experience system of the second invention is
A first virtual space corresponding to a physical space in which a plurality of users including a first user and a second user exist, and a plurality of second virtual spaces independent of the first virtual space and corresponding to the physical space. a virtual space generation unit that generates a virtual space of
A first avatar corresponding to the first user, a second avatar corresponding to the second user , and a first avatar corresponding to the first user or the second user, independent of the first avatar and the second avatar an avatar generation unit that generates a plurality of avatars, including ghosts , which are avatars created in the first virtual space or the second virtual space;
a user action recognition unit that recognizes actions of the first user and the second user in the physical space;
Avatar action for determining control of the first avatar in the virtual space based on the action of the first user and controlling action of the second avatar in the virtual space based on the action of the second user. a control unit;
an avatar environment determination unit that determines environments of the first avatar and the second avatar in the virtual space;
The environment of the virtual space to be recognized by the one user is determined based on the environment of the one avatar, and the environment of the virtual space to be recognized by the second user is determined based on the environment of the second avatar. a user environment determination unit;
a trigger event recognition unit that recognizes a predetermined trigger event;
A virtual space experience system comprising an environment output device that allows the first user and the second user to recognize the environment of the virtual space,
The predetermined trigger event corresponds to the ghost corresponding to one of the first user and the second user, and the other of the first user and the second user among the first avatar and the second avatar. is a predetermined action performed by an avatar that performs
Before the predetermined trigger event is recognized,
The avatar generation unit generates the first avatar in one of the first virtual space and the second virtual space, and generates the second avatar in one of the first virtual space and the second virtual space. The ghost corresponding to one of the first user and the second user is generated in the other virtual space and placed in the other of the first avatar and the second avatar in the first virtual space or the second virtual space. generated in the virtual space where
The avatar environment determination unit determines the environment of the first avatar based on the one virtual space and determines the environment of the second avatar based on the other virtual space;
After the predetermined trigger event is recognized,
The avatar generation unit generates the first avatar and the second avatar in one of the first virtual space and the second virtual space,
The avatar environment determination unit determines the environment of the first avatar based on the one virtual space, and determines the environment of the second avatar based on the one virtual space. .

Here, the avatar's "environment" refers to what affects the avatar in the virtual space. For example, the avatar's environment is the type of existing virtual space itself (whether it is on the ground or in the water). In addition, the “environment to be recognized” by the user refers to the environment of the virtual space that is recognized by the user through the five senses. For example, the user's environment is images, sounds, etc. of the virtual space that the user recognizes.

In the virtual space experience system of the second invention, the motion of each avatar is controlled based on the corresponding motion of the user. Also, the environment of the virtual space to be recognized by each user is determined based on the environment of the corresponding avatar in the virtual space.

Here, even when a plurality of users use the virtual space experience system at the same time, the virtual space experienced by each user (that is, the virtual space environment) does not necessarily have to be based on one virtual space. Instead, it may be based on an independent virtual space for each user.

For example, a first avatar corresponding to the first user may be present in the virtual space of the sea to allow the first user to experience the virtual sea environment, and at the same time, a second avatar corresponding to the second user may be placed in the virtual space of the mountain. , so that the second user can experience a virtual mountain.

Then, when the virtual space in which each avatar exists can be different (that is, when the environment experienced by each user can be different), for example, each user can communicate (specifically, a conversation beyond the type of space). , a user may desire to experience the environment of a user other than himself, or a user may desire to allow a user other than himself to experience his own environment.

Therefore, in the virtual space bodily sensation system of the second invention, before a predetermined trigger event is recognized, the first avatar is generated in one of the first virtual space and the second virtual space, and The environment of the first avatar is determined based on the virtual space, the second avatar is generated in the other virtual space of the first virtual space and the second virtual space, and the second avatar is generated based on the other virtual space. Determines the avatar's environment.

On the other hand, after the predetermined trigger event is recognized, the first avatar and the second avatar are generated in one of the first virtual space and the second virtual space, and based on the one virtual space , determine the environment of the first avatar and the second avatar.

Thereby, the virtual space in which the first avatar corresponding to the first user exists and the virtual space in which the second avatar corresponding to the second user exists are independent before the predetermined trigger event is recognized. It has become a thing. As such, the avatar's environment becomes independent, as in the past. As a result, the environment experienced by the user becomes independent as in the past.

On the other hand, after the predetermined trigger event is recognized, the virtual space in which the first avatar corresponding to the first user exists and the virtual space in which the second avatar corresponding to the second user exists are the same virtual space. become. As such, the avatar's environment will match or be similar. Consequently, the environment experienced by the user will also match or be similar.

Therefore, according to the virtual space bodily sensation system of the second invention, after the predetermined trigger event is recognized, the process of synchronizing the environments of the avatars is executed so that the environments of the virtual space recognized by each user are the same. As a result, the first user and the second user, who have experienced different environments until then, can now experience the same environment.

Here, the "predetermined action" may be any action performed by the avatar with reference to the ghost. For example, the predetermined action is an action in which the avatar touches a ghost, an action in which the avatar moves within a predetermined range based on the ghost, an avatar operating an object existing in the virtual space, An operation of selecting a ghost as the target (for example, an operation of photographing a ghost with a camera-type object, etc.).

When multiple users are using the virtual space experience system at the same time and the virtual space experienced by each user (i.e., virtual space environment) is different, each user has other users. For the purpose of notifying the may generate a ghost that is

When a predetermined action performed by the avatar on the ghost is set as a predetermined trigger event, the user corresponding to the avatar can experience the environment of the user corresponding to the ghost as a result of executing the action. You will be able to intuitively understand what you can do.

In addition, the complex spatial sensation system of the third invention is
a real space recognition unit that recognizes a real space in which a plurality of users including a first user and a second user exist;
an object generation unit that generates a first object and a second object, which are virtual objects to be recognized by the first user and the second user;
generating a plurality of complex spaces including a first complex space that is a space in which the first object exists in the physical space and a second complex space that is a space in which the second object exists in the physical space; a composite space generator;
a user environment determination unit that determines an environment to be recognized by the first user and the second user;
a trigger event recognition unit that recognizes a trigger event;
A complex spatial sensation system comprising an environment output device that allows the first user and the second user to recognize the environment of the complex space,
Before the predetermined trigger event is recognized,
The user environment determination unit determines an environment in one of the first complex space and the second complex space as an environment to be recognized by the first user, and determines the first complex space and the second complex space. determining the environment in the other composite space of the space as the environment to be recognized by the second user;
After the predetermined trigger event is recognized,
The user environment determining unit determines an environment in one of the first complex space and the second complex space as an environment to be recognized by the first user and the second user.

Here, the “environment to be recognized” by the user refers to the environment of the complex space that is recognized by the user through the five senses. For example, the user's environment is an image of an object existing in a complex space recognized by the user, a sound generated based on the object, and the like.

Even when multiple users use the complex spatial sensation system at the same time, the complex space experienced by each user (that is, the environment of the complex space) does not necessarily have to be based on a single complex space. It may be based on different composite spaces that are independent for each user.

For example, a first user is allowed to experience a first composite space in which a first object exists in a predetermined region of the physical space, and at the same time, a second user is instructed to experience the same region of the physical space as the region used for the first composite space. In some cases, a second composite space in which a second object is present is experienced.

Then, when the complex space recognized by each user can be different (that is, when the environment experienced by each user can be different), for example, when each user communicates (specifically, a conversation, etc.), a certain user A user may desire to experience another user's environment, or a user may wish to have another user experience his/her own environment.

Therefore, in the complex spatial sensation system of the third invention, before the trigger event is recognized, the environment in one of the first complex space and the second complex space is determined as the environment to be recognized by the first user. In addition, the environment in the other of the first complex space and the second complex space is determined as the environment to be recognized by the second user.

On the other hand, after the trigger event is recognized, the environment in one of the first complex space and the second complex space is determined as the environment to be recognized by the first user and the second user.

As a result, before the trigger event is recognized, the composite space experienced by the first user and the composite space experienced by the second user become composite spaces in which existing virtual objects are independent of each other.

On the other hand, after the trigger event is recognized, the complex space experienced by the first user and the complex space experienced by the second user become the same complex space in which existing virtual objects match.

Therefore, according to the complex spatial experience system of the third invention, after the trigger event is recognized, the process for synchronizing the objects existing in the complex space is executed, and the environment of the complex space is made to be recognized by each user. become similar, the first user and the second user, who have experienced different environments up until now, can now experience similar environments.

Further, in the complex spatial sensation system of the third invention,
a user action recognition unit that recognizes actions of the user in the physical space;
Preferably, the predetermined trigger event is a predetermined action performed by one of the first user and the second user on the other of the first user and the second user in the complex space.

Here, the "predetermined action" may be any action performed by one of the first user and the second user based on the other. For example, the predetermined action is an action in which the first user touches the second user, an action in which the first user moves within a predetermined range with respect to the second user, and an action in which the first user moves into a complex space. When operating a virtual or real object that exists in a virtual or real object, an action to select the second user as the target (for example, an action to take a picture of the second user with the camera of a virtual mobile terminal or a real mobile terminal etc.).

Thus, when the first user sets a predetermined action to be performed by the second user as a predetermined trigger event, the first user also experiences the second user's environment as a result of executing the action. You will be able to intuitively understand what you can do.

1 is a schematic diagram showing a schematic configuration of a VR system according to a first embodiment; FIG. FIG. 2 is a block diagram showing the configuration of a processing unit of the VR system in FIG. 1; FIG. 3 is a flowchart showing processing executed by the VR system of FIG. 2 in normal use; FIG. FIG. 3 is a schematic diagram showing states of a real space and a virtual space in a normal usage state of the VR system of FIG. 2; FIG. 3 is a flow chart showing the processing performed by the VR system of FIG. 2 when synchronizing avatar characteristics; FIG. FIG. 3 is a schematic diagram showing the state of the virtual space when a trigger event occurs in the VR system of FIG. 2; FIG. 3 is a schematic diagram showing the state of the virtual space after a trigger event has occurred in the VR system of FIG. 2; The block diagram which shows the structure of the process part of the VR system which concerns on 2nd Embodiment. 9 is a flow chart showing processing executed by the VR system of FIG. 8 in normal use. FIG. 9 is a schematic diagram showing the states of the real space and the virtual space in the normal usage state of the VR system of FIG. 8; FIG. 9 is a flow chart showing processing executed when the VR system in FIG. 8 synchronizes the virtual space; FIG. FIG. 9 is a schematic diagram showing the state of the virtual space when a trigger event occurs in the VR system of FIG. 8; FIG. 9 is a schematic diagram showing the state of the virtual space after a trigger event has occurred in the VR system of FIG. 8; The schematic diagram which shows schematic structure of the MR system which concerns on 3rd Embodiment. FIG. 15 is a block diagram showing the configuration of the processing unit of the MR system of FIG. 14; FIG. 16 is a flow chart showing processing executed by the MR system of FIG. 15 in normal use; FIG. FIG. 16 is a schematic diagram showing the states of the real space and the composite space in normal use of the MR system of FIG. 15; FIG. 16 is a flow chart showing processing executed when the MR system of FIG. 15 synchronizes complex spaces; FIG. FIG. 16 is a schematic diagram showing the state of the virtual space when a trigger event occurs in the MR system of FIG. 15; FIG. 16 is a schematic diagram showing the state of the virtual space after a trigger event has occurred in the VR system of FIG. 15;

Hereinafter, referring to the drawings, a virtual space sensation system according to the first embodiment corresponding to the first invention, a virtual space sensation system according to the second embodiment corresponding to the second invention, and a virtual space sensation system according to the third embodiment A complex spatial sensation system that

[First embodiment]
The virtual space experience system S1 according to the first embodiment will be described below with reference to FIGS. 1 to 7. FIG.

The VR system S1 is for a first user U1 and a second user U2 (hereinafter collectively referred to as "users U") existing together in a predetermined area (for example, one room, etc.) of the physical space RS. One virtual space VS corresponding to the area is made to recognize that they exist together via the first avatar A1 corresponding to the first user U1 and the second avatar A2 corresponding to the second user U2 ( See Fig. 4, etc.).

In this embodiment, two users are assumed for easy understanding. However, the virtual space experience system of the first invention is not limited to such a configuration, and the number of users may be three or more.

[Schematic configuration of the system]
First, a schematic configuration of the VR system S1 will be described with reference to FIG.

As shown in FIG. 1, a VR system S1 includes a plurality of markers 1 attached to a user U existing in a physical space RS, and a camera 2 that captures the user U (strictly speaking, the marker 1 attached to the user U). , a server 3 that determines the image and sound of the virtual space VS (see FIG. 4, etc.), and a head-mounted display (hereinafter referred to as “HMD 4”) that allows the user to recognize the determined image and sound. .

In the VR system S1, the camera 2, the server 3, and the HMD 4 can mutually transmit and receive information wirelessly, such as through the Internet network, a public line, or short-range wireless communication. However, any one of them may be configured to transmit and receive information to and from each other by wire.

A plurality of markers 1 are attached to the user's U head, both hands, and both feet via the HMD 4, gloves, and shoes that the user U wears. As will be described later, the plurality of markers 1 are used for recognizing the coordinates and posture of the user U in the physical space RS, as well as actions (for example, movement of coordinates, changes in posture including orientation, etc.). be. Therefore, the attachment position of the marker 1 may be changed as appropriate according to other devices that configure the VR system S1.

The camera 2 is installed so that the user U's operable range (that is, the range in which the user U can move, change the posture, etc.) in the physical space RS where the user U exists can be photographed from multiple directions.

The server 3 recognizes the sign 1 from the image captured by the camera 2, and recognizes the coordinates and orientation of the user U based on the position of the recognized sign 1 in the physical space RS. Also, the server 3 determines images and sounds of the virtual space VS to be recognized by the user U, based on the coordinates and orientation.

The HMD 4 is an environment output device that allows the user to recognize the environment of the virtual space VS. The HMD 4 is worn on the user's U head. The HMD 4 includes a monitor 40 for allowing the user U to recognize the image of the virtual space VS determined by the server 3, and a speaker for allowing the user U to recognize the sound of the virtual space VS determined by the server 3. 41 (see FIG. 2).

When experiencing the virtual space VS using the VR system S1, the user U recognizes only the images and sounds of the virtual space VS, and is made to recognize that the user U himself/herself exists in the virtual space VS. That is, the VR system S1 is configured as a so-called immersive system.

Note that the virtual space sensation system of the first invention is not limited to such a configuration. For example, when using a motion capture device, in addition to the above configuration, the number and arrangement of signs and cameras different from the above configuration (for example, one each is provided) may be used. . Alternatively, the posture and coordinates of the user may be recognized by recognizing feature points from the user's image itself without using the mark.

Also, for example, instead of the motion capture device, a device that recognizes only the coordinates of the user may be used. Specifically, for example, a sensor such as GPS may be installed in the HMD, and the user's coordinates, posture, etc. may be recognized based on the output from the sensor. Also, such a sensor may be used in combination with the motion capture device as described above.

[Configuration of processing unit]
Next, with reference to FIG. 2, the configuration of the processing units included in the server 3 will be described in detail.

The server 3 is composed of one or a plurality of electronic circuit units including CPU, RAM, ROM, interface circuits, and the like. As shown in FIG. 2, the server 3 includes a virtual environment generation unit 30, a user action recognition unit 31, and an avatar action control unit 32 as functions (processing units) implemented by the implemented hardware configuration or programs. , an avatar environment determination unit 33 , a user environment determination unit 34 , and a trigger event recognition unit 35 .

The virtual environment generator 30 has a virtual space generator 30a and an avatar generator 30b.

The virtual space generator 30a generates a virtual space VS corresponding to the physical space RS in which the user U exists. Specifically, the virtual space generator 30a generates images that are the background of the virtual space VS and objects that exist in the virtual space VS, and sounds related to those images.

Although the VR system S1 of this embodiment is not provided, the virtual space sensory system is provided with a configuration that realizes a predetermined feel (for example, a cushion that changes in hardness, etc.) and that generates a predetermined smell. In some cases, the virtual space generation unit may generate the virtual space using not only the images and sounds but also their touches and smells.

The avatar generation unit 30b generates a first avatar A1 corresponding to the first user U1 and a second avatar A2 corresponding to the second user U2 in the virtual space VS (see FIG. 4, etc.).

The first avatar A1 and the second avatar A2 (hereinafter collectively referred to as "avatar A") move in the virtual space VS in response to the corresponding motion of the user U in the physical space RS.

The user motion recognition unit 31 recognizes image data of the user U including the sign 1 captured by the camera 2, and recognizes the motion of the user U in the physical space RS based on the image data. The user action recognition section 31 has a user posture recognition section 31a and a user coordinate recognition section 31b.

The user posture recognition unit 31a extracts the marker 1 from the input image data of the user U, and recognizes the posture including the orientation of the user U based on the extraction result.

The user coordinate recognition unit 31b extracts the marker 1 from the input image data of the user U, and recognizes the coordinates of the user U based on the extraction result.

The avatar action control unit 32 gives characteristics to the avatar A generated by the avatar generation unit 30b. In this embodiment, the properties are assigned when the VR system S1 is started to be used (for example, when the type of avatar A to be used is selected). However, its characteristics may change during use of the VR system S1. For example, it is changed when a trigger event, which will be described later, is recognized.

Here, the "characteristic" is a characteristic that defines the action of the avatar corresponding to the user in response to the action of the user, and there are a plurality of types of such properties. Specifically, the characteristics are the size of the avatar in the virtual space VS, and the functions of the avatar (for example, movable speed, movable range, etc.).

The avatar motion control unit 32 also recognizes the posture of the user U in the physical space RS recognized by the user posture recognition unit 31a, the coordinates of the user U in the physical space RS recognized by the user coordinate recognition unit 31b, and the assigned Based on the characteristics, the action of the avatar A corresponding to the user U in the virtual space VS (for example, movement of coordinates, change of posture including orientation, etc.) is determined.

The avatar environment determining unit 33 determines the environment of the avatar A in the virtual space VS based on the result of the action of the avatar A (for example, coordinates, posture, etc. at that time).

Here, the avatar's "environment" in the present embodiment refers to what affects the avatar in the virtual space. For example, the environment of the avatar includes the state of objects existing in the virtual space, physical laws (speed of time flow, magnitude of gravity, etc.) in the virtual space VS.

Based on the environment of the avatar A, the user environment determination unit 34 determines the environment (image and sound) of the virtual space RS to be recognized by the user U corresponding to the avatar A through the monitor 40 and the speaker 41 of the HMD 4. .

Here, the “environment to be recognized” by the user refers to the environment of the virtual space that is recognized by the user through the five senses. For example, the environment of the user U is images, sounds, etc. of the virtual space that the user recognizes.

Here, the "image of the virtual space" includes, in addition to the image of the background of the virtual space, the image of another avatar, the image of an object that exists only in the virtual space, and the image that exists in the virtual space corresponding to the real space. Images of objects, etc. are included.

The trigger event recognition unit 35 recognizes that a predetermined trigger event has occurred when a condition predetermined by the system designer is satisfied.

Here, the "trigger event" may be one of which the user is unaware of its occurrence. Therefore, as a trigger event, for example, events caused by user actions such as the user performing a predetermined action in the real space (that is, the avatar corresponding to the user performing a predetermined action in the virtual space) are also applicable. In addition, it also applies to events that are not caused by the user's actions, such as the passage of a predetermined period of time.

It should be noted that each processing unit that constitutes the virtual space experience system of the first invention is not limited to the configuration as described above.

For example, part of the processing unit provided in the server 3 in this embodiment may be provided in the HMD 4 . Alternatively, a plurality of servers may be used, or the servers may be omitted and the CPUs mounted on the HMDs may cooperate with each other. Also, a speaker other than the speaker mounted on the HMD may be provided. In addition to devices that affect sight and hearing, devices that affect smell and touch such as generating smells, wind, etc. according to the virtual space may also be included.

[Process to be executed]
Next, with reference to FIGS. 2 to 7, processing executed by the VR system S1 when allowing the user U to experience the virtual space VS using the VR system S1 will be described.

[Processing during normal use]
First, with reference to FIGS. 2 to 4, the processing executed by the VR system S1 at the start of use and in normal use (that is, in a state in which a predetermined trigger event to be described later is not recognized) will be described.

In this process, first, the virtual environment generator 30 of the server 3 generates the virtual space VS, the first avatar A1 and the second avatar A2 (FIG. 3/STEP 100).

Specifically, as shown in FIG. 4, the virtual space generation unit 30a of the virtual environment generation unit 30 generates an image serving as a background of the virtual space VS and various objects existing in the virtual space VS. Also, the avatar generation unit 30b of the virtual environment generation unit 30 generates a first avatar A1 corresponding to the first user U1 and a second avatar A2 corresponding to the second user U2.

Next, the avatar action control unit 32 of the server 3 assigns the first property to the first avatar A1 and the second property to the second avatar A2 (FIG. 3/STEP 101).

In this embodiment, the characteristics given to the avatar A are linked to the shape of the avatar A.

Specifically, the shape of the first avatar A1 is an anthropomorphic shape of a turtle, and the first avatar A1 is given a first characteristic corresponding to the shape. The first characteristic is that it is larger than a predetermined object (for example, grass shown in FIG. 4) placed in the virtual space VS, and the corresponding user's motion in the real space RS is reflected in the avatar's motion. It is a characteristic that includes a characteristic that it takes a long time to complete (that is, the reflection speed of the operation is slow).

On the other hand, the shape of the second avatar A2 is an anthropomorphic shape of a rabbit, and the second avatar A2 is given a second characteristic corresponding to the shape. The second characteristic is that it is smaller than a predetermined object placed in the virtual space VS, and that the corresponding user's motion in the real space RS is reflected in the avatar's motion in a short time (that is, the motion is reflected quickly).

Therefore, even if the first avatar A1 and the second avatar A2 are located at coordinates adjacent to each other in the virtual space VS, the sizes of the avatars A (that is, the height of the viewpoint) Therefore, the environment of the range that the first user U1 can look over as the first avatar A1 (that is, the visible object) and the environment of the range that the second user U2 can look over as the second avatar A2. will be different.

Further, even if the first user U1 and the second user U2 perform the same action in the physical space RS, the characteristic of the reflection speed of the action of the corresponding avatar A is different. The environment of the scenery change speed experienced by the first user U1 as A1 is different from the environment of the scenery change speed experienced by the second user U2 as the second avatar A2 (that is, the relative movement speed of the object). It will be.

Next, the avatar action control unit 32 of the server 3 determines the action of the first avatar A1 based on the action of the first user U1 and the characteristics given to the first avatar A1, and the action of the second user U2 and The action of the second avatar A2 is determined based on the characteristics assigned to the second avatar A2 (FIG. 3/STEP 102).

For the actions of the first user U1 and the second user U2 in the processing after STEP 102, actions recognized by the user action recognition unit 31 of the server 3 based on image data captured by the camera 2 are used.

Next, the avatar environment determination unit 33 of the server 3 determines the environment of the first avatar A1 based on the motion of the first avatar A1, and determines the environment of the second avatar A2 based on the motion of the second avatar A2. Determine (FIG. 3/STEP 103).

Specifically, the avatar environment determining unit 33 determines the environment of the change speed of the scenery in the virtual space VS based on the characteristics of the action process of the avatar A and the reflection speed of the action of the avatar A, and the characteristics of the size of avatar A, determine the environment of landscape that avatar A can overlook.

Next, the user environment determination unit 34 of the server 3 determines the environment to be recognized by the first user U1 based on the environment of the first avatar A1, and determines the environment to be recognized by the second user U2 as the environment of the second avatar A2. It is determined based on the environment (FIG. 3/STEP 104).

Specifically, the user environment determining unit 34 determines the image and sound of the virtual space VS representing the environment of the first avatar A1 as the environment to be recognized by the first user U1, and determines the environment to be recognized by the second user U2. , determine the images and sounds of the virtual space VS representing the environment of the second avatar A2.

Next, the HMD 4 worn by the user U outputs the determined environment (FIG. 3/STEP 105).

Specifically, the HMD 4 causes the monitor 40 mounted on the HMD 4 to display the determined image, and the speaker 41 mounted on the HMD 4 to generate the determined sound.

Next, the user action recognition unit 31 of the server 3 determines whether or not the first user U1 or the second user U2 has performed some action (FIG. 3/STEP 106).

If the first user U1 or the second user U2 performs some action (YES in STEP106), the process returns to STEP102, and the processes after STEP102 are executed again.

On the other hand, if the first user U1 or the second user U2 has not performed any action (NO in STEP 106), the server 3 determines whether or not it has recognized a signal instructing the end of the process (Fig. 3/ STEP 107).

If the signal instructing the end cannot be recognized (NO in STEP 107), the process returns to STEP 106, and the processes after STEP 106 are executed again.

On the other hand, if the signal instructing the end is recognized (YES in STEP 107), the VR system S1 ends the current process.

By the above process, as shown in FIG. 4, a plurality of objects including a first avatar A1 corresponding to the first user U1 and a second avatar A2 corresponding to the second user U2 are installed in the virtual space VS.

Then, the first user U1 and the second user U2 can visualize themselves via the corresponding first avatars A1 and second avatars A2 by the images displayed and the sounds generated by the HMDs 4 they wear. It comes to recognize that it exists in space VS and can move freely.

[Processing when synchronizing characteristics and after]
Next, with reference to FIGS. 2 and 5 to 7, processing to be executed when synchronizing characteristics and thereafter will be described.

First, prior to a specific description of this process, the background of the execution of this process will be described.

As described above, in the VR system S1, the first avatar A1 is given the first property, and the second avatar A2 is given the second property different from the first property.

Therefore, even if the first user U1 corresponding to the first avatar A1 and the second user U2 corresponding to the second avatar A2 perform the same action in the physical space RS, the action of the first avatar A1 and the second avatar The behavior (and thus the environment) of A2 will be different. As a result, the environment recognized by the first user U1 and the environment recognized by the second user U2 are different.

When the environments to be recognized are different in this way, a user may desire to experience the environment of a user other than himself, or a user may desire to allow a user other than himself to experience his own environment.

For example, in the present embodiment, the first avatar A1 is given a first property that it is larger than a predetermined object (for example, grass shown in FIG. 4) placed in the virtual space VS. The avatar A2 is given a second property that it is smaller than a predetermined object placed in the virtual space VS.

Therefore, even if the coordinates of the first avatar A1 are located in the vicinity of the coordinates of the second avatar A2, the object (for example, , the first object O1, which is a distant building object, cannot be visually recognized by the second avatar A2 (and by extension, the corresponding second user U2).

If such a situation occurs, the first user U1 corresponding to the first avatar A1 may desire to view the first object O1 viewed by the second user U2 in the same manner as the second user U2. Sometimes.

Therefore, in the VR system S1 of the present embodiment, in order to respond to such a request, the characteristics imparted to the first avatar A1 and the characteristics imparted to the second avatar A2 are processed by the processing described below. It is configured to be able to execute a process of matching the characteristics of the

In this process, first, the trigger event recognition unit 35 of the server 3 determines whether or not a trigger event has been recognized (FIG. 5/STEP 200).

Specifically, as shown in FIG. 5, the trigger event recognition unit 35 recognizes the action of the first avatar A1 corresponding to the first user U1 in the virtual space VS, and the action is defined in advance as the trigger event. It is determined whether or not it corresponds to the predetermined operation.

When a trigger event is recognized (YES in STEP 200), the trigger event recognition unit 35 determines whether or not the recognized trigger event is a predetermined trigger event (FIG. 5/STEP 201).

In the present embodiment, as shown in FIG. 6, the action of one of the first avatar A1 corresponding to the first user U1 and the second avatar A2 corresponding to the second user U2 touching the other is set as a predetermined trigger event. It is

If the trigger event is a predetermined trigger event (YES in STEP201), the avatar action control unit 32 synchronizes the characteristics of the first avatar A1 and the characteristics of the second avatar A2 (FIG. 5/STEP202).

Specifically, the characteristics of the first avatar A1 are maintained as the first characteristics, and the characteristics of the second avatar A2 are changed from the second characteristics to the same first characteristics as the first avatar A1.

In this embodiment, the characteristics of the avatar A are linked to the shape of the avatar A. Therefore, as shown in FIG. 7, when the process of STEP 202 is executed, the shape of the second avatar A2 changes from the shape of an anthropomorphic rabbit to the shape of an anthropomorphic turtle like the first avatar A1. .

On the other hand, if the trigger event is not the predetermined trigger event (NO in STEP 201), the avatar action control unit 32 controls the characteristics of the first avatar A1 and the second avatar A2 according to the type of the recognized trigger event. change at least one of the characteristics of (FIG. 5/STEP 203).

For example, the action of the first avatar A1 or the second avatar A2 pressing a switch-type object (not shown) is set as another trigger event, and the recognized trigger event is the other trigger event. If there is, the avatar action control unit 32 performs a process of once again giving the characteristics at the start of use of the VR system S1 to the avatar A that caused the other trigger event.

If the trigger event is not recognized (NO in STEP 200), the characteristics up to that point are maintained, and if the trigger event is recognized (YES in STEP 200), processing to change the characteristics of the avatar. (processing of STEP 202 or STEP 203), and then the avatar motion control unit 32 of the server 3 controls the motion of the first avatar A1 based on the motion of the first user U1 and the characteristics given to the first avatar A1. Then, based on the behavior of the second user U2 and the characteristics given to the second avatar A2, the behavior of the second avatar A2 is determined (FIG. 5/STEP 204).

Next, similar to the processing in STEP 103, the avatar environment determination unit 33 of the server 3 determines the environment of the first avatar A1 based on the motion of the first avatar A1, and based on the motion of the second avatar A2, The environment of the second avatar A2 is determined (FIG. 5/STEP 205).

At this time, if the process of synchronizing the characteristics of the avatar A has been executed, the characteristics of the size of the first avatar A1 and the second avatar A2 match. Like the second avatar A2, the first avatar A1 can visually recognize an object that could not be visually recognized due to its small size.

Next, similar to the processing in STEP 104, the user environment determining unit 34 of the server 3 determines the environment to be recognized by the first user U1 based on the environment of the first avatar A1, and determines the environment to be recognized by the second user U2. is determined based on the environment of the second avatar A2 (FIG. 5/STEP 206).

Next, similarly to the processing in STEP105, the HMD 4 worn by the user U outputs the determined environment (FIG. 5/STEP207).

Next, similarly to the processing in STEP 106, the user action recognition section 31 of the server 3 determines whether or not the first user U1 or the second user U2 has performed some action (FIG. 5/STEP 208).

If the first user U1 or the second user U2 performs some action (YES in STEP208), the process returns to STEP200, and the processes after STEP200 are executed again.

On the other hand, if the first user U1 or the second user U2 has not performed any action (NO in STEP 208), the server 3 has recognized a signal instructing the end of the process as in the process in STEP 107. It is determined whether or not (FIG. 5/STEP 209).

If the signal instructing the end cannot be recognized (NO in STEP 209), the process returns to STEP 208, and the processes after STEP 208 are executed again.

On the other hand, when the signal instructing the end is recognized (YES in STEP209), the VR system S1 ends the current process.

As described above, in the VR system S1 of the present embodiment, before a predetermined trigger event is recognized, the first characteristic is given to the first avatar A1, and based on the action of the first avatar A1, , determines the environment of the first avatar A1, gives a second characteristic different from the first characteristic to the second avatar A2, and based on the action of the second avatar A2, Determine A2's environment.

On the other hand, after the predetermined trigger event is recognized, the first characteristic is given to the first avatar A1, and the environment of the first avatar A1 is determined based on the action of the first avatar A1, Similarly, the first property is given to the second avatar A2, and the environment of the second avatar A2 is determined based on the behavior of the second avatar A2.

Accordingly, before the predetermined trigger event is recognized, the characteristics of the first avatar A1 corresponding to the first user U1 and the characteristics of the second avatar A2 corresponding to the second user U2 are different. there is

Therefore, the environment of the avatar A when the avatar A moves in response to the motion of the user U becomes independent as in the conventional case. As a result, the environment experienced by the user (in this embodiment, the types of objects that can be visually recognized) becomes independent as in the conventional case.

On the other hand, after the predetermined trigger event is recognized, the characteristics of the first avatar A1 corresponding to the first user U1 and the characteristics of the second avatar A2 corresponding to the second user U2 are both the same. .

Therefore, the environment of the avatar A when the avatar A moves in response to the motion of the user U matches or is similar. As a result, the environment experienced by the user U (in this embodiment, the types of visible objects) also match or are similar.

Therefore, according to the VR system S1, after the predetermined trigger event is recognized, the process of synchronizing the environment of the avatar A is executed, and the virtual space VS that is recognized by the first user U1 and the second user U2 is displayed. Since the environments become similar, the first user U1 and the second user U2 who have experienced different environments can now experience similar environments.

[Modification of First Embodiment]
Note that, in the present embodiment, an action in which one of the first avatar A1 corresponding to the first user U1 and the second avatar A2 corresponding to the second user U2 touches the other is set as a predetermined trigger event.

This is because when a predetermined action performed by one of the first avatar A1 and the second avatar A2 on the other is set as a predetermined trigger event, the user U on one avatar A performs the action on the other avatar A. This is because the user can intuitively understand that he or she himself can also experience the environment of the user U corresponding to the avatar A of .

However, in the first invention, the "predetermined action" that becomes a predetermined trigger event is not limited to the action of one of the first avatar and the second avatar touching the other. Any action performed by one of the second avatars relative to the other may be acceptable.

For example, the predetermined action is an action in which one avatar touches the other avatar, an action in which one avatar moves within a predetermined range relative to the other avatar, and one avatar moves in a complex space. When operating an object that exists in the virtual space that exists in the good.

Further, for example, as an avatar corresponding to the user, in addition to the avatar that serves as a reference for determining the environment to be recognized by the user, a ghost that acts in response to the user's action but does not serve as a reference may also be generated. . In such a case, the process of synchronizing the characteristics of the avatar may be executed even when the ghost performs a predetermined action.

Furthermore, the predetermined trigger event in the first invention does not necessarily have to be a predetermined action performed by one of the first avatar and the second avatar on the other of the first avatar and the second avatar. . Therefore, for example, when the user gives an instruction to change the characteristics via an interface provided separately, the fact that the instruction is given may be set as the predetermined trigger event.

Further, in the present embodiment, when a predetermined trigger event is recognized, the characteristics of the second avatar A2 are synchronized with the first characteristics given to the first avatar A1, and the shape of the second avatar A2 is changed to the first characteristic. 1 avatar A1 is changed to a shape similar to that of A1. This is because the characteristics given to avatar A are linked to the shape of avatar A in this embodiment.

However, the first invention only needs to synchronize the characteristics of the avatar when a predetermined trigger event is recognized, and the shape of the avatar does not necessarily have to be synchronized with the characteristics. As such, only avatar features (eg, size only) may be synchronized when certain trigger events are recognized.

Further, in this embodiment, when a predetermined trigger event is recognized, the characteristics of the second avatar A2 are synchronized with the first characteristics given to the first avatar A1. That is, they have the same characteristics.

However, in the first invention, the process of synchronizing properties is not limited to changing the properties of the second avatar in that way. Therefore, for example, the characteristics of the first avatar may be synchronized with the second characteristics given to the second avatar.

In addition, in the first invention, the process of synchronizing the characteristics is not limited to the process of completely matching the characteristics. Therefore, for example, it may be a process of synchronizing some of a plurality of characteristics. Specifically, out of the movable speed and the size, only the size may be synchronized. Alternatively, for example, the process may be a process of approximating values that define characteristics. Specifically, it may be a process of bringing the values of movable velocities close to each other.

Furthermore, if there are multiple types of processing for synchronizing characteristics, a trigger event corresponding to each processing may be defined. For example, when the second avatar touches the first avatar with the right hand, the characteristics of the second avatar are synchronized with the characteristics given to the first avatar, and when the second avatar touches the first avatar with the left hand, Alternatively, the properties of the first avatar may be synchronized with the properties imparted to the second avatar.

In the above embodiment, size, movable speed, movable range, etc. are adopted as avatar characteristics, but avatar characteristics are not limited to such configurations, and , the characteristics that define the behavior of the avatar corresponding to the user.

Therefore, for example, a visible light range may be specified as a function of the avatar. Specifically, one avatar is set to be able to see only light in the normal visible light wavelength band, and the other avatar is set to be able to see infrared light in addition to light in the normal visible light wavelength band. If so, each avatar's environment (eg, how objects look, etc.) will be different.

[Second embodiment]
A VR system S2, which is a virtual space sensation system according to the second embodiment, will be described below with reference to FIGS. 8 to 13. FIG.

Note that the VR system S2 of the present embodiment is the same as the VR system S1 of the first embodiment except that only one type of characteristic is given to the avatar and that a plurality of virtual spaces are generated. It has a configuration of Therefore, in the following description, the same reference numerals are given to the same or corresponding configurations as those of the VR system S1 of the first embodiment, and detailed description thereof will be omitted.

[Configuration of processing unit]
The configuration of the processing unit included in the server 3 will be described in detail with reference to FIG.

The server 3 is composed of one or a plurality of electronic circuit units including CPU, RAM, ROM, interface circuits, and the like. As shown in FIG. 8, the server 3 includes a virtual environment generation unit 30, a user action recognition unit 31, and an avatar action control unit 32 as functions (processing units) realized by the implemented hardware configuration or programs. , an avatar environment determination unit 33 , a user environment determination unit 34 , and a trigger event recognition unit 35 .

The virtual environment generator 30 has a virtual space generator 30a and an avatar generator 30b.

The virtual space generator 30a generates a first virtual space VS1 corresponding to the physical space RS (see FIG. 1) in which the user U exists, and a second virtual space VS1 independent of the first virtual space VS1 corresponding to the physical space RS. Create space VS2.

Specifically, the virtual space generation unit 30a becomes the background of the first virtual space VS1 and the second virtual space VS2 (hereinafter collectively referred to as "virtual space VS") and objects existing in the virtual space VS. Generate images and sounds associated with those images.

The avatar generation unit 30b generates a first avatar A1 corresponding to the first user U1 and a second ghost G2 corresponding to the second user U2 in the first virtual space VS1, and generates a second ghost G2 corresponding to the second user U2 in the second virtual space VS2. A second avatar A2 corresponding to the second user U2 and a first ghost G1 corresponding to the first user U1 are generated (see FIG. 9, etc.).

Here, a "ghost" is one of the avatars that act corresponding to the user, and does not affect the environment recognized by the corresponding user. In this embodiment, the ghost G is used to indicate the existence, actions, etc. of the corresponding user U to the users U other than the user U corresponding to the ghost G.

The first avatar A1 and second avatar A2 (hereinafter collectively referred to as "avatar A"), and the first ghost G1 and second ghost G2 (hereinafter collectively referred to as "ghost G") are , operate in the first virtual space VS1 or the second virtual space VS2 corresponding to the action of the corresponding user U in the physical space RS.

The virtual space VS in which the first avatar A1 and the second avatar A2 exist may change during the use of the VR system S2. For example, it is changed when a predetermined trigger event, which will be described later, is recognized. Also, the first ghost G1 and the second ghost G2 may be erased along with the change.

Information indicating which user U the ghost G corresponds to is added to the image of the ghost G in this embodiment. This is so that the user U can easily grasp which user other than himself/herself the ghost G corresponds to.

Specifically, by adopting a translucent shape of the corresponding avatar A as the shape of the ghost G, information as to which avatar A the ghost G corresponds to (and furthermore which user U corresponds to information on whether the

However, the "information indicating which user the service corresponds to" is not limited to such indirect information. As such, it may be direct information such as, for example, a message that is displayed all the time or at the user's request.

The avatar action control unit 32 gives characteristics to the avatar A generated by the avatar generation unit 30b. In this embodiment, the properties are assigned when the use of the VR system S2 is started (for example, when the type of avatar A to be used is selected).

The avatar motion control unit 32 also recognizes the posture of the user U in the physical space RS recognized by the user posture recognition unit 31a, the coordinates of the user U in the physical space RS recognized by the user coordinate recognition unit 31b, and the assigned Based on the characteristics, the actions of the avatar A and the ghost G corresponding to the user U in the virtual space VS (for example, movement of coordinates, change of posture including orientation, etc.) are determined.

Although not adopted in the VR system S2 of the present embodiment, when characteristics unique to the virtual space are set, the avatar motion control unit controls the user's motion and the characteristics of the avatar as well as the virtual space. may be configured to refer to the characteristics of

For example, if the virtual space imitates outer space, the characteristics of the virtual space are set so that the effect of gravity on the avatar in the virtual space is different from the effect of gravity on the user in the virtual space. Sometimes. In such a case, when the user's motion is reflected in the avatar, the characteristics of the virtual space are reflected.

The avatar environment determining unit 33 determines the environment of the avatar A in the virtual space VS based on the result of the action of the avatar A (for example, coordinates, posture, etc. at that time).

Here, the avatar's "environment" in the present embodiment refers to what affects the avatar in the virtual space. For example, the avatar's environment is the type of existing virtual space itself (whether it is on the ground or in the water).

[Process to be executed]
Next, with reference to FIGS. 8 to 13, processing executed by the VR system S2 when allowing the user U to experience the virtual space VS using the VR system S2 will be described.

[Processing during normal use]
First, with reference to FIGS. 8 to 10, the processing executed by the VR system S2 at the start of use and in normal use (that is, in a state in which a predetermined trigger event described later is not recognized) will be described.

In this process, first, the virtual environment generation unit 30 of the server 3 generates a first virtual space VS1 and a second virtual space VS2, a first avatar A1 and a second avatar A2, and a first ghost G1 and a second ghost G2. is generated (FIG. 9/STEP 300).

Specifically, as shown in FIG. 10, the virtual space generation unit 30a of the virtual environment generation unit 30 generates an image serving as a background of the virtual space VS and various objects existing in the virtual space VS. In this embodiment, a background image and objects are generated so that the first virtual space VS1 is a virtual space imitating the sea, and a background image and objects are generated so that the second virtual space VS2 is a virtual space imitating mountains. images and objects are generated.

In addition, the avatar generation unit 30b of the virtual environment generation unit 30 generates a first avatar A1 corresponding to the first user U1 and a second ghost G2 corresponding to the second user U2 in the first virtual space VS1. A second avatar A2 corresponding to U2 and a first ghost G1 corresponding to the first user U2 are generated in the second virtual space VS2.

Next, the avatar action control section 32 of the server 3 gives the first property to the first avatar A1 and gives the second property to the second avatar A2 (FIG. 9/STEP 301).

In this embodiment, the characteristics imparted to the first avatar A1 and the characteristics imparted to the second avatar A2 are the same, and are not changed thereafter. However, the characteristics given to the first avatar A1 and the second avatar A2 may be different from each other as in the first embodiment and may be changed thereafter.

Next, the avatar motion control unit 32 of the server 3 determines the motion of the first avatar A1 and the first ghost G1 based on the motion of the first user U1 and the characteristics given to the first avatar A1, The actions of the second avatar A2 and the second ghost G2 are determined based on the action of the user U2 and the characteristics given to the second avatar A2 (FIG. 9/STEP302).

Next, the avatar environment determination unit 33 of the server 3 determines the environment of the first avatar A1 based on the motion of the first avatar A1 and the situation of the first virtual space VS1, and the motion of the second avatar A2 and the second virtual space VS1. The environment of the second avatar A2 is determined based on the situation of the virtual space VS2 (FIG. 9/STEP303).

Next, the user environment determination unit 34 of the server 3 determines the environment to be recognized by the first user U1 based on the environment of the first avatar A1, and determines the environment to be recognized by the second user U2 as the environment of the second avatar A2. It is determined based on the environment (FIG. 9/STEP 304).

Specifically, the user environment determination unit 34 determines the image and sound of the first virtual space VS1 representing the environment of the first avatar A1 as the environment to be recognized by the first user U1, and makes the second user U2 recognize the environment. As the environment, the image and sound of the second virtual space VS2 representing the environment of the second avatar A2 are determined.

Next, the HMD 4 worn by the user U outputs the determined environment (FIG. 9/STEP 305).

Specifically, the HMD 4 causes the monitor 40 mounted on the HMD 4 to display the determined image, and the speaker 41 mounted on the HMD 4 to generate the determined sound.

Next, the user action recognition unit 31 of the server 3 determines whether or not the first user U1 or the second user U2 has performed some action (FIG. 9/STEP 306).

If the first user U1 or the second user U2 performs some action (YES in STEP306), the process returns to STEP302, and the processes after STEP302 are executed again.

On the other hand, if the first user U1 or the second user U2 has not performed any action (NO in STEP 306), the server 3 determines whether or not it has recognized a signal instructing the end of the process (Fig. 9/ STEP 307).

If the signal instructing the end cannot be recognized (NO in STEP 307), the process returns to STEP 306, and the processes after STEP 306 are executed again.

On the other hand, if the signal instructing the end is recognized (YES in STEP307), the VR system S2 ends the current process.

By the above processing, as shown in FIG. 10, a plurality of objects including a first avatar A1 corresponding to the first user U1 and a second ghost G2 corresponding to the second user U2 are installed in the first virtual space VS1. , in the second virtual space VS2, a plurality of objects including a second avatar A2 corresponding to the second user U2 and a first ghost G1 corresponding to the first user U1 are installed.

Then, the first user U1 and the second user U2, through the corresponding first avatar A1 and second avatar A2, respectively, by the image displayed and the sound generated by the HMD 4 worn by the first user U1 and the second user U2, become the second user. It exists in the first virtual space VS1 or the second virtual space VS2 and recognizes that it can move freely.

At this time, one of the first user U1 and the second user U2 can communicate with the other of the first user U1 and the second user U2 through message transmission or the like. As a result, the first user U1 and the second user U2 are mutually aware that they are using the VR system S2 at the same time, although they are in different virtual spaces VS.

[Processing when synchronizing virtual space and thereafter]
Next, with reference to FIGS. 8 and 11 to 13, processing to be executed when and after synchronizing the virtual space VS will be described.

First, prior to a specific description of this process, the background of the execution of this process will be described.

As described above, in this embodiment, when the use of the VR system S2 is started, the first avatar A1 exists in the first virtual space VS1, and the second avatar A2 exists in the second virtual space VS2. Existing.

When the virtual space VS existing for each avatar A is different, for example, when the user U communicates (specifically, a conversation beyond the type of space), the first user A desire may arise for one of U1 and second user U2 to experience the environment of the other, or for one to allow the other to experience its own environment.

For example, in the present embodiment, the first user U1, through the first avatar A1, can create an object existing only in the first virtual space VS1 (for example, the second object O2, which is a dolphin swimming in the sea). ), the second user U2 may not be able to see the object.

If such a situation occurs, the first user U1 informs the second user U2 of the existence of the second object O2 by using a message function or the like. A desire to visually recognize the visually recognizing object in the same way as the first user U1 may arise.

Therefore, in the VR system S2 of the present embodiment, in order to respond to such a request, the first virtual space VS1 in which the first avatar A1 exists and the second avatar are created by the processing described below. It is configured to be able to execute a process of synchronizing with the second virtual space VS2 in which A2 exists.

In this process, first, the trigger event recognition section 35 of the server 3 determines whether or not a predetermined trigger event has been recognized (FIG. 11/STEP 400).

Specifically, the trigger event recognition unit 35 recognizes the motion of the avatar A corresponding to the user U in the virtual space VS, and determines whether or not the motion corresponds to a predetermined motion defined in advance as a predetermined trigger event. to judge. In this embodiment, as shown in FIG. 12, the action of the avatar A touching the ghost G is set as a predetermined trigger event.

When a predetermined trigger event is recognized (YES in STEP 400), the avatar generation unit 30b of the virtual environment generation unit 30 of the server 3 generates the first virtual space VS1 in which the first avatar A1 exists and the second avatar A2. is synchronized with the second virtual space VS2 in which is present (FIG. 11/STEP 401).

Specifically, first, the avatar generator 30b erases the second ghost G2 and generates the second avatar A2 in the first virtual space VS1. Next, the avatar generator 30b erases the second avatar A2 and the first ghost G1 in the second virtual space VS2. As a result, as shown in FIG. 13, both the first avatar A1 and the second avatar A2 are present in the first virtual space VS1.

On the other hand, the second virtual space VS2 is in a state where the avatar A and the ghost G do not exist after the synchronizing process is executed. Although this is not adopted in the VR system S2 of this embodiment, when a trigger event other than a predetermined trigger event is recognized (for example, when avatar A presses a switch-type object), This is because the avatar A may be generated again in the second virtual space VS2.

Although not adopted in the VR system S2 of this embodiment, by changing the environment of the second virtual space VS2 to the environment of the first virtual space VS1, the first virtual space VS1 and the second virtual space VS2 can be synchronized. In that case, the process of generating the second avatar A2 and the process of erasing the second ghost G2 in the first virtual space VS1, and the process of generating the first avatar A1 and erasing the first ghost G1 in the second virtual space VS2. The process may or may not be performed.

After executing the process of synchronizing the virtual space VS (the process of STEP 401), the avatar action control unit 32 of the server 3 performs the first The action of the avatar A1 is determined, and the action of the second avatar A2 is determined based on the action of the second user U2 and the characteristics given to the second avatar A2 (FIG. 11/STEP 402).

Next, the avatar environment determination unit 33 of the server 3 determines the environment of the first avatar A1 based on the motion of the first avatar A1 and the situation of the first virtual space VS1, and the motion of the second avatar A2 and the first virtual space VS1. The environment of the second avatar A2 is determined based on the situation of the virtual space VS1 (FIG. 11/STEP 403).

On the other hand, if the predetermined trigger event is not recognized (NO in STEP400), similar to the process in STEP302, the avatar action control unit 32 of the server 3 applies the action of the first user U1 and the first avatar A1. Based on the characteristics given, the behavior of the first avatar A1 and the first ghost G1 is determined, and based on the behavior of the second user U2 and the characteristics given to the second avatar A2, the second avatar A2 and the second ghost Determine the operation of G2 (FIG. 11/STEP 404).

Next, the avatar environment determination unit 33 of the server 3 determines the environment of the first avatar A1 based on the motion of the first avatar A1 and the situation of the first virtual space VS1, and the motion of the second avatar A2 and the second virtual space VS1. The environment of the second avatar A2 is determined based on the situation of the virtual space VS2 (FIG. 11/STEP405).

After executing the process of determining the environment of avatar A (the process of STEP 403 or the process of STEP 405), the user environment determination unit 34 of the server 3 determines the environment to be recognized by the first user U1 as the environment of the first avatar A1. The environment is determined based on the environment, and the environment to be recognized by the second user U2 is determined based on the environment of the second avatar A2 (FIG. 11/STEP 406).

Specifically, the user environment determination unit 34 determines the image and sound of the first virtual space VS1 representing the environment of the first avatar A1 as the environment to be recognized by the first user U1, and makes the second user U2 recognize the environment. As the environment, the image and sound of the first virtual space VS1 representing the environment of the second avatar A2, or the image or sound of the second virtual space VS2 is determined.

Next, the HMD 4 worn by the user U outputs the determined environment in the same manner as the process in STEP305 (FIG. 11/STEP407).

Next, the user action recognition unit 31 of the server 3 determines whether or not the first user U1 or the second user U2 has performed some action (FIG. 11/STEP 408).

If the first user U1 or the second user U2 performs some action (YES in STEP408), the trigger event recognition unit 35 performs the process of synchronizing the first virtual space VS1 and the second virtual space VS2 (STEP401 process)) has been executed (FIG. 11/STEP 409).

Specifically, the trigger event recognizing unit 35 determines based on whether or not a predetermined trigger event serving as a key to executing the process of synchronizing the first virtual space VS1 and the second virtual space VS2 has been recognized. , to determine whether the process has been executed.

If the synchronization process has not been executed (NO in STEP409), the process returns to STEP400, and the processes after STEP400 are executed again.

On the other hand, if the synchronizing process has been executed (YES in STEP409), the process returns to STEP402, and the processes after STEP402 are executed again.

On the other hand, if the first user U1 or the second user U2 has not performed any action (NO in STEP 408), the server 3 determines whether or not it has recognized a signal instructing the end of the process (Fig. 11/STEP 410).

If the signal instructing the end cannot be recognized (NO in STEP 410), the process returns to STEP 408, and the processes after STEP 408 are executed again.

On the other hand, when the signal instructing the end is recognized (YES in STEP410), the VR system S2 ends the current process.

As described above, in the VR system S2 of the present embodiment, before a predetermined trigger event is recognized, the first avatar A1 is generated in the first virtual space VS1, and the environment of the first avatar A1 is determined. At the same time, the second avatar A2 is generated in the second virtual space VS2, and the environment of the second avatar A2 is determined.

On the other hand, after a predetermined trigger event is recognized, the first avatar A1 and the second avatar A2 are generated in the first virtual space VS1 to determine the environment of the first avatar A1 and the second avatar A2.

As a result, before the predetermined trigger event is recognized, the virtual space in which the first avatar A1 corresponding to the first user U1 exists (for example, the first virtual space VS1 in this embodiment) and the second user U2 is independent of the virtual space (for example, the second virtual space VS2 in this embodiment) in which the second avatar A2 corresponding to . Therefore, the environment of Avatar A becomes independent as before. As a result, the environment experienced by the user U becomes independent as in the conventional case.

On the other hand, after the predetermined trigger event is recognized, the virtual space in which the first avatar A1 corresponding to the first user U1 exists and the virtual space in which the second avatar A1 corresponding to the second user U2 exists are: It becomes the same virtual space (for example, the first virtual space VS1 in this embodiment). Therefore, Avatar A's environment will match or be similar. As a result, the environment experienced by the user U also matches or is similar.

Therefore, according to the VR system S2, after the predetermined trigger event is recognized, the process of synchronizing the environment of the avatar A is executed, and the virtual space VS recognized by the first user U1 and the second user U2 is displayed. Since the environments become similar, the first user U1 and the second user U2 who have experienced different environments can now experience similar environments.

[Modification of Second Embodiment]
Note that, in the present embodiment, the action of the avatar A touching the ghost G is set as a predetermined trigger event.

This is because when a predetermined action performed by avatar A on ghost G is set as a predetermined trigger event, user U corresponding to avatar A, as a result of performing the action, will This is because they will be able to intuitively understand that they can also experience the environment themselves.

However, in the second invention, the "predetermined action" that is a predetermined trigger event is not limited to the action of the avatar touching the ghost, but any action that the avatar performs with reference to the ghost. good.

For example, the predetermined action is an action in which the avatar touches a ghost, an action in which the avatar moves within a predetermined range based on the ghost, an avatar operating an object existing in the virtual space, An action of selecting a ghost as the target (for example, an action of photographing a ghost with a camera-type object, etc.) may be used.

Furthermore, the trigger event in the second invention must be a predetermined action performed by the avatar corresponding to one of the first user and the second user to the ghost corresponding to the other of the first user and the second user. no. Therefore, for example, when the user gives an instruction to change the characteristics via an interface provided separately, the fact that the instruction is given may be set as the trigger event.

Further, in the above embodiment, unlike the VR system S1 in the first embodiment, only one type of characteristic is given to the avatar A. However, the virtual space bodily sensation system of the second invention is not limited to such a configuration. It may be configured such that a process for synchronizing is also executed.

[Third embodiment]
The MR system S3, which is a complex spatial sensation system according to the third embodiment, will be described below with reference to FIGS. 14 to 20. FIG.

The MR system S3 includes a third user U3 and a fourth user U4 (first user and second user in the third invention; hereinafter collectively referred to as In this case, it is referred to as “user U”). This is for recognition (see FIG. 17, etc.).

In this embodiment, two users are assumed for easy understanding. However, the complex spatial sensation system of the third invention is not limited to such a configuration, and the number of users may be three or more.

[Schematic configuration of the system]
First, a schematic configuration of the MR system S3 will be described with reference to FIG.

As shown in FIG. 14, the MR system S3 includes a user U existing in the physical space RS, a camera 5 that captures the physical space RS, and a server 6 that determines images and sounds of the complex space MS (see FIG. 4, etc.). , and a head-mounted display (hereinafter referred to as “HMD 7”) that allows the user to recognize the determined image and sound.

In the MR system S3, the camera 5, the server 6, and the HMD 7 are capable of mutually transmitting and receiving information wirelessly, such as through the Internet network, public lines, and short-range wireless communication. However, any one of them may be configured to transmit and receive information to and from each other by wire.

The camera 5 is installed in such a way that the user U in the real space RS in which the user U exists can photograph the operable range (that is, the range in which the user U can move the coordinates, change the posture including the orientation, etc.) from multiple directions. there is

The server 6 recognizes feature points on the user U's body or the like from the image captured by the camera 5, and recognizes the coordinates and posture of the user U based on the positions of the recognized feature points in the physical space RS. Also, the server 6 recognizes the feature points of the physical space RS from the image captured by the camera 5, and recognizes the coordinates and orientation of a real object (for example, a bookshelf) existing in the physical space RS.

Further, the server 6 determines images and sounds of the complex space MS that the user U is made to recognize, based on the coordinates and posture of the user U and feature points in the physical space RS.

The HMD 7 is an environment output device that allows the user to recognize the environment of the complex space MS. The HMD 7 is worn on the user's U head. The HMD 7 includes a monitor 70 for allowing the user U to recognize the image of the composite space MS determined by the server 6, and a speaker for allowing the user U to recognize the sound of the composite space MS determined by the server 6. 71 (see FIG. 15).

When experiencing the complex space MS using the MR system S3, the user U is made to recognize images and sounds of a virtual object corresponding to the experienced complex space MS together with the images and sounds of the real space RS. Note that the virtual object is configured to perform a predetermined action (for example, movement of coordinates, etc.) according to the user's U action.

The complex spatial sensation system of the third invention is not limited to such a configuration. For example, when a motion capture device is used, in addition to the configuration described above, a configuration in which the number and arrangement of cameras are different from those shown in the drawings may be used. Also, in order to recognize at least one of the feature points of the user and the feature points of the real space, a marker attached to the user or a real object existing in the real space may be used.

Also, for example, instead of the motion capture device, a device that recognizes only the coordinates of the user may be used. Specifically, for example, a sensor such as GPS may be installed in the HMD, and the user's coordinates, posture, etc. may be recognized based on the output from the sensor. Also, such a sensor may be used in combination with the motion capture device as described above.
[Configuration of processing unit]
Next, using FIG. 15, the configuration of the processing unit provided in the server 6 will be described in detail.

The server 6 is composed of one or more electronic circuit units including a CPU, RAM, ROM, interface circuits, and the like. As shown in FIG. 15, the server 6 includes a real space recognition unit 60, a complex environment generation unit 61, and a user action recognition unit 62 as functions (processing units) realized by implemented hardware configurations or programs. , an object control unit 63 , a user environment determination unit 64 , and a trigger event recognition unit 65 .

The physical space recognition unit 60 recognizes the image data of the physical space RS captured by the camera 5, and recognizes the situation of the physical space RS based on the image data. The situation of the physical space RS is, for example, the coordinates and attitudes of other users U and real objects existing in the physical space RS.

The complex environment generator 61 has an object generator 61a and a complex space generator 61b.

The object generation unit 61a generates a virtual object that exists in the complex space MS and is recognized by the user U (see FIG. 17, etc.). The virtual objects include a third object O3 existing in a first complex space MS1 and a fourth object O4 existing in a second complex space MS2 (the first object and the second object in the third invention). included.

At least a part of the third object O3 and the fourth object O4 (hereinafter collectively referred to as "object O") is configured to move according to the user U's motion in the complex space MS.

The composite space generation unit 61b makes the third object O3 exist in the image of the physical space RS recognized by the physical space recognition unit 60 to generate the first composite space MS1. In addition, the composite space generation unit 61b makes the fourth object O4 exist in the image of the physical space RS recognized by the physical space recognition unit 60 to generate the second composite space MS2.

The user action recognition unit 62 recognizes image data of the user U captured by the camera 5, and recognizes actions of the user U in the physical space RS based on the image data. The user action recognition section 62 has a user posture recognition section 62a and a user coordinate recognition section 62b.

The user posture recognition unit 62a extracts feature points such as the body of the user U from input image data of the user U, and recognizes the posture including the orientation of the user U based on the extraction result.

The user coordinate recognition unit 62b extracts the result of extracting feature points such as the body of the user U extracted from the input image data of the user U, and the state of the real space RS recognized by the real space recognition unit 60 (for example, the actual , etc.), the coordinates of the user U are recognized.

The object control unit 63 determines the position of the user U based on the posture of the user U in the physical space RS recognized by the user posture recognition unit 62a and the coordinates of the user U in the physical space RS recognized by the user coordinate recognition unit 62b. to control the motion of the object O existing in the complex space MS experienced by .

For example, the object control unit 63 moves the coordinates of the object O in the composite space MS when the user U performs an action to move the object O. FIG.

The user environment determining unit 64 determines the environment of the complex space MS to be recognized by the user U through the monitor 70 and the speaker 71 of the HMD 7 .

Here, the “environment to be recognized” by the user refers to the environment of the complex space that is recognized by the user through the five senses. For example, the user's environment is an image of an object existing in a complex space recognized by the user, a sound generated based on the object, and the like.

In this embodiment, the user environment determining unit 64 recognizes the motion of the user U and the type of the composite space MS that the user U is allowed to experience, and based on the motion of the user U, determines which region of the composite space MS the user U Decide whether you want to experience it.

The trigger event recognition unit 65 recognizes that a predetermined trigger event has occurred when a condition predetermined by the system designer is satisfied.

Here, the "trigger event" may be one that the user is unaware of. Therefore, the trigger event includes, for example, events caused by the user's actions such as the user performing a predetermined action in the real space, and events not caused by the user's actions such as the passage of a predetermined time.

It should be noted that each processing unit that constitutes the complex spatial sensation system of the third invention is not limited to the configuration as described above.

For example, part of the processing unit provided in the server 6 in this embodiment may be provided in the HMD 7 . Alternatively, a plurality of servers may be used, or the servers may be omitted and the CPUs mounted on the HMDs may cooperate with each other. Also, a speaker other than the speaker mounted on the HMD may be provided. In addition to devices that affect sight and hearing, devices that affect smell and touch, such as generating smells, wind, etc., according to a complex space may also be included.

[Process to be executed]
Next, with reference to FIGS. 15 to 20, processing executed by the MR system S3 when allowing the user U to experience the complex space MS using the MR system S3 will be described.

[Processing during normal use]
First, with reference to FIGS. 15 to 17, processing executed by the MR system S3 at the start of use and in normal use (that is, in a state in which a predetermined trigger event to be described later is not recognized) will be described.

In this process, first, the composite environment generator 61 of the server 6 generates the first composite space MS1 and the second composite space MS2 (FIG. 16/STEP 500).

Specifically, as shown in FIG. 17, first, the object generation unit 61a of the composite environment generation unit 61 generates a third object O3 to exist in the first composite space MS1 and a third object O3 to exist in the second composite space MS2. 4 Create an object O4.

In the present embodiment, the third object O3 is a potted flower object (hereinafter simply referred to as "flower O30") whose coordinates and orientation are controlled according to the motion of the user U who recognizes the first complex space MS1. ) and a first message board O31 added to another user U. The first message board O31 displays the status of the user U to whom the first message board O31 is added (for example, the type of complex space MS being experienced, etc.).

Further, in the present embodiment, the fourth object O4 is a book object (hereinafter simply referred to as "book O40" whose coordinates and orientation are controlled according to the motion of the user U who is aware of the second composite space MS2. ) and a second message board O41 added to another user U. The second message board O41 displays the status of the user U to whom the second message board O41 is attached (for example, the type of complex space MS being experienced).

After that, the complex space generation unit 61b of the complex environment generation unit 61 superimposes the third object O3 on the image of the physical space RS recognized by the physical space recognition unit 60 to generate the first complex space MS1. The composite space generation unit 61b also superimposes the fourth object O4 on the image of the physical space RS recognized by the physical space recognition unit 60 to generate the second composite space MS2.

Next, the user environment determining unit 64 of the server 6 determines the composite space MS to be recognized by the third user U3 and the composite space MS to be recognized by the fourth user U4 (FIG. 16/STEP 501).

Specifically, the user environment determining unit 64 selects the composite space MS to be recognized by the user U based on an instruction from the user U input via an interface (not shown) provided separately.

In this embodiment, it is assumed that the third user U3 has selected to recognize the first composite space MS1, and the fourth user U4 has selected to recognize the second composite space MS2.

In addition, in the third invention, the method of determining the composite space to be recognized by the user is not limited to such a configuration, and may be appropriately set by the system designer. Therefore, for example, the complex space to be recognized by the user is automatically determined according to the characteristics of the user input in advance (physical characteristics such as the user's age, characteristics such as the purpose of use of the complex spatial sensation system, etc.). It may be configured as

Next, the user environment determining unit 64 of the server 6 makes the third user U3 recognize the first composite space based on the state of the third user U3 and the state of the first composite space MS1 at the start of use of the MR system MS3. The environment in MS1 is determined, and the environment in the second complex space MS2 to be recognized by the fourth user U4 is determined based on the state of the fourth user U4 and the situation of the second complex space MS2 at the start of use of the MR system MS3. (FIG. 16/STEP 502).

Specifically, the user environment determining unit 64, as the environment to be recognized by the third user U3, based on the coordinates and orientation of the physical space RS of the third user U3, creates an image of a region to be recognized in the physical space RS, The image of the third object O3 in that area and the accompanying sound are determined.

In addition, the user environment determination unit 64, as the environment to be recognized by the fourth user U4, based on the coordinates and posture of the physical space RS of the fourth user U4, the image of the region to be recognized in the physical space RS, The image of the fourth object O4 and the accompanying sound are determined.

Next, the HMD 7 worn by the user U outputs the determined environment (FIG. 16/STEP 503).

Specifically, the HMD 7 causes the monitor 70 mounted on the HMD 7 to display the determined image, and the speaker 71 mounted on the HMD 7 to generate the determined sound.

Next, the user action recognition section 31 of the server 6 determines whether or not the third user U3 or the fourth user U4 has performed some action (FIG. 16/STEP 504).

If the user U has made an action (YES in STEP 504), the object control unit 63 of the server 6 determines whether or not the action of the user U is to operate the object O (FIG. 16/STEP 505).

For example, whether the action of the user U is such that the object control unit 63 moves the object O existing in the composite space MS in which the user U exists. to judge. Specifically, the object control unit 63 determines whether or not the action of the user U corresponds to the predetermined action with respect to the object O.

If the action of the user U is to operate the object O (YES in STEP505), the object control unit 63 controls the object O according to the type of action of the user U (FIG. 16/STEP506). .

For example, when the action of the user U is such as to move the object O, the object control unit 63 changes the coordinates of the object O according to the content of the action.

After executing the process of controlling the object O (the process of STEP 506), or when the action of the user U is not the action of operating the object O (NO in STEP 505), the user environment determination unit 64 determines Based on the state of the third user U3 and the situation of the first composite space MS1 at that time, the environment in the first composite space MS1 to be recognized by the third user U3 is determined, and the state of the fourth user U4 at that time and the second composite space Based on the situation of the space MS2, the environment in the second composite space MS2 to be recognized by the fourth user U4 is determined (FIG. 16/STEP 507).

Next, the HMD 7 executes the same processing as in STEP503 and outputs the determined environment (FIG. 16/STEP508).

After executing the process of outputting the determined environment (the process of STEP 508), or when the user U did not act (NO in STEP 504), the server 6 recognized a signal instructing the end of the process. (FIG. 16/STEP 509).

If the signal instructing the end cannot be recognized (NO in STEP 509), the process returns to STEP 504, and the processes after STEP 504 are executed again.

On the other hand, if the MR system S3 recognizes the signal instructing the end (YES in STEP509), the MR system S3 ends the current process.

As a result of the above processing, as shown in FIG. 17, the third user U3 can create a first composite space MS1 in which the third object O3 exists in the physical space RS by means of the image displayed and the sound generated by the HMD 4 worn by the third user U3. is recognized, and the third object O3 can be manipulated in the first composite space MS1.

Further, the fourth user U4 recognizes the second composite space MS2 in which the fourth object O4 exists in the physical space RS from the image displayed and the sound generated by the HMD 4 worn by the user, and recognizes the second composite space MS2. , the fourth object O4 can be operated.

At this time, since the composite space MS recognized by one of the third user U3 and the fourth user U4 is generated based on the physical space RS, the third user existing in the physical space RS The other of user U3 and fourth user U4 is also present.

Then, as described above, the other of the third user U3 and the fourth user U4 is provided with the first message board, which is an object indicating the type of the complex space MS experienced by the third user U3 or the fourth user U4. O31 or a second message board O41 is added.

Therefore, the third user U3 and the fourth user U4 can mutually recognize that they are using the MR system S3 at the same time and, if so, which complex space MS they are experiencing. It's becoming

[Processing when Synchronizing Complex Spaces and Afterwards]
Next, with reference to FIGS. 14 and 18 to 20, processing to be executed when synchronizing the complex space MS and after that will be described.

First, prior to a specific description of this process, the background of the execution of this process will be described.

As described above, in this embodiment, when the MR system S3 is started to be used, the third user U3 is experiencing the first composite space MS1, and the fourth user U4 is experiencing the second composite space MS2. I feel it.

When the complex space MS experienced by each user U is different, for example, when the users U communicate (specifically, a conversation, etc.), the third user U3 and the fourth user U4 A desire may arise for one to experience the environment of the other, or for one to allow the other to experience its own environment.

For example, in the present embodiment, it is assumed that the third user U3, who is experiencing the first complex space MS1, arranges flowers O30, which are virtual objects, on a shelf that actually exists in the physical space RS. At that time, since the fourth user U4 who is experiencing the second composite space MS2 is experiencing the second composite space MS2, the shelves that exist in common in the first composite space MS1 and the second composite space MS2 can be recognized, the flower O30, which is an object of the first composite space MS1 arranged there, cannot be recognized.

In such a case, as a result of the third user U3 communicating to the fourth user U4 through conversation or the like that he would like the fourth user U4 to see the state of the flowers O30 that he has arranged, There may be a desire to visually recognize the flower O30 holding the flower O30 in the same manner as the third user U3.

Therefore, in the MR system S3 of the present embodiment, in order to respond to such a request, the processing described below is performed so that the first complex space MS1 experienced by the third user U3 and the fourth space It is configured to be able to execute a process of synchronizing with the second complex space MS2 experienced by the user U4.

In this process, first, the trigger event recognition section 65 of the server 6 determines whether or not a predetermined trigger event has been recognized (FIG. 18/STEP 600).

Specifically, the trigger event recognition unit 65 recognizes the motion of the user U in the complex space MS, and determines whether or not the motion corresponds to a predetermined motion defined in advance as a predetermined trigger event. In this embodiment, as shown in FIG. 19, an action in which one of the third user U3 and the fourth user U4 touches the other is set as a predetermined trigger event.

When a predetermined trigger event is recognized (YES in STEP600), the user environment determination unit 64 of the server 6 synchronizes the type of complex space MS that the user U experiences (FIG. 18/STEP601).

Specifically, the user environment determining unit 64 maintains the composite space MS experienced by the third user U3 as the first composite space MS1, and changes the composite space MS experienced by the fourth user U4 to the second composite space MS2. to the first composite space MS1.

More specifically, after this process, the user environment determining unit 64 selects the object to be recognized by the third user U3 as the object for generating the second composite space MS2 (for example, the book O40, etc.), but not the object for generating the second complex space MS2. 1 Change to an object (for example, a flower O30, etc.) for generating a composite space MS1.

Next, the user environment determination unit 64 determines the conditions of the third user U3 and the fourth user U4 based on the conditions of the third user U3 and the fourth user U4 and the conditions of the first composite space MS1 at that time. The environment in the first composite space MS1 to be recognized by the user U4 is determined (FIG. 18/STEP 602).

Specifically, the user environment determining unit 64, as the environment to be recognized by the third user U3, based on the coordinates and orientation of the physical space RS of the third user U3, creates an image of a region to be recognized in the physical space RS, The image of the third object O3 in that area and the accompanying sound are determined.

In addition, the user environment determination unit 64, as the environment to be recognized by the fourth user U4, based on the coordinates and posture of the physical space RS of the fourth user U4, the image of the region to be recognized in the physical space RS, The image of the third object O3 and the accompanying sound are determined.

This allows the fourth user U4 to experience the first complex space MS1 together with the third user U3. As a result, as shown in FIG. 20, the fourth user U4 can visually recognize the flower O30, which is a virtual object arranged on the shelf, which is a real object, arranged by the third user U3.

On the other hand, if the predetermined trigger event is not recognized (NO in STEP600), the user environment determining unit 64 determines the third The environment in the first composite space MS1 to be recognized by the user U3 is determined, and based on the state of the fourth user U4 and the situation of the second composite space MS2 at that time, the environment in the second composite space MS2 to be recognized by the fourth user U4 The environment is determined (FIG. 18/STEP 603).

After executing the process of determining the environment of the user U (the process of STEP 602 or the process of STEP 603), the HMD 7 executes the same process as STEP 503 and outputs the determined environment (FIG. 18/STEP 604 ).

Next, the user action recognition section 62 of the server 6 determines whether or not the third user U3 or the fourth user U4 has performed some action (FIG. 18/STEP 605).

If the user U does not act (NO in STEP605), the user action recognition unit 62 continues to perform a predetermined The determination in STEP 605 is repeated at the control cycle of .

On the other hand, if the user U has made a motion (YES in STEP605), the user motion recognition unit 62 determines whether or not the motion is a predetermined motion corresponding to a predetermined trigger event (FIG. 18/STEP606). .

If the motion of the user U is a predetermined motion (YES in STEP606), the trigger event recognizing unit 65 executes the process of synchronizing the first composite space MS1 and the second composite space MS2 (the process of STEP601). It is determined whether or not it is completed (FIG. 18/STEP 607).

Specifically, the trigger event recognition unit 65 determines whether or not a predetermined trigger event that is a key to executing the process of synchronizing the first composite space MS1 and the second composite space MS2 has been recognized. , to determine whether the process has been executed.

If the synchronization process has not been executed (NO in STEP607), the process returns to STEP600, and the processes after STEP600 are executed again.

On the other hand, if the synchronization process has already been executed (YES in STEP607), the process returns to STEP602, and the processes after STEP602 are executed again.

On the other hand, if the action of the user U is not the predetermined action (NO in STEP 606), the object control unit 63 of the server 6 determines that the action of the user U is an action to operate the object O, similar to the process in STEP 505. (FIG. 18/STEP 608).

If the action of the user U is to operate the object O (YES in STEP 608), the object control unit 63 controls the object O according to the type of action of the user U, similar to the process in STEP 506. (FIG. 18/STEP 609).

After executing the process of controlling object O (process of STEP 609), or if the action of user U is not the action of operating object O (NO in STEP 608), the same as the process in STEP 602 or the process in STEP 603 Furthermore, the user environment determination unit 64 determines the environment in the composite space MS that the user U is made to recognize based on the state of the user U at that time and the situation of the composite space MS experienced by the user U ( FIG. 18/STEP 610).

Next, the HMD 7 executes the same processing as STEP604 and outputs the determined environment (FIG. 18/STEP611).

Next, the server 6 determines whether or not it has recognized a signal instructing the end of processing (FIG. 18/STEP 612).

If the signal instructing the end cannot be recognized (NO in STEP612), the process returns to STEP605, and the processes after STEP605 are executed again.

On the other hand, if the MR system S3 recognizes the signal instructing the end (YES in STEP612), the MR system S3 ends the current process.

As described above, in the MR system S3, before the trigger event is recognized, the environment in the first composite space MS1 is determined as the environment to be recognized by the third user U3, and the environment in the second composite space MS2 is determined. is determined as the environment to be recognized by the fourth user U4.

On the other hand, after the trigger event is recognized, the environment in the first complex space MS is determined as the environment recognized by the third user U3 and the fourth user U4.

As a result, before the trigger event is recognized, the complex space experienced by the third user U3 (for example, the first complex space MS1 in this embodiment) and the complex space experienced by the fourth user U4 (for example, The second complex space M2) in the form is a complex space in which existing virtual objects are independent of each other.

On the other hand, after the trigger event is recognized, the composite space experienced by the third user U3 and the composite space experienced by the fourth user U4 are the same composite space in which existing virtual objects match (for example, a real object). It becomes the first virtual space MS1) in the embodiment.

Therefore, according to the MR system S3, after the trigger event is recognized, a process for synchronizing the objects existing in the composite space MS is executed to make the third user U3 and the fourth user U4 recognize the composite event. Since the environment of the space MS becomes similar, the third user U3 and the fourth user U4 who have experienced different environments can now experience similar environments.

[Modified example of the third embodiment]
Note that, in the present embodiment, an action in which one of the third user U3 and the fourth user U4 touches the other is set as the predetermined trigger event.

This is because when one of the third user U3 and the fourth user U4 sets a predetermined action to be performed on the other as a predetermined trigger event, one of the third user U3 and the fourth user U4 who performed the action This is because the user can intuitively understand that he/she can also experience the other environment as a result of executing the action.

However, in the third invention, the "predetermined action" that is a predetermined trigger event is not limited to the action of one of the third user U3 and the fourth user U4 touching the other. Any action performed by one of the user U3 and the fourth user U4 with reference to the other is sufficient.

For example, the predetermined action is an action of the third user contacting the fourth user, an action of the third user moving within a predetermined range with respect to the fourth user, and an action of the third user moving into the complex space. When operating a virtual or real object that exists in a virtual or real object, an action to select the fourth user as the target (for example, an action to take a picture of the fourth user with a camera of a virtual mobile terminal or a real mobile terminal etc.).

Also, for example, a ghost may be generated as a virtual object that moves in accordance with the user's motion, but whose coordinates in the complex space are different from those of the user. In such a case, the process of synchronizing the user characteristics may be executed even when a predetermined action is performed on the ghost.

Furthermore, the predetermined trigger event in the third invention does not necessarily have to be a predetermined action performed by one of the third user and the fourth user on the other of the first user and the fourth user. Therefore, for example, when the user gives an instruction to synchronize the complex space via an interface provided separately, the fact that the instruction is given may be set as a predetermined trigger event.

Reference Signs 2, 5 Cameras 3, 6 Servers 4, 7 HMD (environment output device) 30 Virtual environment generator 30a Virtual space generator 30b Avatar generator 31, 62 User motion recognition unit 31a, 62a User posture recognition unit 31b, 62b User coordinate recognition unit 32 Avatar motion control unit 33 Avatar environment determination unit 34, 64 User environment determination unit 35, 65 Trigger event recognition unit 40, 70 Monitor 41, 71 Speaker 60 Real space recognition unit 61 Complex environment generation unit 61a Object generation unit 61b Complex space generation unit 63 Object control unit , A1... First avatar, A2... Second avatar, G1... First ghost, G2... Second ghost, MS1... First complex space, MS2... Second complex space, O1... First object, O2... Second object , O3... third object, O30... flower, O31... first message board, O4... fourth object, O40... book, O41... second message board, RS... real space, S1, S2... VR system (virtual space experience system), S3... MR system (multiple spatial experience system), U1... first user, U2... second user, U3... third user, U4... fourth user, VS... virtual space, VS1... first virtual space, VS2 . . . second virtual space.

Claims (1)

A first virtual space corresponding to a physical space in which a plurality of users including a first user and a second user exist, and a plurality of second virtual spaces independent of the first virtual space and corresponding to the physical space. a virtual space generation unit that generates a virtual space of
A first avatar corresponding to the first user,PreviousA second avatar corresponding to the second user, and a ghost that corresponds to the first user or the second user and is an avatar independent of the first avatar and the second avataran avatar generation unit that generates a plurality of avatars including in the first virtual space or the second virtual space;
a user action recognition unit that recognizes actions of the first user and the second user in the physical space;
Avatar action for determining control of the first avatar in the virtual space based on the action of the first user, and controlling action of the second avatar in the virtual space based on the action of the second user. a control unit;
an avatar environment determination unit that determines environments of the first avatar and the second avatar in the virtual space;
Make the one user recognize based on the environment of the one avatarof the virtual spaceDetermining an environment and making the second user aware based on the environment of the second avatarof the virtual spacea user environment determination unit that determines the environment;
a trigger event recognition unit that recognizes a predetermined trigger event;
A virtual space experience system comprising an environment output device that allows the first user and the second user to recognize the environment of the virtual space,
The predetermined trigger event corresponds to the ghost corresponding to one of the first user and the second user, and the other of the first user and the second user among the first avatar and the second avatar. is a predetermined action performed by an avatar that performs
Before the predetermined trigger event is recognized,
The avatar generation unit generates the first avatar,generated in one of the first virtual space and the second virtual spacedeath, the second avatar,generated in the other virtual space of the first virtual space and the second virtual space;The ghost corresponding to one of the first user and the second user is placed in the virtual space in which the other of the first avatar and the second avatar exists, out of the first virtual space or the second virtual space. generate and
The avatar environment determination unit determines the environment of the first avatar based on the one virtual space and determines the environment of the second avatar based on the other virtual space;
After the predetermined trigger event is recognized,
The avatar generation unit generates the first avatar and the second avatar,generated in one of the first virtual space and the second virtual space;
The avatar environment determination unit determines the environment of the first avatar based on the one virtual space, and determines the environment of the second avatar based on the one virtual space. Virtual space experience system.

Images