JP6933850B1 - Virtual space experience system - Google Patents

Abstract

The object control unit 34 of the VR system S determines the coordinates of the second object observed by the second avatar corresponding to the second user when the state of the first object is changed by the first avatar corresponding to the first user. While fixing at least one of the posture and the direction, the state other than at least one of the coordinates, the posture and the direction of the second object is changed according to the change of the state of the first object.

Description

The present invention is a virtual that can generate an avatar in a virtual space corresponding to a user existing in the real space and change the state of a predetermined object generated in the virtual space through the operation of the avatar. Regarding the spatial experience system.

Conventionally, a virtual space is generated by a server or the like, and a user is made to recognize an image of the virtual space and an image of an avatar corresponding to the user via a head-mounted display (hereinafter, may be referred to as “HMD”). , There is a virtual space experience system that makes the user feel as if he / she exists in the virtual space.

In this kind of virtual space experience system, a motion capture device or the like recognizes a change in the user's state in the real space (for example, body movement, coordinate movement, posture change, etc.), and the recognized state is recognized. Some change the state of the avatar in the virtual space according to the change (see, for example, Patent Document 1).

In the virtual space experience system of Patent Document 1, a hand-shaped avatar corresponding to a user's hand existing in the real space and an object to be operated are generated in the virtual space. Then, when the hand-shaped avatar performs an action of grabbing and moving the object, the coordinates of the object in the virtual space are changed according to the action.

Japanese Unexamined Patent Publication No. 2018-49629

By the way, when a virtual space is generated by using a system as described in Patent Document 1, a plurality of objects can be generated at different positions in the virtual space by using one object actually existing in the real space as a model.

When a plurality of objects are generated in this way, each of the plurality of users participating in one virtual space has an object corresponding to the same object in their own hands (strictly speaking, in the hands of the corresponding avatars). It is possible to observe in a format that cannot be performed in the real space, such as holding one by one and observing independently at the same time. At this time, if the states of the plurality of objects are configured to change synchronously, each of the plurality of users can independently observe the changes in the states of the objects at the same time.

However, the states of the plurality of users (and thus the corresponding avatars) are usually different for each. Therefore, when the state of a plurality of objects is configured to change synchronously, when the state of the object corresponding to the specific user changes due to the action of a specific user, the object corresponding to the other user. There is a risk that the state of the object will change unintentionally for other users, making it difficult for other users to observe.

The present invention has been made in view of the above points, and provides a virtual space experience system in which a plurality of users can easily observe each of a plurality of objects modeled on one object in the virtual space. The purpose is to do.

In the following, the "state" refers to a state that can be changed according to the intention or operation of the user. For example, it refers to the shape, size, color, posture, direction, coordinate position, operating state of the body or mechanism, and the like. Therefore, the "change" and "change" of the state refer to a change or change of shape, size, color, posture, direction, movement of coordinates, start, progress and stop of operation, and the like. Further, the "corresponding shape" includes, in addition to the same shape, a similar shape (that is, a shape having the same shape but a different size), a deformed shape, the same shape as a partially cut out shape, and the like.

The virtual space experience system of the present invention
A predetermined value generated in the virtual space through the operation of either the first avatar or the second avatar generated in the virtual space corresponding to the first user and the second user existing in the real space. A virtual space experience system that can change at least one of the coordinates, orientation, and direction of an object, as well as a state that includes at least a shape.
The virtual space, and a virtual space generation unit that generates the first avatar, the second avatar, and the predetermined object existing in the virtual space.
A user state recognition unit that recognizes the state of the first user and the state of the second user,
An avatar control unit that controls the state of the first avatar according to the state of the first user and controls the state of the second avatar according to the state of the second user.
An object control unit that controls the state of the predetermined object, and
An image determination unit that determines an image of the virtual space to be recognized by the first user and the second user based on the state of the first avatar, the state of the second avatar, and the state of the predetermined object.
The first user and the second user are provided with an image display for recognizing an image of the virtual space.
The predetermined object includes a first object observed by the first user and a second object observed by the second user.
The second object is located in the virtual space at coordinates different from the coordinates of the first object, and has a shape corresponding to the first object.
The state of the first object can be changed by the first user through the operation of the first avatar.
The object control unit
When the shape of the first object is changed, the shape of the second object is changed according to the change in the shape of the first object.
When a state other than the shape of the first object is changed, the state of the second object other than the fixed state is changed while at least one of the coordinates, the posture, and the direction of the second object is fixed. It is characterized in that it changes according to the change of the state of the first object.

As described above, the virtual sky experience system of the present invention has a shape corresponding to the second object. That is, those objects are generated by using one object (for example, an article existing in the real space) as a model. Further, in this system, when the shape of the first object is changed, the shape of the second object is changed (synchronically) according to the change in the shape of the first object.

Therefore, in this system, for example, the first user who plays the role of a teacher operates the first object in his / her own hands, and the second user who plays the role of a student observes the second object in the hands of the second user. This makes it possible for the first user who plays the role of a teacher to teach the second user who plays the role of a student how to operate the first object and the object that is the model of the second object through the second object. ing.

In addition, the system retains at least one of the coordinates, orientation, and orientation of the second object when a state other than the shape of the first object is changed, while the second object is out of that fixed state. Is configured to change in response to changes in the state of the first object.

As a result, the state of the second object observed by the second user changes according to the result of the operation of the first user on the first object, but is not completely synchronized, and at least one of the coordinates, the posture, and the direction. One does not change. That is, the state related to observation remains fixed to some extent.

Therefore, the first user can observe the first object that he / she observes while freely moving it, while the second user also observes the second object that he / she observes in a somewhat stable state. Can be done.

Therefore, according to the virtual space experience system of the present invention, each of a plurality of objects modeled on one object in the virtual space is stable as freely intended by each corresponding user or to some extent. It can be observed in the state. As a result, it is possible to realize a state in which each of the plurality of users can easily observe a plurality of objects modeled on one object and objects corresponding to each user.

Further, in the virtual space experience system of the present invention,
The state of the second object can be changed by the second user through the action of the second avatar.
It is preferable that the object control unit changes at least one of the corresponding coordinates, posture and direction of the second object independently of at least one of the coordinates, posture and direction of the first object.

With this configuration, when it is difficult for the second user to observe the second object corresponding to himself / herself, at least one of the coordinates, the posture, and the direction of the second object is set to the first object corresponding to the first user. It can be changed to make it easier to see regardless of the state of. This makes it easier for the second user to observe the second object.

Further, in the virtual space experience system of the present invention,
The predetermined object includes a third object operated by the second user.
The color of the second object is translucent or a color different from that of the third object.
The third object has a shape corresponding to the second object and has a shape corresponding to the second object.
The shape of the third object can be changed by the second user through the operation of the second avatar.
It is preferable that the coordinates, posture and direction of the third object match the coordinates, posture and direction of the second object.

With this configuration, the second object and the third object are superimposed in the virtual space. Here, as described above, the shape of the second object changes according to the change in the shape of the first object corresponding to the first user (that is, the operation of the first user). On the other hand, the shape of the third object changes according to the operation of the second user. Therefore, when the shape of the first object changes, only the shape of the second object out of the superimposed second object and the third object changes in response to the change.

As a result, when the shape of the first object changes, only the changed part (that is, only the difference between the shape of the first object and the shape of the third object) is superposed and generated. The 2 object 7 and the 3rd object 8 will be shown to the 2nd user U2 as a translucent part or a part having a color different from that of the 3rd object.

As a result, the second user can intuitively grasp the difference between the shape of the first object and the shape of the third object (and by extension, the operation of the first user on the first object).

Specifically, for example, when the object is like a prefabricated model, the second user can easily grasp the assembling procedure of the first user by observing the difference. ..

Further, in the virtual space experience system of the present invention,
It is provided with an item state recognition unit that recognizes the state of an item existing in the real space.
The predetermined object includes a third object operated by at least one of the first user and the second user.
It is preferable that the virtual space generation unit generates the third object in correspondence with the item.

In this way, when the third object to be operated is generated corresponding to the item that actually exists in the real space, the operation performed by the user on the third object via the avatar in the virtual space is performed. The result will be reflected in the item corresponding to the real space. As a result, the user can feel as if the result of the avatar's operation in the virtual space is reflected in the item in the real space.

Further, in this configuration, when the user touches the third object, the user touches the item in the real space. This gives the user the feeling of touching the third object. As a result, the user's immersive feeling in the virtual space can be enhanced.

The schematic diagram which shows the schematic structure of the VR system which concerns on embodiment. The block diagram which shows the structure of the processing part of the VR system of FIG. The schematic diagram of the virtual space that the VR system of FIG. 1 makes a user recognize. FIG. 5 is a flowchart showing a process to be executed when the VR system of FIG. 1 is started to be used. The flowchart which shows the process which the VR system of FIG. 1 executes when the 1st object is operated. The schematic diagram of the virtual space which the VR system of FIG. 1 makes a user recognize when the shape of the 1st object is changed. FIG. 5 is a schematic diagram of a virtual space in which the VR system of FIG. 1 makes a user recognize when the posture and direction of the first object are changed. The flowchart which shows the process which the VR system of FIG. 1 executes when the 2nd object is operated. FIG. 5 is a schematic diagram of a virtual space in which the VR system of FIG. 1 makes the user recognize when the posture and direction of the second object are changed. The flowchart which shows the process which the VR system of FIG. 1 executes when the 3rd object is operated. The schematic diagram of the virtual space which the VR system of FIG. 1 makes a user recognize when the state of a 3rd object is changed. The schematic diagram of the virtual space which a VR system which concerns on a modification makes a user recognize when an object is operated.

Hereinafter, the VR system S, which is a virtual space experience system according to the embodiment, will be described with reference to the drawings. The VR system S is a system that allows the user to experience virtual reality (so-called VR (virtual reality)) by recognizing that the user himself / herself exists in the virtual space.

As shown in FIG. 1, the VR system S exists in the first user U1 existing in the first real space RS1 and in the second real space RS2 existing at a position different from the first real space RS1. The second user U2 is configured to be able to communicate with each other via the same virtual space VS (see FIG. 3).

In the following description, the first real space RS1 and the second real space RS2 may be collectively referred to as a real space RS. Further, the first user U1 and the second user U2 may be collectively referred to as a user U. Further, the first camera 2a is installed in the first real space RS1, and the second camera 2b is installed in the second real space RS2. The first camera 2a and the second camera 2b are collectively referred to as the camera 2.

Further, in the following description, the "state" refers to a state that can be changed according to the intention or operation of the user U. For example, it refers to the shape, size, color, posture, direction, coordinate position, operating state of the body or mechanism, and the like. Therefore, the "change" and "change" of the state refer to a change or change of shape, size, color, posture, direction, movement of coordinates, start, progress and stop of operation, and the like.

Further, the "corresponding shape" includes, in addition to the same shape, a similar shape (that is, a shape having the same shape but a different size), a deformed shape, the same shape as a partially cut out shape, and the like.

In the present embodiment, the first user U1 who plays the role of a teacher is sent to the second user U2 who is a student who is located away from the first user U1 via the virtual space VS, together with the second user U2. In order to teach how to make a prefabricated model which is an item I existing in the real space RS2, a case where the VR system S is used will be described as an example.

The virtual space experience system of the present invention is not limited to such a configuration, and a user may exist in each of three or more real spaces, or a plurality of users may be in the same real space. May be present in.

[Outline configuration of system]
First, a schematic configuration of the VR system S will be described with reference to FIGS. 1 and 2.

A plurality of signs 1 attached to the user U, the item I, and the table T existing in the real space RS, and a camera 2 for photographing the user U, the item I, and the table T (strictly speaking, the signs 1 attached to them). , A server 3 that determines the image and sound of the virtual space VS (see FIG. 3 and the like) to be recognized by the user U, and a head-mounted display (hereinafter referred to as "HMD4") that causes the player to recognize the determined image and sound. It has.

In the VR system S, the first camera 2a and the HMD4 worn by the first user U1 can wirelessly transmit and receive information to and from the server 3. Similarly, the second camera 2b and the HMD4 worn by the second user U2 can wirelessly transmit and receive information to and from the server 3. However, any one of them may be configured to be able to send and receive information to and from each other by wire instead of wirelessly.

Of the plurality of signs 1, those attached to the user U are attached to the head, both hands, and both feet of the user U via the HMD4, gloves, and shoes worn by the user U.

Further, among the plurality of signs 1, those attached to the item I and the table T are attached to the positions that are feature points in the images of the item I and the table T taken by the second camera 2b.

In the present embodiment, the item I is a model of an assembled car placed on the table T of the second real space RS2, and the sign 1 is a central portion, an edge, a corner vicinity, etc. of each part of the model. It is attached to. On the table T, the sign 1 is composed of a circular top plate and four legs, and is attached to the edge of the top plate, the ends of the legs, and the like.

The sign 1 is used to recognize the shape, posture, coordinates, and direction of the user U, the item I, or the table T in the real space RS as described later. Therefore, the mounting position of the sign 1 may be appropriately changed according to other devices constituting the VR system S.

Of the cameras 2, the first camera 2a installed in the first real space RS1 executes the movable range of the first user U1 (that is, the change in posture, the movement of coordinates, the change in posture and direction, and the like. Multiple possible ranges) are installed so that images can be taken from multiple directions.

Further, among the cameras 2, the second camera 2b installed in the second real space RS2 has a movable range of the second user U2, the item I and the table T (that is, a change in posture, movement of coordinates, and movement of coordinates, and A plurality of areas (ranges in which changes in shape and direction can be performed) are installed so that images can be taken from multiple directions.

The server 3 recognizes the sign 1 from the image taken by the camera 2, and based on the coordinates of the recognized sign 1 in the real space RS, determines the shape, orientation, coordinates, and direction of the user U, the item I, or the table T. recognize. Further, the server 3 determines the image and the sound to be recognized by the user U based on the shape, the posture, the coordinates and the direction.

The HMD4 is attached to the head of the user U. As shown in FIG. 2, the HMD 4 has a monitor 40 (image display) for causing the user U to recognize the image of the virtual space VS determined by the server 3, and a virtual space determined by the server 3. It has a speaker 41 (sound generator) for causing the user U to recognize the VS sound.

When the user U is made to experience the virtual reality by using this VR system S, the user U is made to recognize only the image and the sound of the virtual space VS, and the user U himself / herself is made to recognize the first avatar 5a or the second avatar 5b described later. Is recognized as existing in the virtual space. That is, the VR system S is configured as a so-called immersive system.

The VR system S includes a so-called motion capture device composed of a sign 1, a camera 2, and a server 3 as a system for recognizing the states of the user U, the item I, and the table T in the real space RS.

However, the virtual space experience system of the present invention is not limited to such a configuration. For example, when using a motion capture device, in addition to the above configuration, the number of signs and cameras is different from the above configuration (for example, one is provided in each virtual space). May be good.

In addition, a device other than the motion capture device may be used to recognize the state of the user, item, or table in the real space. Specifically, for example, a sensor such as GPS may be mounted on the HMD, and the state of the user, item, or table may be recognized based on the output from the sensor. Further, such a sensor may be used in combination with the motion capture device as described above.

[Configuration of processing unit]
Next, the configuration of the server 3 will be described in detail with reference to FIGS. 2 and 3.

The server 3 is composed of one or a plurality of electronic circuit units including a CPU, RAM, ROM, an interface circuit, and the like. As shown in FIG. 2, the server 3 has a virtual space generation unit 30, a user state recognition unit 31, an avatar control unit 32, and an item state recognition unit as functions realized by the implemented hardware configuration or program. It includes 33, an object control unit 34, and an output information determination unit 35.

As shown in FIG. 3, the virtual space generation unit 30 is a virtual space VS corresponding to the first real space RS1 in which the first user U1 exists and the second real space RS2 in which the second user U2 exists. Generates an image (strictly speaking, the background image) and related audio.

Further, the virtual space generation unit 30 generates images of the first avatar 5a and the second avatar 5b corresponding to the first user U1 and sounds related thereto as avatars existing in the virtual space VS. In the following, the first avatar 5a and the second avatar 5b may be collectively referred to as the avatar 5. In the present embodiment, each of the avatars 5 is anthropomorphic to an animal, and operates in response to the corresponding operation of the user U.

Further, the virtual space generation unit 30 generates a predetermined object existing in the virtual space VS. In addition to the object for representing the environment of the virtual space VS (for example, a plant existing only in the virtual space VS), the predetermined object includes the first object 6 and the first object U1 observed and operated by the first user U1. It includes a second object 7 observed by the second user U2, a third object 8 operated by the second user U2, and a fourth object 9 on which the third object 8 is placed.

The first object 6 is generated in the virtual space VS at a position near the first avatar 5a corresponding to the first user U1. The shape, coordinates, posture, and direction of the first object 6 can be changed according to the movement of the first avatar 5a (that is, the first user U1).

The second object 7 is generated in the virtual space VS at a position near the second avatar 5b corresponding to the second user U2. That is, the second object 7 is located at a coordinate different from the coordinates of the first object 6. The coordinates, posture, and direction of the second object 7 can be changed according to the movement of the second avatar 5b (that is, the second user U2).

The third object 8 is generated in the virtual space VS at a position corresponding to the item I existing in the second real space. That is, the third object 8 is located at a coordinate different from the coordinates of the first object 6 and the coordinates of the second object 7. Specifically, the third object 8 is generated at the coordinates in which it is placed on the fourth object 9 corresponding to the table T.

The shape, coordinates, posture, and direction of the third object 8 can be changed according to the operation of the second avatar 5b with respect to the third object 8 (that is, the operation of the second user U2 with respect to the item I).

The shape of the third object 8 is a shape corresponding to the item I. When the use of the VR system S is started, the shapes of the first object 6 and the second object 7 are generated so as to correspond to the shapes of the third object 8. After that, the shapes of the first object 6 and the second object 7 change independently of the shapes of the third object 8.

In the present embodiment, the item I existing in the second real space RS2 is a model of an assembled car, and is composed of four parts: a main body, a roof, a windshield, and a tire. Then, the third object 8 is generated as corresponding to the item I.

Therefore, the third object 8 has a first part 80 which is a main body part, a second part 81 which is a roof part, a third part 82 which is a windshield, and a tire so as to correspond to each part of the item I. It is composed of the fourth part 83 (see FIG. 12).

Further, the first object 6 corresponding to the third object 8 is a first part 60 which is a main body part, a second part 61 which is a roof part, a third part 62 which is a windshield, and a fourth part which is a tire. It is composed of 63 (see FIG. 6 and the like). Similarly, the second object 7 is also composed of a first part 70 which is a main body part, a second part 71 which is a roof part, a third part 72 which is a windshield, and a fourth part 73 which is a tire ( See FIG. 6 etc.).

The fourth object 9 is generated in the virtual space VS at a position corresponding to the table T existing in the second real space. The coordinates, posture, and direction of the fourth object can be changed according to the operation of the second avatar 5b with respect to the fourth object 9 (that is, the operation of the second user U2 with respect to the table T). The shape of the fourth object 9 is a shape corresponding to the table T.

As shown in FIG. 2, the user state recognition unit 31 recognizes the state of the first user U1 based on the image data of the first user U1 including the sign 1 taken by the first camera 2a, and the second camera 2b Recognizes the state of the second user U2 based on the image data of the second user U2 including the sign 1 taken by the user.

Further, the user state recognition unit 31 has a user coordinate recognition unit 31a, a user posture recognition unit 31b, and a user direction recognition unit 31c. The user coordinate recognition unit 31a, the user attitude recognition unit 31b, and the user direction recognition unit 31c extract the marker 1 attached to the user U from the image data of the user U, and based on the extraction result, the coordinates of the user U, Recognize posture and direction.

The avatar control unit 32 controls the state (specifically, coordinates, posture and direction) of the first avatar 5a according to the change in the state of the first user U1 recognized by the user state recognition unit 31, and the user. The state (specifically, coordinates, posture, and direction) of the second avatar 5b is controlled according to the change in the state of the second user U2 recognized by the state recognition unit 31.

The item state recognition unit 33 recognizes the state of the item I and the table T based on the image data of the item I and the table T including the sign 1 taken by the second camera 2b. The item state recognition unit 33 includes an item shape recognition unit 33a, an item coordinate recognition unit 33b, an item posture recognition unit 33c, and an item direction recognition unit 33d.

The item shape recognition unit 33a, the item coordinate recognition unit 33b, the item attitude recognition unit 33c, and the item direction recognition unit 33d are attached to the item I and the table T from the image data of the item I and the image data of the table T. Is extracted, and the shape, coordinates, orientation, and direction of the item I and the table T are recognized based on the extraction result.

The object control unit 34 controls the states of the third object 8 and the fourth object 9 according to the operation of the avatar 5 recognized by the avatar control unit 32.

Further, the object control unit 34 has a first object control unit 34a, a second object control unit 34b, a third object control unit 34c, and a fourth object control unit 34d.

The first object control unit 34a responds to the movement of the first avatar 5a (that is, the first user U1) by the shape (assembled state), coordinates, posture (tilt with respect to the yaw axis), and direction (inclination with respect to the yaw axis) of the first object 6. The phase around the yaw axis) is controlled.

The second object control unit 34b controls the coordinates, posture, and direction of the second object 7 according to the operation of the second avatar 5b (that is, the second user U2). Further, the second object control unit 34b controls the shape of the second object 7 according to the change in the shape of the first object 6.

The third object control unit 34c controls the shape, coordinates, posture, and direction of the third object 8 according to the movement of the second avatar 5b (that is, the second user U2) and the state of the item I.

The fourth object control unit 34d controls the coordinates, posture, and direction of the fourth object 9 according to the operation of the second avatar 5b (that is, the second user U2) and the state of the table T. However, in the present embodiment, the state of the fourth object 9 is not changed.

The output information determination unit 35 determines the information regarding the virtual space VS to be recognized by the user U via the HMD4. The output information determination unit 35 has an image determination unit 35a and a voice determination unit 35b.

The image determination unit 35a corresponds to the avatar 5 via the monitor 40 of the HMD4 based on the state of the avatar 5 and the states of the first object 6, the second object 7, the third object 8 and the fourth object 9. The image of the virtual space VS to be recognized by the user U is determined.

The voice determination unit 35b corresponds to the avatar 5 via the speaker 41 of the HMD4 based on the state of the avatar 5 and the states of the first object 6, the second object 7, the third object 8 and the fourth object 9. The sound related to the image of the virtual space VS to be recognized by the user U is determined.

It should be noted that each processing unit constituting the virtual space experience system of the present invention is not limited to the above configuration. For example, a part of the processing unit provided in the server 3 in the present embodiment may be provided in the HMD 4. Further, a plurality of servers may be used, or the server 3 may be omitted and the CPUs mounted on the HMD 4 may be cooperated with each other.

[Process to be executed]
Next, with reference to FIGS. 2 to 12, a process executed by the VR system S when the user U is made to experience the virtual space VS by using the VR system S will be described.

[Processing when use starts]
First, with reference to FIGS. 2 to 4, when the use of the VR system S is started, the processing executed by each processing unit of the VR system S will be described.

In this process, first, the virtual space generation unit 30 of the server 3 generates a virtual space VS, an avatar 5 to exist in the virtual space VS, and various objects based on the image taken by the camera 2 ((). FIG. 4 / STEP100).

Specifically, as shown in FIG. 3, the virtual space generation unit 30 generates an image as a background of the virtual space VS. Further, the virtual space generation unit 30 generates an image of the avatar 5 existing in the virtual space VS based on the image of the user U among the images taken by the camera 2, and the first image is based on the image of the item I. The images of the object 6, the second object 7, and the third object 8 are generated, and the image of the fourth object 9 is generated based on the image of the table T.

Next, the object control unit 34 of the server 3 determines the states of the first object 6, the second object 7, and the third object 8 in the virtual space VS based on the state of the item I in the second real space RS2 ( FIG. 4 / STEP101).

Specifically, first, the item state recognition unit 33 recognizes the state of the item I in the second real space RS2 based on the state of the item I photographed by the camera 2. After that, the third object control unit 34c of the object control unit 34 determines the state of the third object 8 based on the recognized state of the item I.

After that, the first object control unit 34a and the second object control unit 34b of the object control unit 34 determine the states of the first object 6 and the second object 7 so as to correspond to the states of the third object 8. ..

At this time, the item state recognition unit 33 recognizes the state of the table T in the second real space RS2 based on the state of the table T photographed by the camera 2. After that, the fourth object control unit 34d of the object control unit 34 determines the state of the fourth object 9 based on the recognized state of the table T.

Next, the avatar control unit 32 of the server 3 determines the state of the avatar 5 corresponding to each user U in the virtual space VS based on each state of the user U in the real space RS (FIG. 4 / STEP102). ..

Specifically, first, the user state recognition unit 31 recognizes the state of the user U in the real space RS based on the state of the user U photographed by the camera 2. After that, the avatar control unit 32 determines the state of the avatar 5 based on the recognized state of the user U.

Next, the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 determine the states of the first object 6, the second object 7, the third object 8 and the fourth object 9, and the avatar 5. Based on the state, the image and sound to be recognized by the user U are determined (FIG. 4 / STEP103).

Next, the HMD4 worn by the user U displays the determined image on the monitor 40, generates the determined sound on the speaker 41 (FIG. 4 / STEP104), and ends the current process. do.

In the present embodiment, as shown in FIG. 3, in the virtual space VS, the first avatar 5a corresponding to the first user U1 and the second avatar 5b corresponding to the second user U2 are caused by these processes. Arranged so that they exist facing each other.

Further, the first object 6 is arranged in the vicinity of the first avatar 5a corresponding to the first user U1. Further, the second object 7 is arranged in the vicinity of the second user U2. Further, the third object 8 and the fourth object 9 are arranged in the vicinity of the second user U2 and at positions corresponding to the item I and the table T in the second real space RS2.

As a result, the user U who is made to recognize this virtual space VS, as the avatar 5 corresponding to himself / herself, presents the object that exists in the virtual space VS and exists in the vicinity of himself / herself with his / her own hand. It is recognized so that it can be operated (strictly via avatar 5).

[Processing when the first object is operated]
Next, referring to FIGS. 2, 5 to 7, when the first user U1 operates the first object 6 via the first avatar 5a after the use of the VR system S is started, the VR The processing executed by each processing unit of the system S will be described.

In this process, first, the user state recognition unit 31 determines whether or not the state of the first user U1 has changed (FIG. 5 / STEP200).

If the state of the first user U1 has not changed (NO in STEP200), the process returns to STEP200, and the determination of STEP200 is executed again in a predetermined control cycle.

On the other hand, when the state of the first user U1 changes (YES in STEP 200), the avatar control unit 32 changes the state of the first avatar 5a based on the change in the state of the first user U1 (FIG. 5). / STEP201).

Next, the object control unit 34 determines whether or not the change in the state (that is, the operation) of the first avatar 5a executed by the avatar control unit 32 is an operation such as operating the first object 6. (Fig. 5 / STEP202).

Specifically, the object control unit 34 is based on whether or not the change in the coordinates, posture, and direction of the first avatar 5a with respect to the first object 6 in the virtual space VS corresponds to the change in the predetermined coordinates, posture, and direction. Therefore, it is determined whether or not the operation of the first avatar 5a is an operation such as operating the first object 6.

In the present embodiment, for example, as shown in FIG. 6, the object control unit 34 is an operation in which the hand of the first avatar 5a grabs or moves any of the parts included in the first object 6. Whether or not it is determined.

When the operation of the first avatar 5a is an operation such as operating the first object 6 (YES in STEP202), the first object control unit 34a of the object control unit 34 is based on the operation by the first avatar 5a. , The shape, coordinates, posture and direction of the first object 6 are changed (FIG. 5 / STEP203).

Next, the first object control unit 34a of the object control unit 34 changes the shape of the second object 7 located in the vicinity of the second avatar 5b based on the change in the shape of the first object 6 (FIG. 5 / STEP204).

As shown in FIG. 6, when the shape of the first object 6 is changed as a result of the operation on the first object 6 by the processing of STEP 202 to 204, the shape of the second object 7 is also synchronized with it. Change.

Specifically, for example, the first avatar 5a performs an operation of attaching the second part 61, which is the roof portion of the first object 6, which is a model of a car, to the first part 60, which is the main body portion. It is assumed that the shape of 1 object 6 is changed.

At this time, the shape of the second object 7 having the shape corresponding to the first object 6 also changes in synchronization with the change in the shape of the first object 6. Specifically, the second part 71, which is the roof portion of the second object 7, is automatically attached to the second part 71, which is the main body portion.

However, as shown in FIG. 7, when any of the coordinates, the posture, and the direction is changed instead of the shape of the first object 6 as a result of the operation on the first object 6, the second object 7 corresponds to the change. Coordinates, attitudes and directions are out of sync with it and do not change.

Specifically, for example, the first avatar 5a shows the posture (tilt with respect to the yaw axis) and the direction (phase about the yaw axis) of the first object 6 in the direction of the arrow shown in FIG. It is assumed that the posture and direction shown in are changed.

At this time, the posture and direction of the second object 7 do not change in synchronization with the change of the posture and direction of the first object 6. That is, the posture and direction of the second object 7 do not change before and after the state of the first object 6.

Next, the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 are based on the states of the first object 6 and the second object 7 in the virtual space VS and the state of the avatar 5. The image and sound to be recognized by U are determined (Fig. 5 / STEP205).

Similarly, when the operation of the first avatar 5a is not an operation for operating the first object 6 (NO in STEP202), the states of the first object 6 and the second object 7 in the virtual space VS, and the avatar. Based on the state of 5, the image determination unit 35a and the sound determination unit 35b of the output information determination unit 35 of the server 3 determine the image and sound to be recognized by the user U (FIG. 5 / STEP205).

Next, the HMD4 worn by the user U displays the determined image on the monitor 40, generates the determined sound on the speaker 41 (FIG. 5 / STEP206), and ends the current process. do.

The VR system S repeatedly executes the above processing in a predetermined control cycle until the end instruction by the user U is recognized.

As described above, in the VR system S, the first object 6 has a shape corresponding to the second object 7. That is, those objects are generated by using one object (in this embodiment, item I, which is a model of an assembled car existing in the second real space RS2) as a model. Further, in the VR system S, when the shape of the first object 6 is changed, the shape of the second object 7 is changed (synchronically) according to the change in the shape of the first object 6.

Therefore, in the VR system S, for example, the first user U1 who plays the role of a teacher operates the first object 6 which is in his / her own hand, and the second user U2 who plays the role of a student operates the second object U2 which is in the hand of the second user U2. By observing the object 7, the first user U1 acting as a teacher causes the second user U2 acting as a student to act as a model for the first object 6 and the second object 7 via the second object 7. It is possible to use it by teaching how to assemble (a model of a car in this embodiment).

Further, this VR system S keeps the coordinates, posture and direction of the second object 7 fixed when any one of the coordinates, the posture and the direction is changed as well as the shape of the first object 6. 2 The shape of the object 7 is configured to be changed according to the change in the shape of the first object.

As a result, the shape of the second object 7 observed by the second user U2 changes according to the result of the operation of the first user U1 on the first object 6, but the coordinates, posture, and direction do not change. That is, the state related to observation remains fixed to some extent.

Therefore, the first user U1 can observe the first object 6 that he / she observes while freely moving, while the second user U2 also observes the second object 7 that he / she observes in a stable state to some extent. Can be observed at.

Therefore, according to the VR system S, in the virtual space VS, each of the first object 6 and the second object 7 modeled on the item I of the second real space RS2 is the corresponding first user U1. Alternatively, it can be observed in a state that the second user U2 freely intended, or in a state that is stable to some extent. As a result, each of the first user U1 and the second user U2 can realize a state in which the first object 6 or the second object 7 corresponding to the first user U1 and the second user U2 can be easily observed.

The virtual space experience system of the present invention is not limited to such a configuration, and at least the coordinates, posture, and direction of the second object are changed when the state including the shape other than the shape of the first object is changed. It suffices that the state other than the fixed state of the second object may be changed according to the change of the state of the first object while keeping one fixed.

Therefore, as in the present embodiment, it is not always necessary to keep all the coordinates, posture, and direction of the second object in a fixed state regardless of the change in the state of the first object, and any of the coordinates, posture, and direction. One or two may be configured to be changed according to the corresponding state of the first object. At this time, the correspondingly changed state may be appropriately set according to the shape and type of the object to be observed, the purpose of observing the object, and the like.

[Processing when the second object is operated]
Next, referring to FIGS. 2, 8 and 9, when the second user U2 operates the second object 7 via the second avatar 5b after the use of the VR system S is started, the VR The processing executed by each processing unit of the system S will be described.

In this process, first, the user state recognition unit 31 determines whether or not the state of the second user U2 has changed (FIG. 8 / STEP300).

If the state of the second user U2 has not changed (NO in STEP300), the process returns to STEP300, and the determination of STEP300 is executed again in a predetermined control cycle.

On the other hand, when the state of the second user U2 changes (YES in STEP 300), the avatar control unit 32 changes the state of the second avatar 5b based on the change in the state of the second user U2 (FIG. 8). / STEP301).

Next, the object control unit 34 determines whether or not the change in the state (that is, the operation) of the second avatar 5b executed by the avatar control unit 32 is an operation such as operating the second object 7. (Fig. 8 / STEP302).

Specifically, the object control unit 34 is based on whether or not the change in the coordinates, posture, and direction of the second avatar 5b with respect to the second object 7 in the virtual space VS corresponds to the change in the predetermined coordinates, posture, and direction. Therefore, it is determined whether or not the operation of the second avatar 5b is an operation such as operating the second object 7.

In the present embodiment, for example, as shown in FIG. 9, the object control unit 34 is an operation in which the hand of the second avatar 5b grabs or moves any of the parts included in the second object 7. Whether or not it is determined.

When the operation of the second avatar 5b is an operation such as operating the second object 7 (YES in STEP 302), the second object control unit 34b of the object control unit 34 is based on the operation by the second avatar 5b. , The coordinates, posture and direction of the second object 7 are changed (FIG. 8 / STEP303).

Here, the second object 7 is an object for the second user U2 to observe. Therefore, in the VR system S, the second user U2 cannot change the shape of the second object 7 via the second avatar 5b corresponding to the second user U2.

Further, unlike the case where the first user U1 operates the first object 6, the state of the first object 6 does not change in synchronization with the change of the state of the second object 7.

Specifically, for example, the second avatar 5b changes the posture (tilt with respect to the yaw axis) and the direction (phase around the yaw axis) of the second object 7 in the direction of the arrow shown in FIG. do.

At this time, the posture and direction of the first object 6 do not change in synchronization with the change of the posture and direction of the second object 7. That is, the posture and direction of the first object 6 do not change before and after the state of the second object 7.

Next, the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 are based on the states of the first object 6 and the second object 7 in the virtual space VS and the state of the avatar 5. The image and sound to be recognized by U are determined (FIG. 8 / STEP304).

Similarly, when the operation of the second avatar 5b is not the operation of operating the second object 7 (NO in STEP 302), the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 , The image and sound to be recognized by the user U are determined based on the states of the first object 6 and the second object 7 in the virtual space VS and the states of the avatar 5 (FIG. 8 / STEP304).

Next, the HMD4 worn by the user U displays the determined image on the monitor 40, generates the determined sound on the speaker 41 (FIG. 8 / STEP305), and ends the current process. do.

The VR system S repeatedly executes the above processing in a predetermined control cycle until the end instruction by the user U is recognized.

As described above, in the VR system S, the coordinates, posture and direction of the second object 7 can be changed independently of the coordinates, posture and direction of the first object 6. As a result, when it is difficult for the second user U2 to observe the second object 7 corresponding to itself, the coordinates, posture, and direction of the second object 7 are set to the state of the first object 6 corresponding to the first user U1. Regardless, you can change it to make it easier to see.

The virtual space experience system of the present invention is not limited to such a configuration, and is independent of at least one of the coordinates, posture, and direction of the first object, and the corresponding coordinates, posture, and the second object. It may be configured to change at least one of the directions.

Therefore, unlike the present embodiment, it is not always necessary to make all the coordinates, postures, and directions of the second object change independently of the coordinates, postures, and directions of the first object, and the state of the second object does not necessarily have to be changed. For any one or two of them, the state of the first object may be configured to be changed according to the change of the state of the second object.

With such a configuration, for example, when the first user plays the role of a teacher and the second user plays the role of a student as in the present embodiment, the first user changes the state according to the change in the state of the second object. By observing the first object, the degree of understanding of the second user can be estimated.

[Processing when the third object is operated]
Next, referring to FIGS. 2, 10 and 11, when the second user U2 operates the third object 8 via the second avatar 5b after the use of the VR system S is started, the VR The processing executed by each processing unit of the system S will be described.

In this process, first, the user state recognition unit 31 determines whether or not the state of the second user U2 has changed (FIG. 10 / STEP400).

If the state of the second user U2 has not changed (NO in STEP400), the process returns to STEP400, and the determination of STEP400 is executed again in a predetermined control cycle.

On the other hand, when the state of the second user U2 changes (YES in STEP 400), the avatar control unit 32 changes the state of the second avatar 5b based on the change in the state of the second user U2 (FIG. 10). / STEP401).

Next, the object control unit 34 determines whether or not the change in the state (that is, the operation) of the second avatar 5b executed by the avatar control unit 32 is an operation such as operating the third object 8. (Fig. 10 / STEP402).

Specifically, the object control unit 34 is based on whether or not the change in the coordinates, posture, and direction of the second avatar 5b with respect to the third object 8 in the virtual space VS corresponds to the change in the predetermined coordinates, posture, and direction. Then, it is determined whether or not the operation of the second avatar 5b is an operation such as operating the third object 8.

In the present embodiment, for example, as shown in FIG. 11, the object control unit 34 is an operation in which the hand of the second avatar 5b grabs or moves any of the parts included in the third object 8. Whether or not it is determined.

When the operation of the second avatar 5b is an operation such as operating the third object 8 (YES in STEP 402), the third object control unit 34c of the object control unit 34 is based on the operation by the second avatar 5b. , The shape, coordinates, posture and direction of the third object 8 are changed (FIG. 10 / STEP403).

Here, the third object 8 is an object for the second user U2 to operate, and the change in the state according to the result of the operation is independent of the first object 6 and the second object 7. ing. Therefore, as shown in FIG. 11, even if the state of the third object 8 changes, the state of the first object 6 or the second object 7 does not change in synchronization with it.

Next, the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 make the user U recognize the image and sound based on the state of the third object 8 and the state of the avatar 5 in the virtual space VS. (Fig. 10 / STEP404).

Similarly, when the operation of the second avatar 5b is not the operation of operating the third object 8 (NO in STEP402), the image determination unit 35a and the voice determination unit 35b of the output information determination unit 35 of the server 3 , The image and sound to be recognized by the user U are determined based on the state of the third object 8 and the state of the avatar 5 in the virtual space VS (FIG. 10 / STEP404).

Next, the HMD4 worn by the user U displays the determined image on the monitor 40, generates the determined sound on the speaker 41 (FIG. 10 / STEP405), and ends the current process. do.

The VR system S repeatedly executes the above processing in a predetermined control cycle until the end instruction by the user U is recognized.

As described above, the third object 8 is generated in the virtual space VS, but is generated so as to overlap the item I that actually exists in the second real space RS2. In addition, the change in the state of the third object 8 also corresponds to the item I. That is, the change in the state of the third object 8 corresponds to the change in the state of the item I.

Therefore, for example, the second user U2 observes the assembling method through the second object 7, and then operates the third object 8 according to the method, so that the VR system S is not assembled before the start of use. Item I will be in the assembled state after use.

That is, in the VR system S, the result of the operation performed by the second user U2 on the third object 8 via the second avatar 5b in the virtual space VS is reflected in the item I corresponding to the second real space RS2. become. As a result, the second user U2 can feel as if the result of the operation of the second avatar 5b in the virtual space VS is reflected in the item I of the second real space RS2.

Further, in this configuration, when the second user U2 touches the third object 8, the second user U2 touches the item I in the second real space RS2. As a result, the second user U2 can be given the feeling of touching the third object 8. As a result, the immersive feeling of the second user U2 in the virtual space VS can be enhanced.

Further, when the first user U1 and the second user U2 exist in the same real space, or when the third object 8 is generated corresponding to the item existing in the first real space RS1. Can also touch the third object 8 by the first user U1.

In such a case, the first user U1 can feel as if the result of the operation of the first avatar 5a in the virtual space VS is reflected in the item I in the real space. In addition, the first user U1 can be given the feeling of touching the third object 8. As a result, the immersive feeling of the first user U1 in the virtual space VS can be enhanced.

In the present embodiment, as described above, even if the state of either the first object 6 or the second object 7 changes, the state of the third object 8 does not change in synchronization with the change. However, when the state of the first object 6 is also changed in response to the change of the state of the third object 8 (synchronically), the operation of the second user U2 on the third object 8 is displayed. The first user U1 can use the object such as grasping it through the first object 6 corresponding to the first user U1.

[Modification example]
Next, a modified example of the VR system S will be described with reference to FIG. In this modification, only the color of the second object and the coordinates of the third object 8 are different from the configuration of the above embodiment. Therefore, in the following description, only the different configurations will be described in detail. The same configurations as those in the above-described embodiment or the corresponding configurations are designated by the same reference numerals, and detailed description thereof will be omitted.

As shown in FIG. 12, the second object 7 and the third object 8 are generated by using the item I, which is a model of a prefabricated car, as a model.

The coordinates, posture, and direction of the first part 70, which is the main body part of each part constituting the second object 7, are the coordinates of the first part 80, which is the main body part of each part constituting the third object 8. It is set to match the posture and direction. Therefore, the first part 70 of the second object 7 and the first part 80 of the third object 8 are superimposed in the virtual space VS and exist.

Then, the shape of the second object 7 changes according to the change in the shape of the first object 6 corresponding to the first user U1 (that is, the operation of the first user U1). On the other hand, the shape of the third object 8 changes according to the operation of the second user U2.

Therefore, when the shape of the first object 6 changes, only the shape of the second object 7 out of the superimposed second object 7 and the third object 8 changes in response to the change.

Specifically, for example, it is assumed that the second part 61, which is a roof portion, is attached to the first part 60, which is the main body portion of the first object 6, by the first user U1. In this case, the second part 71, which is the roof portion of the second object 7, is also automatically attached to the first part 70, which is the main body portion.

Here, unlike the third object 8, the second object 7 is generated as a semi-transparent object. Therefore, when the shape of the first object 6 changes, only the changed part (that is, only the difference between the shape of the first object 6 and the shape of the third object 8) is sent to the second user U2. It will be shown as a translucent part.

As a result, the second user U2 can intuitively grasp the difference between the shape of the first object 6 and the shape of the third object 8 (and by extension, the operation of the first user U1 on the first object 6). can. Specifically, the second user U2 can easily grasp the assembly procedure of the first user U1 by observing the difference portion.

The color of the second object 7 does not necessarily have to be translucent, and may be a color different from the color of the third object 8.

[Other Embodiments]
Although the illustrated embodiment has been described above, the present invention is not limited to such an embodiment.

For example, in the above embodiment, as shown in FIG. 3, the VR system S is not only the first object 6 observed by the first user U1 and the second object 7 observed by the second user U2, but also the second object. The user U2 operates to generate a third object 8 corresponding to the item I and a fourth object 9 corresponding to the table T on which the item I is placed in the virtual space VS.

This is because the second user U2, who is the student role, is the third object 8 (and thus, according to the method shown through the first object 6 (and thus the second object 7) by the first user U1 who is the teacher role. As a result, the item I) that actually exists can be operated.

However, the virtual space experience system of the present invention is not limited to such a configuration, and creates a first object observed by the first user and a second object observed by the second user in the virtual space. Anything is fine.

Therefore, when it is not necessary to operate an existing item, it is not necessary to generate the corresponding third object and the fourth object on which the third object is placed. Further, since it is not necessary to recognize the third object and the fourth object, the processing executed by the item recognition unit and the third object control unit and the fourth object control unit of the object control unit is omitted. May be good.

1 ... sign, 2 ... camera, 2a ... 1st camera, 2b ... 2nd camera, 3 ... server, 4 ... HMD, 5 ... avatar, 5a ... 1st avatar, 5b ... 2nd avatar, 6 ... 1st object, 7 ... 2nd object, 8 ... 3rd object, 9 ... 4th object, 30 ... Virtual space generation unit, 31 ... User state recognition unit, 31a ... User coordinate recognition unit, 31b ... User attitude recognition unit, 31c ... User direction Recognition unit, 32 ... Avatar control unit, 33 ... Item state recognition unit, 33a ... Item shape recognition unit, 33b ... Item coordinate recognition unit, 33c ... Item attitude recognition unit, 33d ... Item direction recognition unit, 34 ... Object control unit, 34a ... 1st object control unit, 34b ... 2nd object control unit, 34c ... 3rd object control unit, 34d ... 4th object control unit, 35 ... output information determination unit, 35a ... image determination unit, 35b ... voice determination unit , 40 ... monitor, 41 ... speaker, 60, 70, 80 ... first part, 61, 71, 81 ... second part, 62, 72, 82 ... third part, 63, 73, 83 ... fourth part, I ... Item, U ... User, U1 ... 1st user, U2 ... 2nd user, RS ... Real space, RS1 ... 1st real space, RS2 ... 2nd real space, S ... VR system (virtual space experience system), T ... table, VS ... virtual space.

Claims (4)

A predetermined value generated in the virtual space through the operation of either the first avatar or the second avatar generated in the virtual space corresponding to the first user and the second user existing in the real space. A virtual space experience system that can change at least one of the coordinates, orientation, and direction of an object, as well as a state that includes at least a shape.
The virtual space, and a virtual space generation unit that generates the first avatar, the second avatar, and the predetermined object existing in the virtual space.
A user state recognition unit that recognizes the state of the first user and the state of the second user,
An avatar control unit that controls the state of the first avatar according to the state of the first user and controls the state of the second avatar according to the state of the second user.
An object control unit that controls the state of the predetermined object, and
An image determination unit that determines an image of the virtual space to be recognized by the first user and the second user based on the state of the first avatar, the state of the second avatar, and the state of the predetermined object.
The first user and the second user are provided with an image display for recognizing an image of the virtual space.
The predetermined object includes a first object observed by the first user and a second object observed by the second user.
The second object is located in the virtual space at coordinates different from the coordinates of the first object, and has a shape corresponding to the first object.
The state of the first object can be changed by the first user through the operation of the first avatar.
The object control unit
When the shape of the first object is changed, the shape of the second object is changed according to the change in the shape of the first object.
When a state other than the shape of the first object is changed, the state of the second object other than the fixed state is changed while at least one of the coordinates, the posture, and the direction of the second object is fixed. A virtual space experience system characterized in that it changes according to a change in the state of the first object. In the virtual space experience system according to claim 1,
The state of the second object can be changed by the second user through the action of the second avatar.
The object control unit changes at least one of the corresponding coordinates, posture, and direction of the second object independently of at least one of the coordinates, posture, and direction of the first object. Experience system. In the virtual space experience system according to claim 1 or 2.
The predetermined object includes a third object operated by the second user.
The color of the second object is translucent or a color different from that of the third object.
The third object has a shape corresponding to the second object and has a shape corresponding to the second object.
The shape of the third object can be changed by the second user through the operation of the second avatar.
A virtual space experience system characterized in that the coordinates, posture, and direction of the third object match the coordinates, posture, and direction of the second object. In the virtual space experience system according to any one of claims 1 to 3.
It is provided with an item state recognition unit that recognizes the state of an item existing in the real space.
The predetermined object includes a third object operated by at least one of the first user and the second user.
The virtual space generation unit is a virtual space experience system characterized in that the third object is generated in correspondence with the item.

Images