JP2024047008A - Information Processing System - Google Patents

Abstract

[Problem] To broaden the range of uses of virtual space. [Solution] An information processing system includes a first display means for providing a first user with a first view, which is a view of a virtual space simulating a predetermined real space, a second display means for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object corresponding to a first virtual object in the virtual space is placed in the predetermined real space, a first input receiving means for receiving input from the first user, a first control means, and a second control means, the first control means controls the first virtual object in the virtual space based on the input from the first user received by the first input receiving means, and the second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on the input from the first user received by the first input receiving means. [Selected Figure] Figure 8

Description

The present invention relates to an information processing system and a program.

Technologies that provide users with mixed reality (MR) spaces and virtual reality (VR) spaces have been known (see, for example, Patent Documents 1 and 2).

Japanese Patent No. 6860488 Patent No. 6800599

However, there has long been a demand for expanding the range of uses for virtual space.

The present invention was made in consideration of the above circumstances, and aims to broaden the range of uses for virtual space.

According to one embodiment shown in the present disclosure,
a first display means for providing a first user with a first view, the first view being a view of a virtual space simulating a predetermined real space;
a second display means for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object corresponding to the first virtual object in the virtual space is arranged in the predetermined real space;
a first input receiving means for receiving an input from the first user;
a first control means for controlling a virtual object in the virtual space;
a second control means for controlling a virtual object in the mixed reality space,
the first control means controls the first virtual object in the virtual space based on an input from the first user accepted by the first input accepting means;
An information processing system is provided in which the second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on input from the first user accepted by the first input accepting means.

This invention can broaden the range of uses for virtual space.

FIG. 1 is a diagram illustrating a schematic configuration of an information processing system. FIG. 1 is a diagram showing a schematic configuration of a VR system. FIG. 1 is a diagram showing a schematic configuration of an MR system. FIG. 1 is a diagram conceptually illustrating a virtual space. 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. 1 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from a Y direction. FIG. 2 is a diagram illustrating a schematic configuration of a controller. FIG. 2 is a diagram illustrating a mixed reality space and a virtual space. FIG. 2 is a diagram illustrating a functional configuration of the information processing system. 1A and 1B are diagrams illustrating a real object and a virtual object corresponding to the real object. 13 is a flowchart illustrating an example of a process executed by the VR system. 13 is a flowchart illustrating an example of a process executed by the MR system. 13 is a flowchart showing an example of a process for reflecting the behavior of an MR user in a virtual space. 13 is a flowchart illustrating an example of a process for reflecting the behavior of a VR user in a mixed reality space.

The following describes an embodiment of the present invention with reference to the drawings.

<Configuration of Information Processing System>
As shown in FIG. 1, the information processing system 100 of the present embodiment includes a VR (Virtual Reality) system 200, an MR (Mixed Reality) system 400, a server 600, and an external device 700. The VR system 200 is configured to be able to communicate with the server 600, the MR system 400, and the external device 700 via a network 2. The MR system 400 is configured to be able to communicate with the server 600, the VR system 200, and the external device 700 via the network 2. The number of MR systems 400 constituting the information processing system 100 is not limited to one, and may be multiple. The number of VR systems 200 constituting the information processing system 100 is not limited to one, and may be multiple. In the present embodiment, communication between the VR system 200 and the MR system 400 via the network 2 is performed via the server 600, but may be performed without the server 600. The network 2 may be composed of, for example, the Internet, a mobile communication system (e.g., 3G, 4G, 5G, LTE (Long Term Evolution), etc.), Wi-Fi (registered trademark), Bluetooth (registered trademark), other communication lines, or a combination of these.

(VR system configuration)
2, the VR system 200 includes a VR device 210, a computer 300, a detection device 260, a display 270, and a controller 280. The VR device 210 includes a display 211, a gaze sensor 212, a first camera 213, a second camera 214, a microphone 215, a speaker 216, and a sensor 217. Hereinafter, a user who uses the VR device 210 is referred to as a VR user.

The computer 300 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the MR system 400, and other computers connected to the network 2. Examples of other computers include computers of other VR systems 200 and external devices 700.

The VR device 210 is worn on the head of a VR user and can provide the VR user with a virtual space during operation. The VR device 210 can be, for example, a so-called head-mounted display equipped with a display, or a head-mounted device equipped with a smartphone or other terminal having a display. The VR device 210 displays, for example, an image for the right eye and an image for the left eye on the display 211. When each eye of the VR user views the respective image, the VR user can recognize the image as a three-dimensional image based on the parallax between the two eyes.

The display 211 is realized, for example, as a non-transparent display device. The display 211 is arranged on the main body of the VR device 210 so as to be located, for example, in front of both eyes of the VR user. Therefore, when the VR user visually recognizes the three-dimensional image displayed on the display 211, he or she can become immersed in the virtual space. Note that the display 211 may also be realized by a display provided on a so-called smartphone or other terminal.

The detection device 260 detects the movement of a VR user who uses the VR device 210. The detection device 260 may have a position tracking function for detecting the movement of the VR device 210, thereby detecting the movement of the VR user. Specifically, the detection device 260 may have, for example, a sensor that reads light (e.g., infrared light) from the VR device 210 as a sensor for detecting the movement of the VR user, and detect the position, inclination, etc. of the VR device 210 in real space. In this case, the VR device 210 may have multiple light sources not shown. Furthermore, each light source may be realized, for example, by an LED (Light Emitting Diode) that emits infrared light.

The detection device 260 may also be realized by, for example, a camera. Specifically, the detection device 260 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor, etc.) as a sensor for detecting the movement of the VR user. In other words, the movement of the VR user may be detected by a camera. The detection device 260 may be, for example, a depth camera equipped with a device that emits a predetermined light such as infrared light or a predetermined pattern of light, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the VR user based on the output from the image sensor. The detection device 460 may also be equipped with such a depth camera and a camera that can acquire an RGB image, and may detect the position, posture, etc. of the VR user based on the output from these cameras. The detection device 260 may also be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the VR user based on the output from the multiple image sensors. Note that detecting the position, posture, etc. of the VR user may be detecting the position, posture, etc. of the VR user's body, or may be detecting the position, tilt, etc. of the VR device 210.

The VR system 200 may have one or more types of detection devices as the detection device 260, and may have multiple detection devices of each type. Furthermore, the VR system 200 does not have to have the detection device 260. A part of the detection device 260 may be configured by the computer 300 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 300 or the like.

In addition to the various sensors described above, the detection device 260 may have a sensor capable of detecting the position, inclination, etc. of the detection device 260 itself. Specifically, for example, the detection device 260 may have an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 300, etc., and may be used, for example, when a predetermined process is performed based on the output from an image sensor provided in the detection device 260.

The VR device 210 may also include a sensor 217 instead of or in addition to the detection device 260 as a detection means for detecting the movement of the VR user. The VR device 210 may use the sensor 217 to detect the position and inclination of the VR device 210 itself. The sensor 217 may be, for example, an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). The VR device 210 may also have one or more types of sensors as the sensor 217, and may have a plurality of sensors of each type. As an example, when an angular velocity sensor is used as the sensor 217, the VR device 210 can detect the angular velocity of the VR device 210 around the three axes in the real space over time. Then, the VR device 210 can calculate the change over time in the angle of the VR device 210 around the three axes based on each angular velocity, and further calculate the inclination of the VR device 210 based on the change over time in the angle. Furthermore, the sensor 217 may be, for example, an image sensor. The position, posture, etc. of the VR user may be detected based on the output from the image sensor. In other words, the position, posture, etc. of the VR user may be detected based on information from a camera that is provided in the VR device 210 and captures the surroundings of the VR device 210. In other words, tracking of the VR device 210 may be performed by an outside-in method or an inside-out method.

The gaze sensor 212 detects the direction in which the gaze of the right and left eyes of the VR user is directed. That is, the gaze sensor 212 detects the gaze of the VR user (in other words, the movement of the eyes). The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 212 is realized by a sensor having the eye tracking function. The gaze sensor 212 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 212 may be, for example, a sensor that irradiates the right and left eyes of the VR user with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. In this case, the gaze sensor 212 can detect the gaze of the VR user based on each detected rotation angle.

The first camera 213 photographs the lower part of the VR user's face. More specifically, the first camera 213 photographs the nose, mouth, etc. of the VR user. The second camera 214 photographs the eyes, eyebrows, etc. of the VR user. Here, the VR user side of the housing of the VR device 210 is defined as the inside of the VR device 210, and the opposite side of the housing of the VR device 210 from the VR user is defined as the outside of the VR device 210. The first camera 213 may be disposed outside the VR device 210, and the second camera 214 may be disposed inside the VR device 210. The images photographed by the first camera 213 and the second camera 214 are input to the computer 300. Note that the first camera 213 and the second camera 214 may be realized as a single camera, and the face of the VR user may be photographed by this single camera.

The microphone 215, serving as a sound input means, converts the voice of the VR user into a sound signal (in other words, an electrical signal) and outputs it to the computer 300. The speaker 216, serving as a sound output means, converts the sound signal into sound and outputs it to the VR user. Note that the VR device 210 may include earphones instead of the speaker 216 as a sound output means.

The display 270 displays an image similar to the image displayed on the display 211. This allows users other than the VR user wearing the VR device 210 to view the same image (in other words, a virtual space) as the VR user. The image displayed on the display 270 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the VR device 210. Examples of the display 270 include a liquid crystal display and an organic EL display.

The controller 280 is connected to the computer 300 by wire or wirelessly. The controller 280 accepts input operations related to instructions from the VR user to the computer 300. The controller 280 also accepts input operations by the VR user to control the position and movement of a virtual object placed in a virtual space. The controller 280 may be configured to be held by the VR user, for example. The controller 280 may also be configured to be wearable on the body or a part of the clothing of the VR user, for example. Specifically, the controller 280 may be, for example, a glove type controller. The controller 280 may also be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 300.

The controller 280 may also have multiple light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 260 may then read the infrared light from the controller 280 and detect the position and inclination of the controller 280 in real space. In other words, the detection device 260 may have a position tracking function that detects the movement of the controller 280, thereby detecting the movement of the VR user.

Note that the VR system 200 does not necessarily have to include the controller 280.

The information processing system 100 may also have a sensor 286 that detects the movement of the VR user. The sensor 286 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 286 may also be provided in the controller 280 (see FIG. 7). The information processing system 100 may have one or more types of sensors as the sensor 286, and may have multiple sensors of each type. The device including the sensor 286 (for example, the controller 280 or a specific camera) and the computer 300 may be connected to each other, for example, wirelessly. The information acquired by the sensor 286 may also be transmitted to the computer 300, for example, by wireless communication.

The position and movement of the VR user may also be detected by the VR device 210 being equipped with a device that performs a specific wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB (Ultra Wide Band) communication), and the device performing wireless communication with surrounding devices to obtain position information of the VR device 210.

The position and movement of the VR user may also be detected using an external device 700 (and a device that tracks the movement of the VR user in cooperation with the device via short-range wireless communication or the like) that can track the movement of the VR user, such as a device worn by the VR user (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device, etc.). These external devices 700 may also be used as the controller 280. The position and movement of the VR user may also be detected by a GPS sensor or the like.

(Configuration of computer 300)
2, the computer 300 includes, as its main components, a processor 301, a memory 302, a storage 303, an input/output interface 304, and a communication interface 305. The components are connected to each other via a bus.

The processor 301 controls the operation of the VR device 210. The processor 301 reads a program from the storage 303 and loads it into the memory 302. The processor 301 executes the loaded program. The processor 301 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 302 is a main storage device. Memory 302 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 302 provides a working area for processor 301 by temporarily storing programs and various data that processor 301 reads from storage 303. Memory 302 also temporarily stores various data generated while processor 301 is operating according to a program, various data input to computer 300, etc.

Storage 303 is an auxiliary storage device. Storage 303 is composed of a storage device such as a flash memory or a hard disk drive (HDD). Storage 303 stores programs for providing services in information processing system 100. Storage 303 also stores various data for providing services in information processing system 100. The data stored in storage 303 includes data for defining a virtual space and data related to virtual objects.

The storage 303 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 303 built into the computer 300, programs and data stored in an external storage device may be used.

The input/output interface 304 is an interface through which the computer 300 accepts data input, and is also an interface through which the computer 300 outputs data. The input/output interface 304 may transmit or receive data between the VR device 210, the detection device 260, and the display 270. It may also transmit or receive data between the display 211, the gaze sensor 212, the first camera 213, the second camera 214, the microphone 215, the speaker 216, and the sensor 217 included in the VR device 210.

The input/output interface 304 may transmit or receive data to and from the controller 280. For example, the input/output interface 304 may receive input of signals output from the controller 280 and the sensor 286. The input/output interface 304 may also transmit to the controller 280 an instruction output from the processor 301. The instruction may instruct the controller 280 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 280 performs vibration, output sound, emit light, or the like in accordance with the instruction.

Note that each of the VR device 210, the detection device 260, the display 270, the controller 280, and the sensor 286 may be connected to the computer 300 via a wired or wireless connection.

The communication interface 305 controls the transmission and reception of various data with other computers (e.g., the server 600, the computer 500, or other computers 300, etc.) via the network 2.

The processor 301 accesses the storage 303, loads a program stored in the storage 303 into the memory 302, and executes a series of instructions contained in the program. The processor 301 also sends a signal to the VR device 210 via the input/output interface 304 to provide a virtual space.

The computer 300 may be provided outside the VR device 210, or a part or the whole of the computer 300 may be built into the VR device 210. Also, a portable terminal (e.g., a smartphone) including a display 211 may function as at least a part of the computer 300. Note that when there are multiple VR devices 210, the computer 300 may be provided for each VR device 210, or may be shared by the multiple VR devices 210.

(Coordinate system)
A real coordinate system, which is a coordinate system in real space, is set in advance in the VR system 200. The real coordinate system has three reference directions (axes) that are parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-rear direction perpendicular to both the vertical and horizontal directions.

The position and inclination of the VR device 210 in real space can be detected by the detection device 260 and the sensor 217. The detected inclination of the VR device 210 corresponds to, for example, each of the inclinations around the three axes of the VR device 210 in the real coordinate system. The computer 300 sets a uvw field of view coordinate system for the VR device 210 based on the inclination of the VR device 210 in the real coordinate system (see FIG. 4). The uvw field of view coordinate system set for the VR device 210 corresponds to the viewpoint coordinate system when a VR user wearing the VR device 210 views an object in a virtual space.

Now, the uvw field of view coordinate system will be explained. The computer 300 sets a three-dimensional uvw field of view coordinate system with the head of the VR user wearing the VR device 210 as its center (origin). More specifically, the computer 300 tilts the horizontal, vertical, and front-to-rear directions that define the real coordinate system around each axis by the tilt of the VR device 210 around each axis in the real coordinate system, and sets the three newly obtained directions as the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the VR device 210.

For example, when a VR user wearing the VR device 210 stands upright and looks straight ahead, the processor 301 sets a field of view coordinate system for the VR device 210 that is parallel to the real coordinate system. In this case, the horizontal direction, vertical direction, and front-to-back directions in the real coordinate system coincide with the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the field of view coordinate system in the VR device 210.

After the uvw field of view coordinate system is set in the VR device 210, the detection device 260 or the sensor 217 can detect the tilt of the VR device 210 in the set uvw field of view coordinate system based on the movement of the VR device 210. In this case, the detection device 260 or the sensor 217 detects the pitch angle, yaw angle, and roll angle of the VR device 210 in the uvw field of view coordinate system as the tilt of the VR device 210. The pitch angle represents the tilt angle of the VR device 210 around the pitch axis in the uvw field of view coordinate system. The yaw angle represents the tilt angle of the VR device 210 around the yaw axis in the uvw field of view coordinate system. The roll angle represents the tilt angle of the VR device 210 around the roll axis in the uvw field of view coordinate system.

Based on the detected inclination of the VR device 210, the computer 300 sets the uvw field of view coordinate system of the VR device 210 after the VR device 210 moves to the VR device 210. The relationship between the VR device 210 and the uvw field of view coordinate system of the VR device 210 is always constant, regardless of the position and inclination of the VR device 210. When the position and inclination of the VR device 210 change, the position and inclination of the uvw field of view coordinate system of the VR device 210 in the real coordinate system change in conjunction with the change in the position and inclination.

In addition, the detection device 260 may identify the position of the VR device 210 in real space as a relative position with respect to the detection device 260. Furthermore, the processor 301 may determine the origin of the uvw field of view coordinate system of the VR device 210 in real space (actual coordinate system) based on the identified relative position.

(Virtual space)
The virtual space will be further described with reference to FIG. 4. FIG. 4 is a diagram conceptually showing one mode of expressing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12. In FIG. 4, in order to avoid complicating the description, the upper half of the celestial sphere in the virtual space 11 is illustrated. Each mesh is defined in the virtual space 11. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 300 associates each partial image constituting the panoramic image 13 (still image, video, etc.) that can be deployed in the virtual space 11 with each corresponding mesh in the virtual space 11.

In the virtual space 11, an XYZ coordinate system is defined with a specific point as the origin 12. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis of the XYZ coordinate system is parallel to the horizontal direction of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the vertical direction of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the front-to-back direction of the real coordinate system.

When the VR device 210 is started up, i.e., in the initial state of the VR device 210, the virtual camera 14 is placed at a predetermined position (e.g., the center) in the virtual space 11. The processor 301 also displays an image captured by the virtual camera 14 on the display 211 of the VR device 210. The virtual camera 14 moves within the virtual space 11 in conjunction with the movement of the VR device 210 in the real space. This allows changes in the tilt and position of the VR device 210 in the real space to be reproduced in the virtual space 11 in the same way.

A uvw field of view coordinate system is defined for the virtual camera 14, as in the case of the VR device 210. The uvw field of view coordinate system of the virtual camera 14 in the virtual space 11 is defined so as to be linked to the uvw field of view coordinate system of the VR device 210 in real space (actual coordinate system). Therefore, when the inclination of the VR device 210 changes, the inclination of the virtual camera 14 also changes accordingly. Note that the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in real space, but in this embodiment, it does not move in the virtual space 11 even if the VR user moves in real space.

The processor 301 of the computer 300 determines the viewing area 15 in the virtual space 11 based on the position and inclination of the virtual camera 14. The viewing area 15 corresponds to the area of the virtual space 11 that is viewed by a VR user wearing the VR device 210. In other words, the position of the virtual camera 14 can be said to be the viewpoint of the VR user in the virtual space 11.

The uvw field of view coordinate system of the VR device 210 is equal to the viewpoint coordinate system when the VR user views the display 211. The uvw field of view coordinate system of the virtual camera 14 is linked to the uvw field of view coordinate system of the VR device 210. Therefore, the line of sight of the VR user detected by the gaze sensor 212 can be regarded as the line of sight of the VR user in the uvw field of view coordinate system of the virtual camera 14.

(Visibility area)
The field of view 15 will be described with reference to Fig. 5 and Fig. 6. Fig. 5 is a diagram showing a YZ cross section of the field of view 15 in the virtual space 11 as viewed from the X direction. Fig. 6 is a diagram showing an XZ cross section of the field of view 15 in the virtual space 11 as viewed from the Y direction.

As shown in FIG. 5, the field of view 15 in the YZ cross section includes an area 18. The area 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 301 defines the range including the polar angle α centered on the reference line of sight 16 in the virtual space as the area 18.

As shown in FIG. 6, the field of view 15 in the XZ cross section includes area 19. Area 19 is defined by the position of virtual camera 14, reference line of sight 16, and the XZ cross section of virtual space 11. Processor 301 defines a range including azimuth angle β centered on reference line of sight 16 in virtual space 11 as area 19. Polar angles α and β are determined according to the position of virtual camera 14 and the inclination (direction) of virtual camera 14.

The VR system 200 provides the VR user with a field of view in the virtual space 11 by displaying a field of view image 17 on the display 211 based on a signal from the computer 300 (see FIG. 4). The field of view image 17 is an image that corresponds to the portion of the panoramic image 13 that corresponds to the field of view area 15. When the VR user moves the VR device 210 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the field of view area 15 in the virtual space 11 changes. As a result, the field of view image 17 displayed on the display 211 is updated to an image of the panoramic image 13 that is superimposed on the field of view area 15 in the direction in which the VR user is facing in the virtual space 11. The VR user can view the desired direction in the virtual space 11.

While wearing the VR device 210, the VR user can only view the panoramic image 13 displayed in the virtual space 11, without being able to see the real world. Therefore, the information processing system 100 can give the VR user a highly immersive feeling in the virtual space 11.

The virtual camera 14 may include two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In this case, an appropriate parallax is set for the two virtual cameras so that the VR user can recognize the three-dimensional virtual space 11. The virtual camera 14 may also be realized by a single virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by the single virtual camera.

(controller)
An example of the controller 280 will now be described with reference to FIG.

As shown in FIG. 7, the controller 280 may include a right controller 280R and a left controller (not shown). In this case, the right controller 280R is operated by the right hand of the VR user. The left controller is operated by the left hand of the VR user. In one aspect, the right controller 280R and the left controller are configured as separate devices. Therefore, the VR user can freely move both the right hand holding the right controller 280R and the left hand holding the left controller. In another aspect, the controller 280 may be an integrated controller that accepts operation from both hands. The right controller 280R will be described below.

The right controller 280R includes a grip 281, a frame 282, a top surface 283, buttons 284 and 285, a sensor (e.g., a motion sensor) 286, an infrared LED 287, buttons 288 and 289, and an analog stick 290. The grip 281 is configured to be held by the right hand of a VR user. For example, the grip 281 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the VR user's right hand.

Button 284 is disposed on the side of grip 281 and is operable by the middle finger of the right hand. Button 285 is disposed on the front of grip 281 and is operable by the index finger of the right hand. Buttons 284 and 285 are also equipped with switches that detect the movement of the user pressing buttons 284 and 285. Buttons 284 and 285 may also be configured as trigger-type buttons.

The sensor 286 is built into the housing of the grip 281. The sensor 286 detects the movement of the VR user. Specifically, the sensor 286 detects the movement of the VR user's hand. For example, the sensor 286 detects the rotation speed and number of rotations of the hand. Note that the controller 280 does not necessarily have to include the sensor 286.

A number of infrared LEDs 287 are arranged along the circumferential direction of the frame 282. When a program using the controller 280 is being executed, the infrared LEDs 287 emit infrared light in accordance with the progress of the program. The infrared light emitted from the infrared LEDs 287 can be used to detect the positions and attitudes (tilt, direction) of the right controller 280R and the left controller.

The top surface 283 is equipped with buttons 288, 289 and an analog stick 290. The buttons 288, 289 are operated by the thumb of the VR user's right hand. The buttons 288, 289 are also equipped with switches that detect the movement of the user pressing the buttons 288, 289. The analog stick 290 is equipped with operations in any direction within 360 degrees from the initial position (neutral position). Such operations include, for example, operations for moving an object placed in the virtual space 11. The analog stick 290 is also equipped with a sensor that detects the movement of the user operating the analog stick 290.

Information regarding the VR user's input operations (in other words, the VR user's movements) detected by the controller 280 is sent to the computer 300.

As shown in FIG. 7, the controller 280 defines the yaw, roll, and pitch directions for, for example, the right hand of a VR user. Specifically, for example, when a VR user extends his or her thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the yaw direction axis and the roll direction axis is defined as the pitch direction.

(Configuration of MR System)
3, the MR system 400 includes an MR device 410, a computer 500, a detection device 460, a display 470, and a controller 480. The MR device 410 includes a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417. Hereinafter, a user who uses the MR device 410 is referred to as an MR user 6.

The computer 500 can be connected to the Internet or other network 2, and can communicate with, for example, a server 600, a computer of the VR system 200, and other computers connected to the network 2. Examples of other computers include computers of other MR systems 400 and external devices 700.

The MR device 410 is worn on the head of the MR user 6 and can provide a mixed reality space to the MR user 6 during operation. The MR device 410 may be, for example, a glasses-type device equipped with a display (for example, so-called MR glasses). The MR device 410 may also be a contact lens-type device, etc.

The display 411 is realized as, for example, a transmissive display device. The transmissive display 411 may temporarily function as a non-transmissive display device by adjusting its transmittance. The display 411 is disposed on the main body of the MR device 410 so as to be located, for example, in front of both eyes of the MR user 6. Therefore, the MR user 6 is provided with a display in which a virtual object displayed by the display 411 is superimposed on the real space seen through the transmissive display 411. That is, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is disposed in the real space. In other words, the MR user 6 can simultaneously see a virtual object and an object existing in the real space (hereinafter referred to as a "real object") by the MR device 410.

The detection device 460 detects the movement of the MR user 6 who uses the MR device 410. The detection device 460 may have a position tracking function for detecting the movement of the MR device 410, thereby detecting the movement of the MR user 6. Specifically, the detection device 460 may have, for example, a sensor that reads light (e.g., infrared light) from the MR device 410 as a sensor for detecting the movement of the MR user 6, and detect the position, inclination, etc. of the MR device 410 in the mixed reality space. In this case, the MR device 410 may have multiple light sources not shown. Furthermore, each light source may be realized, for example, by an LED (Light Emitting Diode) that emits infrared light.

The detection device 460 may also be realized by, for example, a camera. Specifically, the detection device 460 may have an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a black and white image, or a depth sensor, etc.) as a sensor for detecting the movement of the MR user 6. In other words, the movement of the MR user 6 may be detected by a camera. The detection device 460 may be, for example, a depth camera equipped with a device that emits a predetermined light such as infrared light or a predetermined pattern of light, and an image sensor (for example, a depth sensor), and may detect the reflected light of the light emitted from the device by the image sensor, and may detect the position, posture, etc. of the MR user 6 based on the output from the image sensor. The detection device 460 may also be equipped with such a depth camera and a camera that can acquire an RGB image, and may detect the position, posture, etc. of the MR user 6 based on the output from these cameras. The detection device 460 may also be, for example, a stereo camera equipped with multiple image sensors, and may detect the position, posture, etc. of the MR user 6 based on the output from the multiple image sensors. Note that detection of the position, posture, etc. of the MR user 6 may mean detection of the body position, posture, etc. of the MR user 6, or detection of the position, inclination, etc. of the MR device 410.

The MR system 400 may have one or more types of detection devices as the detection device 460, and may have multiple detection devices of each type. For example, the MR system 400 may have multiple cameras (in other words, multiple sensors), and the multiple cameras may function as the detection device 460. In this embodiment, the MR system 400 is described as having three detection devices 460, and each detection device 460 is described as having a depth camera and a camera capable of acquiring RGB images. A part of the detection device 460 may be configured by the computer 500 or the like. For example, analysis of the output from the image sensor (e.g., image recognition, etc.) may be performed by the computer 500 or the like. The MR system 400 may not have the detection device 460.

In addition to the various sensors described above, the detection device 460 may have a sensor capable of detecting the position, inclination, etc. of the detection device 460 itself. Specifically, for example, the detection device 460 may have an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). In addition, the output from these sensors may be sent to the computer 500 or the like, and may be used, for example, when a predetermined process is performed based on the output from an image sensor provided in the detection device 460.

Also, as will be understood from the configuration of the MR device 410 described below, the MR device 410 of another MR user 6 may be used as the detection device 460.

The MR device 410 may also include a sensor 417 instead of or in addition to the detection device 460 as a detection means for detecting the movement of the MR user 6. The MR device 410 may use the sensor 417 to detect the position and inclination of the MR device 410 itself. The sensor 417 may be, for example, an angular velocity sensor (e.g., a three-axis angular velocity sensor), an acceleration sensor (e.g., a three-axis acceleration sensor), or a geomagnetic sensor (e.g., a three-axis geomagnetic sensor). The MR device 410 may also include one or more types of sensors as the sensor 417, and may include multiple sensors of each type.

The gaze sensor 412 detects the direction in which the right and left eyes of the MR user 6 are directed. That is, the gaze sensor 412 detects the gaze (in other words, the movement of the eyes) of the MR user 6. The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 412 is realized by a sensor having the eye tracking function. The gaze sensor 412 may include a sensor for the right eye and a sensor for the left eye. The gaze sensor 412 may be, for example, a sensor that irradiates the right and left eyes of the MR user 6 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. In this case, the gaze sensor 412 can detect the gaze of the MR user 6 based on each detected rotation angle.

The camera 413 captures the surroundings (e.g., in front) of the MR user 6. The MR device 410 may include, for example, a depth camera and a camera capable of acquiring RGB images as the camera 413. Based on the output from the camera 413, the computer 500 may detect the shape of objects around the MR device 410 (in other words, the MR user 6) and the relative distance between the MR device 410 and the surrounding objects. The position and inclination of the MR device 410 may also be detected based on the output from the camera 413. In other words, tracking of the MR device 410 may be performed by an outside-in method or an inside-out method.

The microphone 415 as a sound input means converts the voice of the MR user 6 into a sound signal (in other words, an electrical signal) and outputs it to the computer 500. The speaker 416 as a sound output means converts the sound signal into sound and outputs it to the MR user 6. Note that the MR device 410 may include earphones as a sound output means instead of the speaker 416.

The display 470 displays, for example, an image similar to the image displayed on the display 411. This allows users other than the MR user 6 wearing the MR device 410 to view an image similar to that of the MR user 6. The image displayed on the display 470 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed by the MR device 410. Examples of the display 470 include a liquid crystal display and an organic EL display. The display 470 may display an image showing a view of the mixed reality space provided to the MR user 6 by the MR device 410.

The controller 480 is connected to the computer 500 by wire or wirelessly. The controller 480 accepts input operations related to instructions from the MR user 6 to the computer 500. The controller 480 also accepts input operations by the MR user 6 to control the position and movement of a virtual object placed in the mixed reality space. The controller 480 may be configured to be held by the MR user 6, for example. The controller 480 may also be configured to be worn on a part of the body or clothing of the MR user 6, for example. The controller 480 may also be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 500.

The controller 480 may also have multiple light sources. Each light source may be realized, for example, by an LED that emits infrared light. The detection device 460 may then read the infrared light from the controller 480 and detect the position and inclination of the controller 480 within the mixed reality space. In other words, the detection device 460 may have a position tracking function that detects the movement of the controller 480, thereby detecting the movement of the MR user 6.

The MR system 400 does not have to have the controller 480. The controller 480 may have, for example, some or all of the components of the controller 280.

The information processing system 100 may have a sensor 486 that detects the movement of the MR user 6. The sensor 486 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. The sensor 486 may also be provided in the controller 480, for example. The information processing system 100 may have one or more types of sensors as the sensor 486, and may have multiple sensors of each type. The device including the sensor 486 (for example, the controller 480 or a specific camera) and the computer 500 may be connected to each other, for example, wirelessly. The information acquired by the sensor 486 may be transmitted to the computer 300, for example, by wireless communication.

The position and movement of the MR user 6 may also be detected by the MR device 410 being equipped with a device that performs predetermined wireless communication (e.g., short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB communication) and having the device perform wireless communication with surrounding devices to obtain position information of the VR device 210.

The position and movement of the MR user 6 may also be detected using an external device 700 (and a device that tracks the movement of the MR user 6 in cooperation with the device via short-range wireless communication or the like) that can track the movement of the MR user 6, such as a device worn by the MR user 6 (for example, a wearable device such as a watch type, wristband type, ring type, or clothing type, or an implantable device, etc.). These external devices 700 may also be used as the controller 480. The position and movement of the MR user 6 may also be detected by a GPS sensor or the like.

(Configuration of computer 500)
3, the computer 500 includes, as its main components, a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505. The components are connected to each other via a bus.

The processor 501 controls the operation of the MR device 410. The processor 501 reads a program from the storage 503 and loads it in the memory 502. The processor 501 executes the loaded program. The processor 501 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 502 is a main storage device. Memory 502 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 502 provides a working area for processor 501 by temporarily storing programs and various data that processor 501 reads from storage 503. Memory 502 also temporarily stores various data generated while processor 501 is operating according to a program, various data input to computer 500, etc.

Storage 503 is an auxiliary storage device. Storage 503 is composed of a storage device such as a flash memory or a hard disk drive (HDD). Storage 503 stores programs for providing services in information processing system 100. Storage 503 also stores various data for providing services in information processing system 100. Data stored in storage 503 includes data for defining a mixed reality space and data related to virtual objects.

The storage 503 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 503 built into the computer 500, programs and data stored in an external storage device may be used.

The input/output interface 504 is an interface through which the computer 500 accepts data input, and is also an interface through which the computer 500 outputs data. The input/output interface 504 may transmit or receive data between the MR device 410, the detection device 460, and the display 470. The input/output interface 504 may also transmit or receive data between the display 411, the gaze sensor 412, the camera 413, the microphone 415, the speaker 416, and the sensor 417 included in the MR device 410.

The input/output interface 504 may transmit or receive data to and from the controller 480. For example, the input/output interface 504 may receive input of signals output from the controller 480 and the sensor 486. The input/output interface 504 may also transmit to the controller 480 an instruction output from the processor 501. The instruction may instruct the controller 480 to vibrate, output sound, emit light, or the like. Upon receiving the instruction, the controller 480 performs vibration, output sound, emit light, or the like in accordance with the instruction.

Note that each of the MR device 410, the detection device 460, the display 470, the controller 480, and the sensor 486 may be connected to the computer 500 via a wired or wireless connection.

The communication interface 505 controls the transmission and reception of various data with other computers (e.g., the server 600, the computer 300, or other computers 500, etc.) via the network 2.

The processor 501 accesses the storage 503, loads a program stored in the storage 503 into the memory 502, and executes a series of instructions contained in the program. The processor 501 also sends a signal to the MR device 410 via the input/output interface 504 to provide a mixed reality space.

The computer 500 may be provided outside the MR device 410, or a part or the whole of the computer 500 may be built into the MR device 410. Furthermore, when there are multiple MR devices 410, the computer 500 may be provided for each MR device 410, or may be used commonly for the multiple MR devices 410. Furthermore, in this case, a part of the computer 500 may be built into each of the multiple MR devices 410. In this embodiment, an example will be described in which multiple MR devices 410 are connected to the computer 500.

Methods for acquiring the position and inclination of the MR device 410 in real space (in other words, mixed reality space) are well known, and can be detected in a similar manner to the method for the VR device 210 described above, so a detailed explanation is omitted.

(Server configuration)
The server 600 may transmit a program to the computer 300. The server 600 may also transmit a program to the computer 500. The server 600 also enables communication between the computer 300 and the computer 500. The server 600 also enables communication between the computer 300 and other computers 300. For example, when the information processing system 100 provides a service in which multiple VR users can participate, each computer 300 of the VR user may communicate with the other computers 300 via the server 600, thereby enabling multiple VR users to share experiences in the same virtual space. Note that each computer 300 may communicate with the other computers 300 without going through the server 600.

As shown in FIG. 1, the server 600 includes a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605. Each component is connected to each other via a bus.

The processor 601 controls the operation of the entire server 600. The processor 601 reads a program from the storage 603 and loads it into the memory 602. The processor 601 executes the loaded program. The processor 601 may be configured to include, for example, one or more of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 602 is a main storage device. Memory 602 is composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 602 provides a working area for processor 601 by temporarily storing programs and various data that processor 601 reads from storage 603. Memory 602 also temporarily stores various data generated while processor 601 is operating according to a program, various data input to server 600, etc.

Storage 603 is an auxiliary storage device. Storage 603 is composed of a storage device such as a flash memory or a HDD (Hard Disk Drive). Storage 603 stores programs for providing services in information processing system 100. Storage 603 also stores various data for providing services in information processing system 100. Data stored in storage 603 includes data for defining a virtual space, data for defining a mixed reality space, data related to virtual objects, and the like.

The storage 603 may be implemented as a removable storage device such as a memory card. Also, instead of the storage 603 built into the server 600, programs and data stored in an external storage device may be used.

The input/output interface 604 is an interface through which the server 600 accepts data input, and is also an interface through which the server 600 outputs data. The input/output interface 604 can transmit or receive data between, for example, input devices such as a mouse or keyboard, and output devices such as a display.

The communication interface 605 controls the sending and receiving of various data with other computers (e.g., computer 300, computer 500, etc.) via network 2.

The external device 700 may be any device capable of communicating with the computer 300, the computer 500, or the server 600. The external device 700 may be, for example, a device capable of communicating with the computer 300 via the network 2, or a device capable of communicating with the computer 300 via short-range wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the computer 500 via the network 2, or a device capable of communicating with the computer 500 via short-range wireless communication or a wired connection. The external device 700 may be, for example, a device capable of communicating with the server 600 via the network 2. Examples of the external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripheral devices of the computers 300 and 500.

<Control of Information Processing System>
In the MR system 400 of this embodiment, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which a virtual object is arranged in a real space, similar to a known MR device. In the VR system 200 of this embodiment, the VR device 210 provides the VR user with a view of a virtual space, similar to a known VR device. In the MR system 400 of this embodiment, the MR user 6 is capable of operating a virtual object in the mixed reality space. In the VR system 200 of this embodiment, the VR user is capable of operating a virtual object in the virtual space. In addition, in the information processing system 100 of this embodiment, the movement of the VR user using the VR system 200 is reflected in a virtual object included in the view of the mixed reality space provided by the MR system 400. In the information processing system 100 of this embodiment, the movement of the MR user 6 using the MR system 400 is reflected in a virtual object included in the view of the virtual space provided by the VR system 200. In this embodiment, the motion of the VR user and the like are reflected in the virtual objects in the virtual space and the mixed reality space in real time. In addition, in this embodiment, the motion of the MR user 6 and the like are reflected in the virtual objects in the virtual space and the mixed reality space in real time.

In the following, the configuration according to the present embodiment is applied to a service (in other words, an application) that enables communication between an MR user 6 and a VR user. Specifically, the configuration according to the present embodiment is applied to a service that enables an MR user 6 in a conference room as a predetermined real space and a VR user in a location away from the conference room to hold a conference while simultaneously viewing an object such as a product mockup as a virtual object. The configuration according to the present embodiment can also be applied to a system that enables a student to take school classes from home. The configuration according to the present embodiment can also be applied to a system that enables a student to participate in a home party held at a specific house from another location. The application of the configuration according to the present embodiment is not limited to these services. In the following, the description will be given assuming that a plurality of detection devices 460 are installed in advance at predetermined positions in the conference room as shown in FIG. 8. In the following, the description will be given assuming that three MR users 6, MR user 6A, MR user 6B, and MR user 6C, exist in the conference room. In the following, the avatar object of the VR user is called a VR avatar 25. Additionally, the avatar object of MR user 6 is called MR avatar 26. Specifically, the avatar object of MR user 6A is called MR avatar 26A, the avatar object of MR user 6B is called MR avatar 26B, and the avatar object of MR user 6C is called MR avatar 26C.

With reference to FIG. 8, the field of view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the field of view provided to the VR user who enters the virtual space 11 simulating the conference room 21 through the VR device 210 will be described. FIG. 8 is a schematic diagram showing the state of the real conference room 21 and the state of the virtual space 11 when the MR users 6A, 6B, and 6C and the VR user are using the service. The upper part of FIG. 8 shows the state of the real conference room 21, and the lower part of FIG. 8 shows the state of the virtual space 11. Note that in FIG. 8, the VR avatar 25 is wearing the VR device 210, but this is for ease of explanation, and in reality, the VR avatar 25 does not need to wear the VR device 210 (virtual VR device 210). Also, in FIG. 8, the MR avatar 26 is not wearing an MR device 410 (virtual MR device 410), but the MR avatar 26 may be wearing an MR device 410.

As shown in FIG. 8, a desk 8 and the like are arranged in the conference room 21 as real objects. Three detection devices 460 are also arranged in the conference room 21. Three MR users 6A, 6B, and 6C are also present in the conference room 21. On the other hand, no VR users are present in the conference room 21. Here, a VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as a "virtual model 30") are displayed as virtual objects on the display 411 of the MR device 410 used by each of the MR users 6A, 6B, and 6C. That is, the MR device 410 displays the VR avatar 25 and the virtual model 30 in the real conference room 21 visible through the transparent display 411. Therefore, the MR users 6A, 6B, and 6C are provided with a view as if the VR avatar 25 and the virtual model 30, which do not actually exist in the conference room 21, exist in the conference room 21.

The virtual space 11 that the VR user sees through the VR device 210 is modeled after the conference room 21, and includes a virtual object 31 of a desk 8 that exists in the real conference room 21. The virtual space 11 also contains MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, a virtual model 30, and the like. This provides the VR user with a view as if they were in the conference room 21 where the MR users 6A, 6B, 6C are located.

In this way, in the information processing system 100 of this embodiment, the MR user 6 can see other MR users 6 and real objects that are actually present in the same place through the transparent display 411. In addition, from the viewpoint of the MR user 6, the VR avatar 25 of the VR user that appears as a virtual object can be seen. Therefore, the MR user 6 can communicate with the VR user and other MR users 6 without losing the sense of being in the real world. In addition, the VR user is in a different place from the MR user 6 in the real world, and in such a case, the shape of the room where the VR user is and the place where the MR user 6 is often different. Therefore, if the VR user also uses the MR device 410 or the like to display the MR avatar 26 in the real space, there is a risk that the display will be unnatural. In this embodiment, the VR user can enter the virtual space 11 and communicate with the MR user 6, so that such an unnatural feeling can be prevented. In addition, because the virtual space 11 provided to the VR user mimics the location of the MR user 6, when the VR user or MR user 6 moves around or moves a virtual object in the virtual space 11 or real space, respectively, the movements of the avatar or virtual model 30 displayed to the other party become natural. In addition, when the VR user or MR user 6 operates a virtual object such as the virtual model 30, it becomes easy for them to imagine how it will look to the other party, enabling smooth communication.

The functional configuration of the information processing system 100 will be described below with reference to FIG. 9. Note that the functional configuration shown in FIG. 9 is merely an example. Each of the VR system 200, the computer 500, and the server 600 may have at least some of the functions of the other devices. In other words, some or all of the functional blocks of the computer 300, the computer 500, and the server 600 in this embodiment may be provided by the computer 300, the computer 500, the server 600, or other devices. In addition, each device such as the computer 300, the computer 500, and the server 600 does not have to be realized by an integrated device, and may be realized by, for example, multiple devices connected via a network, etc.

In the present embodiment, the processor 301, the processor 501, or the processor 601 executes a program stored in the information processing system 100 to perform each process described below. However, at least a part of the process described below and performed by the processor 301 may be executed by a processor other than the processor 301. At least a part of the process described below and performed by the processor 501 may be executed by a processor other than the processor 501. At least a part of the process described below and performed by the processor 601 may be executed by a processor other than the processor 601. In other words, the computer that executes the program in this embodiment may be any computer including the computer 300, the computer 500, and the server 600, or may be realized by a combination of multiple devices.

FIG. 9 is a block diagram showing the functional configuration of the information processing system 100. As shown in FIG. 9, the computer 300 (in other words, the VR system 200) functions as a control unit 310 and a storage unit 311 through cooperation between the processor 301, memory 302, storage 303, input/output interface 304, and communication interface 305. The computer 500 (in other words, the MR system 400) functions as a control unit 510 and a storage unit 511 through cooperation between the processor 501, memory 502, storage 503, input/output interface 504, and communication interface 505. The server 600 functions as a control unit 610 and a storage unit 611 through cooperation between the processor 601, memory 602, storage 603, input/output interface 604, and communication interface 605.

The control unit 510 of the MR system 400 includes a virtual space generation unit 810, a coordinate definition unit 812, an MR side input reception unit 816, a user information acquisition unit 818, an MR side object control unit 820, a display control unit 840, a sound control unit 845, and a communication control unit 850.

The virtual space generating unit 810 generates a virtual space (in other words, virtual space data representing a virtual space) based on a predetermined real space. In other words, the virtual space generating unit 810 generates a virtual space that imitates a predetermined real space. Here, the predetermined real space is a predetermined place that exists in the real world, and may be, for example, a specific room such as a conference room or a school classroom. The predetermined real space may also be, for example, a school or a specific building such as a house. The predetermined real space may also be, for example, a specific town. In other words, the predetermined real space does not have to be a space separated by walls or the like. Furthermore, simulating a predetermined real space means that the generated virtual space has a structure similar to the basic structure of the real space. Specifically, for example, a virtual space that imitates a real-world conference room can be one in which the shape of the virtual room as a virtual space (for example, the shape of the walls, floor, etc.) is approximately the same as the shape of the conference room in the real world, and virtual objects corresponding to real objects such as desks and chairs that exist in the conference room in the real world are arranged. In this case, virtual objects such as desks, chairs, walls, and floors do not need to be exact copies of the shapes of real objects, and may have simplified shapes, patterns, or colors, for example. Also, real objects such as posters on the wall or small items placed on a desk or floor may be objects that do not appear in the virtual world (real objects that do not have corresponding virtual objects).

In this embodiment, as described above, the specified real space is a specific conference room 21, and a virtual space 11 that mimics this specific conference room 21 is generated.

A known method can be used to generate a virtual space that imitates a real space, and is not particularly limited, but may be, for example, as follows. That is, the virtual space generation unit 810 generates the virtual space 11 based on information from a sensor that can detect the shape and position of real objects such as walls, floors, desks, and chairs that constitute the real space. Specifically, the virtual space generation unit 810 may obtain three-dimensional information of the real space based on information from an image sensor (e.g., an image sensor that obtains RGB images, an image sensor that obtains black and white images, or a depth sensor) provided in the detection device 460 or the MR device 410, and generate the virtual space. In addition, the virtual space generation unit 810 may use information from, for example, an acceleration sensor, an angular velocity sensor, or a geomagnetic sensor to generate the virtual space 11.

The virtual space generation unit 810 stores data representing the generated virtual space 11 (hereinafter referred to as "virtual space data"), for example, in the storage unit 611, which serves as a virtual space data storage unit.

The virtual space generation unit 810 may generate virtual space data representing the virtual space 11 in advance before the VR user or MR user 6 starts using the service. Also, the virtual space generation unit 810 may generate virtual space data representing the virtual space 11 in real time, for example, while the VR user or MR user 6 is using the service.

The coordinate definition unit 812 defines the coordinates of the real space (in other words, the coordinates of the mixed reality space 21, hereinafter referred to as "mixed reality coordinates"). The coordinate definition unit 812 also defines the coordinates of the virtual space 11 (hereinafter referred to as "virtual space coordinates").

The mixed reality coordinates have three mutually orthogonal axes, for example, the x-axis, y-axis, and z-axis. The virtual space coordinates have three mutually orthogonal axes, for example, the x-axis, y-axis, and z-axis. The coordinate definition unit 812 defines (in other words, associates) each coordinate for each point of the virtual space 11 (in other words, the virtual conference room) that corresponds to (in other words, indicates the same point) each point of the real space (in other words, the real conference room 21) so that the mixed reality coordinates indicating each point of the real space and the virtual space coordinates indicating each point of the virtual space correspond one-to-one. For example, when the virtual space generation unit 810 generates a virtual space that imitates the real space, the coordinate definition unit 812 defines the virtual space coordinates associated with the mixed reality coordinates. At this time, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates based on, for example, a feature point of the conference room detected by the detection device 460 or the MR device 410. Furthermore, the coordinate definition unit 812 may associate the mixed reality coordinates with the virtual space coordinates, for example, based on the position of the detection device 460 or the position of a marker or the like that has been installed in advance in the conference room.

In addition, the coordinate definition unit 812 stores information regarding the correspondence between mixed reality coordinates and virtual space coordinates (in other words, the correspondence between each point in the real space and each point in the virtual space) in the memory unit 611, which functions as a positional relationship memory unit.

The method for defining mixed reality coordinates and virtual space coordinates can be any known method, and is not particularly limited.

The timing for defining the mixed reality coordinates and the virtual space coordinates (in other words, the timing for associating the mixed reality coordinates with the virtual space coordinates) may be, for example, when the virtual space generating unit 810 generates a virtual space that imitates the real space, or when the MR device 410 starts displaying a virtual object.

The generation of the virtual space and the definition of each coordinate may be performed in advance, for example, by a dedicated device serving as an external device 700 capable of scanning the real space with high accuracy, and the generated data may be stored in the storage unit 611.

The MR side input receiving unit 816 receives input from the MR user 6. In other words, the MR side input receiving unit 816 acquires input information from the MR user 6. Specifically, the MR side input receiving unit 816 receives the output of the detection device 460, the controller 480, the camera 413, the device worn by the MR user, the gaze sensor 412, the sensor 417, the sensor 486, and the image sensor provided in the detection device 460 and the MR device 410 as input from the MR user 6. In other words, the MR side input receiving unit 816 receives the output of the detection means (e.g., the detection device 460, the controller 480, the camera 413, the gaze sensor 412, the sensor 417, the sensor 486, and the image sensor provided in the detection device 460 and the MR device 410) that detects the movement of the MR user 6 as input from the MR user 6. As described below, these devices and sensors are intended to obtain information about the movements of the MR user 6, which is used to control virtual objects in the virtual space 11 and the mixed reality space 21, and the MR user 6 can input information to the MR system 400 to move the virtual objects, etc., by his or her own movements.

Specifically, the MR side input receiving unit 816 may receive, for example, an output from the camera or camera 413 serving as the detection device 460 as an input from the MR user 6. More specifically, the MR side input receiving unit 816 may receive, for example, position information indicating the position of the MR user 6 in the mixed reality space 21, detected by image recognition from an image captured by an image sensor provided in each of these cameras, as an input from the MR user 6. In other words, the MR side input receiving unit 816 may receive, for example, an output from a position detection means that detects the position of the MR user 6 in the mixed reality space 21 as an input from the MR user 6. The position detection means may detect the position of the MR user 6 and obtain the position information based on, for example, wireless communication (e.g., Wi-Fi communication, Bluetooth communication, or UWB communication) between the MR device 410 and a specific device of the MR system 400 (e.g., the detection device 460 or a beacon (not shown)), or positioning using a GPS (Global Positioning System) sensor provided in the MR device 410. The position information indicating the position of the MR user 6 can also be considered information regarding the movement of the MR user 6.

The MR side input receiving unit 816 may also receive, as input from the MR user 6, information regarding the movement of the MR user 6 detected by image recognition from an image captured by a camera serving as the detection device 460 or an image sensor provided in the camera 413. For example, the MR side input receiving unit 816 may receive, as input from the MR user 6, information regarding the hand movement of the MR user 6 detected by image recognition, or information regarding the inclination or facing direction of the MR user 6.

The MR side input receiving unit 816 may also receive, as input from the MR user 6, information regarding the movements of the MR user 6 obtained using, for example, a device worn by the MR user 6 (for example, a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device) capable of tracking the movements of the MR user 6 (and a device that tracks the movements of the MR user 6 in cooperation with the device via short-range wireless communication, etc.).

The MR side input receiving unit 816 may also receive, for example, information indicating the operation of the MR user 6 detected by the controller 480 (in other words, information regarding the movement of the MR user 6) as input from the MR user 6.

The MR side input receiving unit 816 may also receive output data (in other words, information regarding the movement of the MR user 6) regarding the inclination or direction of the MR device 410 from a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 460 or the MR device 410 as input from the MR user 6.

The user information acquisition unit 818 acquires information about the MR user 6 present in the mixed reality space 21. Specifically, for example, the user information acquisition unit 818 acquires identification information that enables each MR user 6A, 6B, and 6C in the mixed reality space 21 to be identified (in other words, identification information that enables each MR device 410 to be identified). For example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information about the user using the MR device 410 from the MR device 410 (for example, information about an account logged into the MR device 410, etc.) as the identification information. Also, for example, the user information acquisition unit 818 may communicate with the MR device 410 worn by each user and acquire information specific to each MR device 410 that enables each MR device 410 to be identified from the MR device 410 as the identification information. Also, for example, the user information acquisition unit 818 may acquire the identification information by identifying the MR user 6 in the mixed reality space 21 through image recognition from an image captured by an image sensor provided in the detection device 460 or the MR device 410. The user information acquisition unit 818 also associates the information acquired by the MR side input acceptance unit 816 with the identification information. In other words, the user information acquisition unit 818 performs control so that it is possible to determine which MR user 6 the information belongs to when information about the movement of the MR user 6 is used in controlling a virtual object, which will be described later.

The MR side object control unit 820 controls virtual objects in the mixed reality space 21.

The MR-side object control unit 820 places virtual objects in the mixed reality space 21. Specifically, the MR-side object control unit 820 places, for example, the VR avatar 25 of the VR user, and objects that are given predetermined changes by at least one of the VR user and the MR user 6. In this embodiment, as shown in FIG. 8, the MR-side object control unit 820 places the VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as the "virtual model 30") that can be moved by the VR user and the MR user 6 in the mixed reality space 21.

The MR-side object control unit 820 places a virtual object corresponding to the virtual object at a position in the mixed reality space 21 corresponding to the position where the virtual object in the virtual space 11 is placed, for example. Here, the virtual object corresponding to the virtual object, i.e., the virtual object in the mixed reality space 21 corresponding to the virtual object in the virtual space 11, may have the same form as the virtual object in the virtual space 11 (in other words, it represents the same object), or may have a different form. In this embodiment, a virtual model 30 having the same form as the virtual model 30 in the virtual space 11 is placed in the mixed reality space 21. Also, a VR avatar 25 having the same form as the VR avatar 25 in the virtual space 11 is placed in the mixed reality space 21.

The placement of the VR avatar 25 in the mixed reality space 21 can also be described as follows. That is, the MR-side object control unit 820 can also be said to place the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11. Here, the position of the VR user in the virtual space 11 is, for example, the position of the VR avatar 25 in the virtual space 11. Note that in content in which the VR user enters the virtual space 11 from a first-person perspective, there may be cases in which the VR user (the VR avatar 25) cannot be seen (specifically, only a part of the user's body, such as the hands, or even a part of the user cannot be seen), but even in such cases, the VR user in the virtual space 11 recognized by the computer 300 can be said to be the VR avatar 25, and the position of the VR user in the virtual space 11 recognized by the computer 300 can be said to be the position of the VR avatar 25 in the virtual space.

The appearance (in other words, the form) of the VR avatar 25 may imitate the appearance of the VR user, but it does not have to imitate the appearance. For example, the appearance of the VR avatar 25 may imitate the appearance of a specific animal. The VR avatar 25 placed in the virtual space 11 and the VR avatar 25 placed in the mixed reality space 21 may have different appearances. With this configuration, the VR user can display his/her avatar in the mixed reality space 21 while hiding the form of his/her avatar in the virtual space 11. The appearance of the VR avatar 25 placed in the virtual space 11 may be an appearance that is visible to the VR user operating the VR avatar 25, or an appearance that is visible to other VR users operating other VR avatars 25 in the virtual space 11 in which the VR avatar 25 exists.

The MR-side object control unit 820 may also place a virtual object in the mixed reality space 21, for example, as follows:

That is, for example, when placing the VR avatar 25, the MR-side object control unit 820 determines where in the mixed reality space 21 the VR avatar 25 should be placed based on position information indicating the position of the VR user (in other words, the VR avatar 25) in the virtual space 11. Specifically, for example, when the VR-side object control unit 920 described later places the VR avatar 25 in the virtual space 11, it transmits information regarding the position of the VR avatar 25 in the virtual space 11 (for example, the coordinate value of the virtual space coordinates) to the MR-side object control unit 820 via the server 600. Then, based on the information, the MR-side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11 (for example, the position of the coordinate value of the mixed reality coordinates corresponding to the coordinate value of the virtual space coordinates). As a result, the VR avatar 25 is placed at the same position in the conference room in the virtual space 11 and the mixed reality space 21.

In other words, when placing a virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the mixed reality space 21, the MR side object control unit 820 may determine the position in the mixed reality space 21 to place the virtual object based on position information (e.g., position information transmitted by the VR side object control unit 920) indicating the position of the virtual object (e.g., a VR avatar 25 or a virtual model 30, etc.) in the virtual space corresponding to the virtual object.

For example, when placing a virtual object in the mixed reality space 21, the MR-side object control unit 820 may determine a position in the mixed reality space 21 to place the virtual object based on information detected by a predetermined sensor of the MR system 400. Specifically, for example, a predetermined marker may be set in advance in the real space, and the MR-side object control unit 820 may place the virtual model 30 as a virtual object at the position of the marker photographed by an image sensor provided in the detection device 460 or the MR device 410. The MR-side object control unit 820 may place the virtual model 30 as a virtual object at a position indicated by the MR user 6. The instruction may be made by a predetermined gesture (for example, a gesture of pointing at the point where the virtual model 30 is to be placed), or may be to place the marker. The MR-side object control unit 820 may detect a flat surface in the mixed reality space 21 from an image photographed by an image sensor provided in the detection device 460 or the MR device 410, and place the virtual model 30 on the flat surface.

When arranging a virtual object in the mixed reality space 21, the MR-side object control unit 820 receives object data indicating the form of the virtual object, and arranges the virtual object based on the object data. Here, the MR-side object control unit 820 may receive the object data, for example, from the control unit 610 of the server 600, or from the VR-side object control unit 920 of the VR system 200. Specifically, when arranging the VR avatar 25 in the mixed reality space 21, the MR-side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the control unit 610, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21. When arranging the VR avatar 25 in the mixed reality space 21, the MR-side object control unit 820 may receive object data indicating the form of the VR avatar 25 from the VR-side object control unit 920, and arrange the VR avatar 25 in the form indicated by the object data in the mixed reality space 21.

The display control unit 840 controls the image display on the display 411 of the MR device 410. The display control unit 840 generates an image for displaying a virtual object in the mixed reality space 21 placed by the MR side object control unit 820 at the position placed by the MR side object control unit 820. The display control unit 840 also causes the image to be displayed on the display 411. This provides the MR user 6 with a view of the mixed reality space in which the virtual object is placed at the desired position in the real space.

The display control unit 840 also controls the image display on the display 470. For example, the display control unit 840 causes the display 470 to display an image similar to the image displayed on the display 411. Note that the display control unit 840 may also cause the display 470 to display an image showing a view of the mixed reality space provided to the MR user 6 by the MR device 410.

When the microphone 415 of the MR device 410 detects speech from the MR user 6, the sound control unit 845 acquires sound data corresponding to the speech. The sound control unit 845 also transmits the acquired sound data to the computer 300, etc. via the network 2. When the sound control unit 845 receives sound data from the computer 300 via the network 2, it outputs sound (speech) corresponding to the sound data from the speaker 416. This allows the MR user 6 to communicate with the VR user, for example, by phone. The MR side input receiving unit 816 may also receive sound data related to the speech of the MR user 6 detected by the microphone 415 as input from the MR user 6.

The communication control unit 850 can communicate with the server 600, the computer 300, and other information communication devices via the network 2. For example, the communication control unit 850 transmits information used by the server 600 or the computer 300 to the server 600 or the computer 300. The communication control unit 850 also receives information used by the computer 500 from the server 600 or the computer 300, for example.

The control unit 310 of the VR system 200 includes a VR side input receiving unit 916, a VR side object control unit 920, a virtual camera control unit 930, a display control unit 940, a sound control unit 945, and a communication control unit 950.

The VR side input receiving unit 916 receives input from the VR user. In other words, the VR side input receiving unit 916 acquires input information from the VR user. Specifically, the VR side input receiving unit 916 receives outputs from the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and image sensors provided in the detection device 260 and the VR device 210 as input from the VR user. In other words, the VR-side input receiving unit 916 receives the output of detection means for detecting the movements of the VR user (e.g., the detection device 260, the controller 280, the first camera 213, the second camera 214, a wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and the image sensors provided in the detection device 260 and the VR device 210) as input from the VR user. As described below, these devices and sensors are for obtaining information about the movements of the VR user that is used to control virtual objects in the virtual space 11 and the mixed reality space 21, and the VR user can input information for moving the virtual objects, etc., to the VR system 200 through his or her own movements.

Specifically, the VR-side input receiving unit 916 may receive, for example, an output from a camera serving as the detection device 260 or a camera provided in the VR device 210 that captures the surroundings of the VR device 210 as an input from the VR user. More specifically, the VR-side input receiving unit 916 may receive, as an input from the VR user, position information indicating the position of the VR user in real space, detected by image recognition from an image captured by an image sensor provided in each of these cameras. In other words, the VR-side input receiving unit 916 may receive, as an input from the VR user, an output from a position detection means that detects the position of the VR user in real space. The position detection means may detect the position of the VR user and acquire the position information based on, for example, positioning based on wireless communication (e.g., Wi-Fi communication, Bluetooth communication, or UWB communication) between the VR device 210 and a specific device of the VR system 200 (e.g., the detection device 260 or a beacon (not shown)), or positioning using a GPS (Global Positioning System) sensor provided in the VR device 210. The position information indicating the position of the VR user can also be considered information regarding the movement of the VR user.

The VR-side input receiving unit 916 may also receive, as input from the VR user, information about the VR user's movements detected by image recognition from images captured by an image sensor provided in a camera serving as the detection device 260 or a camera provided in the VR device 210 that captures the surroundings of the VR device 210. For example, the VR-side input receiving unit 916 may receive, as input from the VR user, information about the VR user's hand movements detected by image recognition, or information about the VR user's inclination or facing direction.

The VR side input receiving unit 916 may also receive, as input from a VR user, information regarding the user's movements obtained using, for example, a device worn by the VR user (e.g., a wearable device such as a watch-type, wristband-type, ring-type, or clothing-type device, or an implantable device) capable of tracking the movements of the VR user (and a device that tracks the movements of the MR user 6 in cooperation with the device via short-range wireless communication, etc.).

The VR-side input receiving unit 916 may also receive, for example, information indicating the VR user's operation detected by the controller 280 (in other words, information regarding the VR user's movements) as input from the VR user.

The VR-side input receiving unit 916 may also receive output data regarding the inclination or direction of the VR device 210 (in other words, information regarding the movement of the VR user) from a sensor (e.g., an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) provided in the detection device 260 or the VR device 210 as input from the VR user.

The control unit 310 controls the virtual space 11. The control unit 310 acquires virtual space data indicating the virtual space 11 stored in the storage unit 611, and defines the virtual space 11 to be provided to the VR user based on the virtual space data.

The VR side object control unit 920 controls virtual objects in the virtual space 11.

The VR-side object control unit 920 places a virtual object in the virtual space 11 indicated by the virtual space data. The object data indicating the virtual object is stored, for example, in the storage unit 611 or the storage unit 311 as an object data storage unit. Then, the VR-side object control unit 920 places the virtual object in the virtual space 11 using the object data.

Specifically, the VR-side object control unit 920 places, for example, in a virtual conference room, MR avatars 26A, 26B, 26C of MR users 6A, 6B, 6C, virtual objects to which predetermined changes are applied by at least one of the VR user and MR user 6, etc. In this embodiment, the VR-side object control unit 920 places, in the virtual space 11, the VR avatar 25, the MR avatar 26, and a mock-up virtual object 30 (hereinafter also referred to as the "virtual model 30") that can be moved by the VR user and the MR user 6. Note that a virtual object 31 corresponding to a desk 8 as a real object, etc., is also placed in the virtual space 11.

For example, when a VR system 200 (in other words, computer 300) used by a VR user is connected to a VR system 200 (in other words, computer 300) used by another VR user via network 2, the VR side object control unit 920 may place a VR avatar 25 of the other VR user in the virtual space 11. The VR avatar 25 of the other VR user can be operated by the other VR user.

The VR-side object control unit 920 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21. In other words, the VR-side object control unit 920 places the MR avatars 26A, 26B, and 26C so that the positions of the MR users 6A, 6B, and 6C in the real conference room 21 are the same as the positions of the MR avatars 26A, 26B, and 26C in the virtual conference room.

The appearance (in other words, the form) of the MR avatar 26 may imitate the appearance of the MR user 6, but it does not have to imitate it. For example, the appearance of the MR avatar 26 may imitate the appearance of a specific animal. When placing a virtual object in the mixed reality space 21, the VR-side object control unit 920 receives object data indicating the form of the virtual object and places the virtual object based on the object data. Here, the VR-side object control unit 920 may receive the object data from the control unit 610 of the server 600, or may receive it from the MR-side object control unit 820 of the MR system 400. Specifically, for example, when placing the MR avatar 26 in the virtual space 11, the VR-side object control unit 920 may receive object data indicating the form of the MR avatar 26 from the control unit 610 or the MR-side object control unit 820, and place the MR avatar 26 in the form indicated by the object data in the virtual space 11. The appearance of the MR avatar 26 may be preset by the MR user 6, and when the MR avatar 26 is displayed on the VR device 210, it may be displayed with the appearance set by the MR user 6. In addition, when the appearance of the MR avatar 26 imitates the appearance of the MR user 6, the VR-side object control unit 920 may place the MR avatar 26 in the virtual space 11 (e.g., generated by the control unit 510) based on an image of the MR user 6 acquired by a specific camera such as the camera 413 or the camera serving as the detection device 460.

The virtual camera control unit 930 places the virtual camera 14 in the virtual space 11. The virtual camera control unit 930 also controls the position of the virtual camera 14 in the virtual space 11 and the inclination (direction) of the virtual camera 14. In this embodiment, the virtual camera control unit 930 places the virtual camera 14 at the eye position of the VR avatar 25 in the virtual space 11. In other words, the virtual camera control unit 930 links the position of the virtual camera 14 to the position of the VR avatar 25, and moves the virtual camera 14 in the virtual space 11 when the VR avatar 25 moves in the virtual space 11. The position of the virtual camera 14 does not have to be linked to the position of the VR avatar 25. For example, it may be possible to move the VR avatar 25 in the virtual space 11 while keeping the virtual camera 14 fixed in a predetermined position.

The display control unit 940 controls the image display on the display 211 of the VR device 210. The display control unit 940 defines the field of view 15 according to the position and tilt of the virtual camera 14 (in other words, the tilt of the head of the VR user wearing the VR device 210). The display control unit 940 also generates a field of view image 17 to be displayed on the display 211 based on the defined field of view area 15. The field of view image 17 generated by the display control unit 940 is output to the VR device 210.

The display control unit 940 also controls the image display on the display 270. For example, the display control unit 940 causes the display 270 to display an image similar to the image displayed on the display 211.

When the sound control unit 945 detects speech from the VR user using the microphone 215 from the VR device 210, it acquires sound data corresponding to the speech. The sound control unit 945 also transmits the acquired sound data to the computer 500 or other VR users' computers 300 via the network 2. When the sound control unit 945 receives sound data from the computer 500 or other users' computers 300 via the network 2, it outputs sound (speech) corresponding to the sound data from the speaker 216. This allows the VR user to communicate with the MR user 6, for example, by phone call. The VR side input receiving unit 916 may also receive sound data related to the VR user's speech detected by the microphone 215 as input from the VR user.

The communication control unit 950 can communicate with the server 600, the computer 500, other VR users' computers 300, and other information communication devices via the network 2. The communication control unit 950 transmits, for example, information used by the server 600, the computer 500, or other VR users' computers 300 to the server 600, the computer 500, or other VR users' computers 300. The communication control unit 850 also receives, for example, information used by the computer 300 from the server 600, the computer 500, or other VR users' computers 300.

(Integration of mixed reality and virtual space)
In this embodiment, the virtual objects displayed by the MR device 410 and the virtual objects displayed by the VR device 210 are controlled based on input from the MR user 6 acquired by the MR side input receiving unit 816.

Specifically, the MR-side object control unit 820 applies a predetermined change to a virtual object in the mixed reality space 21, specifically the virtual model 30, based on input from the MR user 6 received by the MR-side input receiving unit 816. Here, the predetermined change may be a change to move a virtual object in the mixed reality space 21, such as moving the virtual object, changing the inclination or direction of the virtual object, or changing the relative positional relationship between parts of the virtual object. In addition, the predetermined change may be a change to the color of the virtual object, a change to information displayed by the virtual object, or a change to the form of the virtual object.

More specifically, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on, for example, information on the movement of the MR user 6 received by the MR-side input receiving unit 816. Specifically, for example, when a gesture of the MR user 6 carrying the virtual model 30 is detected by a detection means (for example, the detection device 460 or the camera 413, etc.) and the detection means outputs information related to the gesture, the MR-side input receiving unit 816 receives the output as an input from the MR user 6. Also, based on the input received by the MR-side input receiving unit 816, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 according to the instruction from the MR user 6 by the gesture. Note that the gesture of the MR user 6 that causes a predetermined change to the virtual model 30 as a virtual object does not have to be one that touches the virtual model 30 in the mixed reality space 21. For example, it may be possible to rotate the virtual object by holding a hand toward the virtual model 30 and waving the hand sideways.

The MR-side object control unit 820 may also make a predetermined change to the virtual object based on, for example, information about the operation of the MR user 6 on the controller 480 (in other words, information about the movement of the MR user 6) received by the MR-side input receiving unit 816. The MR-side object control unit 820 may also make a predetermined change to the virtual object based on, for example, sound data about the speech of the MR user 6 received by the MR-side input receiving unit 816.

The VR-side object control unit 920 also applies predetermined changes to virtual objects in the virtual space 11 based on input from the MR user 6 received by the MR-side input receiving unit 816.

Specifically, the VR-side object control unit 920 applies a change to the virtual model 30 in the virtual space 11 similar to the change applied to the virtual model 30 in the mixed reality space 21 based on, for example, input from the MR user 6 received by the MR-side input receiving unit 816. That is, for example, as described above, when the virtual model 30 in the mixed reality space 21 moves due to a gesture of the MR user 6 to carry the virtual model 30, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 in the same manner.

The method of imparting a change to the virtual model 30 in the virtual space similar to the change imparted to the virtual model 30 in the mixed reality space 21 is not particularly limited, but may be, for example, as follows. That is, for example, when the MR-side object control unit 820 moves the virtual model 30 (in other words, the virtual object) in the mixed reality space 21 based on an input from the MR user 6 (in other words, at a predetermined opportunity), it may send information indicating the position and tilt of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the VR-side object control unit 920. Then, the VR-side object control unit 920 may place the virtual model 30 at the position indicated by the information in the virtual space 11, or place the virtual model 30 at the tilt indicated by the information. Also, for example, the communication control unit 850 may send information based on the input from the MR user 6 (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 510 based on the input) to the communication control unit 950, and the VR-side object control unit 920 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the VR-side object control unit 920 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 accepted by the MR-side input acceptance unit 816, but "based on the input from the MR user 6" is not limited to the VR-side object control unit 920 itself receiving the input and controlling the virtual object, and may be any virtual object in the virtual space 11 that ultimately makes a predetermined change in response to the input.

The VR-side object control unit 920 also moves the MR avatar 26 in the virtual space 11 based on, for example, input from the MR user 6 received by the MR-side input receiving unit 816. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information on the movement of the MR user 6 received by the MR-side input receiving unit 816. For example, the VR-side object control unit 920 moves the MR avatar 26 based on position information indicating the position of the MR user 6 in the mixed reality space 21, which the MR-side input receiving unit 816 receives as input from the MR user 6. That is, the VR-side object control unit 920 places the MR avatar 26 at a position in the virtual space 11 corresponding to the position of the MR user 6 in the mixed reality space 21 based on the position information, and when the MR user 6 moves in the mixed reality space 21, places the MR avatar 26 at a position in the virtual space 11 corresponding to the position after the movement.

The VR-side object control unit 920 also moves the MR avatar 26 based on, for example, information on the movements of each part of the MR user 6's body (e.g., hand movements, head movements, eye movements, changes in facial expressions, etc.) and information on the inclination and direction of the MR user 6, which are received by the MR-side input receiving unit 816 as input from the MR user 6. That is, the VR-side object control unit 920 reflects the movements of each part of the MR user 6's body (in other words, posture) and the inclination and direction of the MR user 6 in the MR avatar 26 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the MR avatar 26 so that the movement of the MR avatar 26 in the virtual space 11 imitates the movement of the MR user 6 detected by a detection means for detecting the movement of the MR user 6. In other words, the VR-side object control unit 920 controls the movement of the MR avatar 26 so that the movement of at least part of the body (e.g., hands, feet, head, eyes, mouth, etc.) of the MR avatar 26 is linked to the movement of the MR user 6.

In addition, at least a part of the movement of the MR avatar 26 may be realized by the MR user 6 operating the controller 480 (e.g., operating an analog stick, a button, etc.). That is, for example, the VR-side object control unit 920 may move a part of the body of the MR avatar 26 based on information indicating the MR user 6's operation of the controller 480 as input from the MR user 6.

Note that at least one of the position and movement of the MR avatar 26 in the virtual space 11 does not have to be linked to the position or movement of the MR user 6 in the mixed reality space 21. That is, for example, even if the MR user 6 moves in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move in response to this. Also, for example, even if the MR user 6 moves his/her hand in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not make the same movement.

In addition, in this embodiment, the virtual objects displayed by the VR device 210 and the virtual objects displayed by the MR device 410 are controlled based on input from the VR user acquired by the VR side input receiving unit 916.

Specifically, the VR-side object control unit 920 applies a predetermined change to a virtual object in the virtual space 11, specifically the virtual model 30, based on input from the VR user received by the VR-side input receiving unit 916.

More specifically, the VR-side object control unit 920 makes a predetermined change to the virtual model 30 in the virtual space 11 based on, for example, information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movement) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 according to instructions from the VR user via operations on the controller 280.

The VR-side object control unit 920 may also make a predetermined change to a virtual object based on, for example, a gesture of the VR user detected by a detection means. The VR-side object control unit 920 may also make a predetermined change to a virtual object based on, for example, sound data related to the speech of the VR user received by the VR-side input receiving unit 916.

The MR-side object control unit 820 also applies predetermined changes to virtual objects in the mixed reality space 21 based on input from the VR user received by the VR-side input receiving unit 916.

Specifically, the MR-side object control unit 820 applies a change to the virtual model 30 in the mixed reality space 21 similar to the change applied to the virtual model 30 in the virtual space 11 based on, for example, input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the virtual model 30 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 in the same manner.

The method of imparting a change to the virtual model 30 in the mixed reality space 21 similar to the change imparted to the virtual model 30 in the virtual space 11 is not particularly limited, but may be, for example, as follows. That is, for example, when the VR-side object control unit 920 moves the virtual model 30 (in other words, the virtual object) in the virtual space 11 based on an input from the VR user (in other words, at a predetermined opportunity), it may send information indicating the position and tilt of the virtual model 30 after the movement (in other words, information regarding the state of the virtual model 30) to the MR-side object control unit 820. Then, the MR-side object control unit 820 may place the virtual model 30 at the position indicated by the information in the mixed reality space 21, or place the virtual model 30 at the tilt indicated by the information. Also, for example, the communication control unit 950 may send information based on the input from the VR user (for example, information on the movement amount, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 310 based on the input) to the communication control unit 850, and the MR-side object control unit 820 may move a virtual object such as the virtual model 30 based on the information. That is, in this embodiment, the MR-side object control unit 820 makes a predetermined change to a virtual object in the mixed reality space 21 based on the input from the VR user received by the VR-side input receiving unit 916, but "based on the input from the VR user" does not necessarily mean that the MR-side object control unit 820 itself receives the input and controls the virtual object, but may mean that the virtual object in the mixed reality space 21 ultimately makes a predetermined change in response to the input.

The VR-side object control unit 920 also makes a predetermined change to the VR avatar 25 in the virtual space 11 based on input from the VR user received by the VR-side input receiving unit 916. Specifically, the VR-side object control unit 920 makes a predetermined change to the VR avatar 25 in the virtual space 11 based on, for example, information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movements) received by the VR-side input receiving unit 916. Specifically, for example, the VR-side object control unit 920 moves the VR avatar 25 in the virtual space 11 in accordance with instructions from the VR user via operation on the controller 280.

The VR-side object control unit 920 also moves the VR avatar 25 based on, for example, information on the movements of each part of the VR user's body (e.g., hand movements, head movements, eye movements, changes in facial expressions, etc.) and information on the VR user's inclination and facing direction, which the VR-side input receiving unit 916 receives as input from the VR user. That is, the VR-side object control unit 920 reflects the movements of each part of the VR user's body (in other words, posture) and the VR user's inclination and facing direction in the VR avatar 25 in the virtual space 11 based on each of these pieces of information. That is, in this embodiment, the VR-side object control unit 920 controls the movement of the VR avatar 25 so that the movement of the VR avatar 25 in the virtual space 11 imitates the movement of the VR user detected by a detection means for detecting the movement of the VR user. In other words, the VR-side object control unit 920 controls the movement of the VR avatar 25 so that the movement of at least part of the body (e.g., hands, feet, head, eyes, mouth, etc.) of the VR avatar 25 is linked to the movement of the VR user.

The MR-side object control unit 820 also imparts to the VR avatar 25 in the mixed reality space 21 a change similar to the change imparted to the VR avatar 25 in the virtual space 11 based on, for example, input from the VR user received by the VR-side input receiving unit 916. That is, for example, as described above, when the VR avatar 25 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR-side object control unit 820 moves the VR avatar 25 in the mixed reality space 21 in the same manner. Note that the method of imparting to the VR avatar 25 in the mixed reality space 21 a change similar to the change imparted to the VR avatar 25 in the virtual space 11 is not particularly limited, but may be the same as in the case of the virtual model 30 described above, for example.

The position of the VR avatar 25 in the virtual space 11 and the position in the mixed reality space 21 may move in conjunction with the movement of the VR user in real space. That is, for example, the VR-side object control unit 920 may move the VR avatar 25 based on position information indicating the position of the VR user in real space as an input from the VR user.

Furthermore, the movement of the VR avatar 25 in the virtual space 11 and a part of the VR avatar 25 in the mixed reality space 21 may be realized by an operation on the controller 280. That is, for example, the VR side object control unit 920 and the MR side object control unit 820 may move a part of the body of the VR avatar 25 based on information indicating the VR user's operation on the controller 280 as an input from the VR user.

In this embodiment, the results of actions taken on real objects that actually exist in the conference room 21 as a specified real space can also be shared between the VR user and the MR user 6. In this regard, the following describes an example of control of a real object on which a specific piece of writing can be made (e.g., a blackboard or whiteboard). As shown in FIG. 8, a blackboard 35 is placed as a real object in the mixed reality space 21, and a virtual blackboard 36 is placed as a virtual object corresponding to the blackboard 35 in the virtual space 11 at a position corresponding to the blackboard 35.

In this embodiment, the action (in other words, the result of the action) taken by the MR user 6 on the blackboard 35 as a real object in the mixed reality space 21 is reflected on the blackboard 35 in the mixed reality space 21. Also, in this embodiment, the action is reflected on the virtual blackboard 36 in the virtual space 11.

In the mixed reality space 21, the MR user 6 can make a predetermined entry on the blackboard 35. The entry action on the blackboard 35 by the MR user 6 is detected by a detection means for detecting the entry action (hereinafter referred to as the "MR side entry detection means"), and the output from the MR side entry detection means (in other words, information on the entry action) is received by the MR side input reception unit 816 as an input from the MR user 6. The MR side entry detection means may be, for example, a detection means capable of detecting the movement of the hand of the MR user 6. Specifically, the MR side entry detection means may be, for example, a pen-type controller as the controller 480. The MR side entry detection means may be, for example, a camera as the detection device 460, the camera 413, or the like. The MR side entry detection means may be, for example, a device worn by the MR user 6. The MR side entry detection means may be a sensor provided on the blackboard 35 (for example, a touch sensor, a sensor that reads reflected infrared light from an infrared light emitting device provided on the blackboard 35, or the like). In other words, the blackboard 35 can communicate with the computer 500, and information regarding the writing action of the MR user 6 detected by a sensor provided on the blackboard 35 can be received by the MR side input receiving unit 816 as input from the MR user 6.

The writing action of the MR user 6 is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the MR user 6 performs a writing action, the blackboard 35 changes to a state in which the writing content of the MR user 6 is displayed (for example, a picture of a moon, a picture of a heart, and the word "Sample" in FIG. 10(a)) as shown in FIG. 10(a). For example, the blackboard 35 may be an electronic blackboard that electronically displays (in other words, reflects) the writing content of the MR user 6, and when the MR user 6 performs a writing action, it may change to a state in which the writing content is displayed. Specifically, for example, the writing surface of the blackboard 35 itself may be a display, and the display may display the writing content. The blackboard 35 may also be provided with a projector that projects an image onto the writing surface, and the projector may project the writing content onto the writing surface. In other words, the reflection of the user's behavior in the mixed reality space 21 (in other words, a change in the mixed reality space 21 seen through the MR device 410 due to the behavior of the MR user 6) may be realized by some change (for example, a change in the display content) occurring in the real object itself. When the written content is electronically displayed, the computer of the blackboard 35 may control the display based on the detection result of a sensor equipped in the blackboard, or the computer 500 may control the display based on the detection result of the MR side writing detection means. That is, for example, the control unit 510 of the computer 500 or the like may function as a control means (hereinafter referred to as "blackboard display control means") that controls the display of the blackboard 35 based on the input from the MR user 6 received by the MR side input receiving unit 816. Specifically, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 to be an image showing the written content based on the input from the MR user 6. The writing action of the MR user 6 on the blackboard 35 in the mixed reality space 21 may be reflected by the MR user 6 actually writing characters or pictures on the blackboard 35 with a writing implement such as chalk. In other words, the action of actually writing characters or pictures on the blackboard 35 with a writing implement may be detected by the MR-side writing detection means as the writing action of the MR user 6.

For example, the MR-side object control unit 820 may generate a virtual object (hereinafter, referred to as the "entry object 37") indicating the contents entered by the MR user 6 and place it at the position of the blackboard 35 in the mixed reality space 21 (see FIG. 10(a)). Specifically, the MR-side object control unit 820 may generate the entry object 37 based on the input from the MR user 6 received by the MR-side input receiving unit 816. When the blackboard 35 is viewed through the MR device 410, the state in which the blackboard 35 displays the entry contents (in other words, the entry object 37) may be visually recognized. In other words, the reflection of the user's actions in the mixed reality space 21 (in other words, the change in the mixed reality space 21 seen through the MR device 410 due to the actions of the MR user 6) may be realized by adding a predetermined virtual object to the real object.

In addition, the VR-side object control unit 920 applies a predetermined change to the virtual object corresponding to the real object based on the input from the MR user 6 received by the MR-side input receiving unit 816. Specifically, as shown in FIG. 10(b), the VR-side object control unit 920 controls the virtual blackboard 36 to reflect the input from the MR user 6, and to display the contents written by the MR user 6. In other words, the VR-side object control unit 920 reflects the change of the real object (e.g., the blackboard 35) in the mixed reality space 21 based on the action of the MR user 6 to the virtual object (e.g., the virtual blackboard 36) in the virtual space 11 corresponding to the real object. Specifically, for example, when information on the writing action of the MR user 6 is input to the MR-side input receiving unit 816, the VR-side object control unit 920 may place the writing object 38 indicating the contents written by the MR user 6 at the position of the virtual blackboard 36 in the virtual space 11 (see FIG. 10(b)). The VR-side object control unit 920 may also control the virtual blackboard 36 to display the written contents by, for example, changing the form of the virtual blackboard 36 itself. The virtual blackboard 36 may be a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, and the VR-side object control unit 920 may cause the image to be displayed on the virtual blackboard 36. The MR system 400 may also be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR-side object control unit 920 may cause the image to be displayed on the virtual blackboard 36.

In addition, in this embodiment, the action (in other words, the result of the action) taken by the VR user in the virtual space 11 on the virtual blackboard 36 serving as a virtual object is reflected on the virtual blackboard 36 in the virtual space 11. In addition, in this embodiment, the action is reflected on the blackboard 35 in the mixed reality space 21.

In the virtual space 11, the VR user can make predetermined entries on the virtual blackboard 36. Furthermore, the VR user's writing action on the virtual blackboard 36 is detected by a detection means for detecting the writing action (hereinafter referred to as "VR side writing detection means"), and the output from the VR side writing detection means is received by the VR side input receiving unit 916 as an input from the VR user. The VR side writing detection means may be, for example, a detection means capable of detecting the movement of the VR user's hands. Specifically, the VR side writing detection means may be, for example, a controller 280 or the like. Furthermore, the VR side writing detection means may be a camera or the like as the detection device 260. Furthermore, the VR side writing detection means may be, for example, a device worn by the VR user.

The writing action of the VR user is reflected on the blackboard 35 in the mixed reality space 21. In other words, when the VR user performs a writing action, the blackboard 35 changes to a state in which the writing content written by the VR user is displayed. In other words, the input from the VR user received by the VR side input receiving unit 916 is reflected on the blackboard 35. The input from the VR user may be reflected on the blackboard 35, for example, as follows. That is, for example, the MR side object control unit 820 may place a writing object 37 indicating the writing content written by the VR user at the position of the blackboard 35 in the mixed reality space 21. In addition, the blackboard display control means may control the image displayed by the display or projector of the blackboard 35 to be an image indicating the writing content written by the VR user, based on the input from the VR user. In other words, the MR-side object control unit 820 or the blackboard display control means can function as a reflection means that reflects input from a VR user received by the VR-side input receiving unit 916 in a real object in the mixed reality space 21.

The writing action of the VR user is also reflected on the virtual blackboard 36 in the virtual space 11. In other words, when the VR user performs a writing action, the virtual blackboard 36 changes to a state in which the writing content written by the VR user is displayed. In other words, the input from the VR user received by the VR side input receiving unit 916 is reflected on the virtual blackboard 36. Specifically, the VR side object control unit 920 applies a predetermined change to the virtual blackboard 36 based on the input from the VR user received by the VR side input receiving unit 916. More specifically, the VR side object control unit 920 reflects the input from the VR user on the virtual blackboard 36, and controls the virtual blackboard 36 to a state in which the writing content written by the VR user is displayed. Specifically, for example, when information on the writing action of the VR user is input to the VR-side input receiving unit 916, the VR-side object control unit 920 may place an entry object 38 indicating the contents written by the VR user at the position of the virtual blackboard 36 in the virtual space 11. Also, the VR-side object control unit 920 may control the virtual blackboard 36 to display the contents written by the VR user, for example, by changing the form of the virtual blackboard 36 itself. Also, the virtual blackboard 36 may be a virtual object that displays an image displayed by the display or projector of the blackboard 35, and the VR-side object control unit 920 may display the image on the virtual blackboard 36. Also, the MR system 400 may be provided with a camera that captures the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 may be a virtual object that displays an image captured by the camera, and the VR-side object control unit 920 may display the image on the virtual blackboard 36. In other words, the input from the VR user received by the VR-side input receiving unit 916 may be reflected in a real object in the mixed reality space 21, thereby causing a change in the virtual blackboard 36 based on the input from the VR user.

As described above, in this embodiment, actions taken by the MR user 6 with respect to a real object and actions taken by the VR user with respect to a virtual object corresponding to the real object are reflected in the mixed reality space 21 and the virtual space 11. The reflection of each action in the mixed reality space 21 or the virtual space 11 may be as follows:

That is, for example, when the MR user 6 points to an arbitrary position in the mixed reality space 21 with a predetermined pointer such as a laser pointer, the pointed position (hereinafter referred to as the "pointed position") may be known in the virtual space 11. Specifically, for example, in the mixed reality space 21, the detection device 460 or the camera 413 of the MR system 400 may function as a pointed position detection means for detecting the pointed position. Then, information on the pointed position detected by the pointed position detection means may be received by the MR side input reception unit 816 as an input from the MR user 6. In addition, the VR side object control unit 920 may place a virtual object representing light from a laser pointer at a position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21. In other words, the control unit 310 may control the display of the virtual space 11 in the VR device 210 so that the position in the virtual space 11 corresponding to the pointed position in the mixed reality space 21 is illuminated in a predetermined color representing light from a laser pointer.

Also, for example, when a VR user points to an arbitrary position in the virtual space 11 with a predetermined pointer such as a virtual laser pointer, the indicated position may be known in the mixed reality space 21. Specifically, for example, when information on the operation of the VR user on the controller 280 for operating the virtual laser pointer (in other words, information on the movement of the VR user) is input to the VR side input receiving unit 916, the VR side object control unit 920 may place a virtual object indicating light from the virtual laser pointer at the position indicated by the operation of the VR user (in other words, the position indicated by the virtual laser pointer). In other words, the control unit 310 may control the display of the virtual space in the VR device 210 so that the position in the virtual space 11 indicated by the operation of the virtual VR user is illuminated in a predetermined color representing light from the virtual laser pointer. Also, the control unit 510, etc. may control so that the position in the mixed reality space 21 corresponding to the indicated position in the virtual space 11 (specifically, the indicated position by the VR user) is illuminated in a predetermined color representing light from the virtual laser pointer. Specifically, the MR-side object control unit 820 may place a virtual object showing light from a virtual laser pointer at a position in the mixed reality space 21 corresponding to the indicated position in the virtual space 11. Also, for example, when the indicated position by the VR user is a predetermined position on the virtual blackboard 36, the blackboard display control means may control, based on the input from the VR user, the image displayed by the display or projector of the blackboard 35 to be an image in which a position corresponding to the predetermined position on the virtual blackboard 36 is illuminated (i.e., an image indicating that a predetermined location is being pointed to by the laser pointer). Also, for example, the detection device 460, the MR device 410, etc. may include a device that emits laser light, and the control unit 510 may control the device to apply laser light to a position in the mixed reality space 21 corresponding to the indicated position in the virtual space 11 in response to the input from the VR user received by the VR-side input receiving unit 916.

Furthermore, the behavior of the MR user 6 or the VR user may be reflected in the mixed reality space 21 and the virtual space 11 as follows. That is, for example, a device capable of inputting or outputting sound as a real object (e.g., an electronic piano) may be placed in the mixed reality space 21, and the behavior of the MR user 6 or the VR user related to the device may be reflected in the mixed reality space 21 and the virtual space 11. In the following, it is assumed that an electronic piano is placed as a real object in the mixed reality space 21, and a virtual electronic piano is placed as a virtual object corresponding to the electronic piano in the virtual space 11 at a position corresponding to the electronic piano.

The electronic piano may be capable of communicating with the computer 500, for example. When the MR user 6 operates the electronic piano in the mixed reality space 21, information about the operation (for example, sound data corresponding to the operation) may be sent to the computer 300 via the computer 500. The sound control unit 945 may output a sound from the speaker 216 based on the information. As a result, for example, when the MR user 6 operates a key of the electronic piano (for example, a key of the note "C"), a sound corresponding to the operated key (for example, the note "C") may be output from the speaker 216. In other words, the MR side input receiving unit 816 may receive an output (for example, sound data corresponding to the operation) from the electronic piano (in other words, a sensor that detects an operation on the key, etc.) as a detection means for detecting the operation of the MR user 6 as an input from the MR user 6, and the control unit 310 (for example, the sound control unit 945) may output a sound corresponding to the operation of the MR user 6 on the electronic piano to the speaker 216 based on the input. The sound control unit 945 may also control the direction from which the sound is emitted so that the sound corresponding to an operation on the electronic piano sounds as if it is being output from a virtual electronic piano in the virtual space 11.

In addition, when the VR user operates the virtual electronic piano in the virtual space 11, information on the operation may be sent to the computer 500. The sound control unit 845 may output a sound from the speaker 416 based on the information. As a result, for example, when the VR user operates a key (e.g., a key of the note "C") of the virtual electronic piano, a sound corresponding to the operated key (e.g., the note "C") may be output from the speaker 416. In other words, the VR side input receiving unit 916 may receive an output from a detection means (e.g., the detection device 260 or the controller 280) that detects the operation of the VR user on the virtual electronic piano as an input from the VR user, and the control unit 510 (e.g., the sound control unit 845) may output a sound corresponding to the operation of the VR user on the virtual electronic piano to the speaker 416 based on the input. The control unit 510 may output a sound corresponding to the operation of the VR user on the virtual electronic piano to the speaker of the electronic piano.

When the MR user 6 operates the electronic piano, the sound corresponding to the operation may be output in the mixed reality space 21 by the speaker of the electronic piano or by the speaker 416. Also, when the VR user operates the virtual electronic piano, the sound corresponding to the operation may be output in the virtual space 11 by the speaker 216.

<Processing Executed by Information Processing System>
Next, the flow of processing executed by the information processing system 100 will be described with reference to FIGS.

An example of the processing performed by the VR system 200 is described with reference to FIG. 11.

In step S101, the processor 301 of the computer 300 identifies virtual space data and defines the virtual space 11.

In step S102, the processor 301 places the VR avatar 25 of the VR user in the virtual space 11.

In step S103, the processor 301 acquires position information indicating the position of the MR user 6 in the mixed reality space 21 from the processor 501 of the computer 500, which is connected to the computer 300 via the network 2 and controls the mixed reality space 21 corresponding to the virtual space 11.

In step S104, the processor 301 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21 based on the position information acquired in step S103.

In step S105, the processor 301 places the virtual model 30 as a virtual object in the virtual space 11. The processor 301 places the virtual model 30 at a position in the virtual space 11 that corresponds to the position of the virtual model 30 in the mixed reality space 21. Note that placing at a corresponding position may mean that the position in the mixed reality space 21 is determined first, or that the position in the virtual space 11 is determined first.

In step S106, the processor 301 accepts input from the VR user. Specifically, the processor 301 accepts, for example, the output of a detection means that detects the movement of the VR user as input from the VR user.

In step S107, the processor 301 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the VR user. Specifically, the processor 301 moves the VR avatar 25 or the virtual model 30 in the virtual space 11 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 301 moves the VR avatar 25. Also, for example, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30. Also, the processor 301 transmits information related to the input from the VR user to the processor 501.

In step S108, the processor 301 acquires information regarding the input from the MR user 6 that was accepted by the processor 501.

In step S109, the processor 301 makes a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 acquired in step S108. Specifically, the processor 301 moves the MR avatar 26 or the virtual model 30 in the virtual space 11, for example, based on the input from the MR user 6. For example, if the input is related to the MR avatar 26, the processor 301 moves the MR avatar 26. Also, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30.

Next, an example of the processing performed by the MR system 400 will be described with reference to FIG. 12.

In step S201, the processor 501 of the computer 500 defines the mixed reality space 21.

In step S202, the processor 501 detects the position of the MR user 6 in the mixed reality space 21. The processor 501 also transmits position information indicating the detected position of the MR user 6 to the processor 301 of the computer 300 that is connected to the computer 500 via the network 2 and that controls the virtual space 11 that corresponds to the mixed reality space 21.

In step S203, the processor 501 acquires location information indicating the location of the VR avatar 25 in the virtual space 11 from the processor 301.

In step S204, the processor 501 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR avatar 25 in the virtual space 11, based on the position information acquired in step S203. Note that placing at a corresponding position may mean that the position in the virtual space 11 is determined first, or that the position in the mixed reality space 21 is determined first.

In step S205, the processor 501 places the virtual model 30 as a virtual object in the mixed reality space 21. The processor 501 places the virtual model 30 at a position in the mixed reality space 21 that corresponds to the position of the virtual model 30 in the virtual space 11.

In step S206, the processor 501 accepts input from the MR user 6. Specifically, the processor 501 accepts, for example, the output of a detection means that detects the movement of the MR user 6 as input from the MR user 6.

In step S207, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the MR user 6. Specifically, the processor 501 moves the virtual model 30 in the mixed reality space 21, for example, based on the input from the MR user 6. The processor 501 also transmits information regarding the input from the MR user 6 to the processor 301.

In step S208, the processor 501 acquires information regarding the input from the VR user that was accepted by the processor 301.

In step S209, the processor 501 makes a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user acquired in step S208. Specifically, the processor 501 moves the VR avatar 25 or the virtual model 30 in the mixed reality space 21 based on, for example, the input from the VR user. For example, if the input is related to the VR avatar 25, the processor 501 moves the VR avatar 25. Also, if the input is related to the virtual model 30, the processor 501 moves the virtual model 30.

Next, an example of a process for reflecting the behavior of the MR user 6 in the virtual space 11 will be described with reference to FIG. 13.

In step S301, the processor 501 accepts an input from the MR user 6. Specifically, the processor 501 accepts, as an input from the MR user 6, the output of a detection means that detects the behavior of the MR user 6 with respect to a specific real object (e.g., a blackboard 35, a laser pointer, or an electronic piano, etc.).

In step S302, the processor 501 transmits information regarding the input from the MR user 6 received in step S301 to the processor 301 (in other words, the VR system 200).

In step S303, the processor 301 makes a predetermined change to the virtual object corresponding to the specific real object on which the MR user 6 performed an action, based on the information transmitted in step S302. For example, when the MR user 6 performs a writing action to write predetermined contents on the blackboard 35, the processor 301 changes the virtual blackboard 36 corresponding to the blackboard 35 to a state in which the predetermined contents are displayed.

Next, an example of a process for reflecting the actions of a VR user in the mixed reality space 21 will be described with reference to FIG. 14.

In step S351, the processor 301 accepts an input from a VR user. Specifically, the processor 301 accepts, as an input from the VR user, the output of a detection means that detects the VR user's behavior with respect to a virtual object (e.g., a virtual blackboard 36 or a virtual electronic piano) that corresponds to a specific real object.

In step S352, the processor 301 transmits information regarding the input from the VR user received in step S351 to the processor 501 (in other words, the MR system 400).

In step S353, the processor 501 reflects the action performed by the VR user on a virtual object corresponding to a specific real object in the specific real object based on the information transmitted in step S352. For example, when the VR user performs a writing action to write a specific entry on the virtual blackboard 36, the processor 501 changes the blackboard 35 to a state in which the specific entry is displayed.

The present invention is not limited to the above-described embodiment, and can be modified in various ways without departing from the spirit of the invention. Within the scope of the invention, the components can be freely combined, any component can be modified, or any component can be omitted. The process flow described in this specification is merely an example, and the order and configuration of each process can be different.

<Additional Notes>
The matters described in the above embodiment can also be described as follows.

(Appendix 1)
A first display means (e.g., a VR device 210) for providing a first user with a first view, which is a view of a virtual space simulating a predetermined real space;
a second display means (e.g., an MR device 410) for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object (e.g., a VR avatar 25 or a virtual model 30) corresponding to a first virtual object (e.g., a VR avatar 25 or a virtual model 30) in the virtual space is arranged in the predetermined real space;
A first input receiving means (e.g., a VR side input receiving unit 916) for receiving an input from the first user;
A first control means (e.g., the control unit 310 or the VR-side object control unit 920) for controlling a virtual object in the virtual space;
A second control means (e.g., the control unit 510 or the MR-side object control unit 820) for controlling a virtual object in the mixed reality space,
the first control means controls the first virtual object in the virtual space based on an input from the first user accepted by the first input accepting means;
The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on input from the first user accepted by the first input accepting means.
According to this configuration, the second virtual object displayed in the predetermined real space is controlled based on the input from the first user, similarly to the first virtual object in the virtual space simulating the predetermined real space. For example, when the first user moves the first virtual object in the virtual space simulating the predetermined real space, the second virtual object in the predetermined real space (in other words, the mixed reality space) also moves in the same manner. And since the virtual space is a simulation of the predetermined real space, the first user can imagine how the second virtual object is controlled in the real space, how it looks, etc., through the first virtual object controlled in the virtual space. Thus, according to the present invention, the real space and the virtual space are highly integrated, and the range of uses of the virtual space is expanded.

(Appendix 2)
The information processing system according to claim 1, wherein the first virtual object and the second virtual object are avatars of the first user.
According to this configuration, an avatar of the first user controlled based on an input from the first user is displayed in the mixed reality space. Also, a second user can share an experience with the avatar of the first user in the mixed reality space. Therefore, the real space and the virtual space are highly integrated, and the range of uses of the virtual space is expanded.

(Appendix 3)
the first input receiving means receives an output of a detection means for detecting a movement of the first user as an input from the first user;
The information processing system described in Appendix 2, wherein the second control means controls the second virtual object so that movement of an avatar of the first user in the mixed reality space mimics the movement of the first user detected by the detection means.
According to such a configuration, the movement of the first user is reflected as it is in the avatar, so that the first user can easily communicate something to the second user or express emotions. At this time, the first user can move the avatar in a virtual space that imitates the real space, so that the avatar can move more naturally. In addition, from the viewpoint of the second user, the avatar of the first user appears in the real space, and the avatar moves naturally in the real space and highly reflects the intention of the first user. Therefore, the quality of communication between the first user and the second user via the virtual space is improved, and the range of uses of the virtual space is expanded.

(Appendix 4)
The first control means is
A specific virtual object (e.g., a virtual blackboard 36 or a virtual electronic piano) corresponding to a specific object (e.g., a blackboard 35 or an electronic piano) actually existing in the predetermined real space is placed in the virtual space;
imparting a predetermined change to the specific virtual object based on the input from the first user received by the first input receiving means;
The information processing system described in Appendix 1 includes a reflection means (e.g., an MR side object control unit 820, a blackboard display control means, or a sound control unit 845) that reflects the input from the first user received by the first input receiving means to the specific object in the mixed reality space.
According to this configuration, an operation performed by a first user on a virtual object in a virtual space can be reflected on an object that actually exists in the real space in which a second user is located. This allows a high level of integration between the real space and the virtual space, expanding the range of uses for the virtual space.

(Appendix 5)
A second input receiving unit (e.g., an MR side input receiving unit 816) is provided to receive an input from the second user,
The information processing system according to claim 1, wherein the first control means applies a predetermined change to the first virtual object based on the input from the second user received by the second input receiving means.
According to this configuration, the second user can also affect the virtual object in the virtual space. In addition, since the first user and the second user affect changes to the target object in a space of similar shape, it becomes possible for each user to easily understand how the target object looks from the other user's perspective when affecting the target object. Therefore, the range of uses of the virtual space is expanded.

(Appendix 6)
A second input receiving unit (e.g., an MR side input receiving unit 816) is provided to receive an input from the second user,
the second input receiving means receives an output from a position detection means that detects a position of the second user in the mixed reality space as an input from the second user;
The information processing system according to any one of appendices 1 to 5, wherein the first control means places an avatar of the second user at a position in the virtual space corresponding to the position of the second user in the mixed reality space detected by the position detection means.
According to this configuration, in the virtual space, the avatar of the second user is placed at a position corresponding to the actual position of the second user. Therefore, the first user can easily grasp where the second user is actually located. Therefore, the first user can easily grasp the relationship between the position of the second virtual object that moves according to the first user's input and the position of the second user. Therefore, the range of uses of the virtual space is expanded.

(Appendix 7)
A first display means (e.g., a VR device 210) for providing a first user with a first view, which is a view of a virtual space simulating a predetermined real space;
a second display means (e.g., an MR device 410) for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object (e.g., a VR avatar 25 or a virtual model 30) corresponding to a first virtual object (e.g., a VR avatar 25 or a virtual model 30) in the virtual space is arranged in the predetermined real space;
A first input receiving means (e.g., a VR side input receiving unit 916) for receiving an input from the first user;
A first control means (e.g., a control unit 310 or a VR-side object control unit 920) that controls a virtual object in the virtual space and controls the first virtual object in the virtual space based on an input from the first user that is accepted by the first input accepting means;
a second control unit (e.g., the control unit 510 or the MR-side object control unit 820) that controls a virtual object in the mixed reality space; and causing a computer of the information processing system to function as the second control unit;
The second control means is a program that places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on input from the first user accepted by the first input accepting means.
With this configuration, the same effects as those of the program described in Supplementary Note 1 can be achieved.

11 Virtual space, 21 Mixed reality space, 25 VR avatar, 26 MR avatar, 30 Virtual model, 35 Blackboard, 36 Virtual blackboard, 100 Information processing system, 200 VR system, 210 VR device, 211 Display, 212 Gaze sensor, 213 First camera, 214 Second camera, 215 Microphone, 216 Speaker, 217 Sensor, 260 Detection device, 270 Display, 280 Controller, 286 Sensor, 300 Computer, 301 Processor, 302 Memory, 303 Storage, 304 Input/output interface, 305 Communication interface, 310 Control unit, 311 Storage unit, 400 MR system, 410 MR device, 411 Display, 412 Gaze sensor, 413 Camera, 415 Microphone, 416 Speaker, 417 Sensor, 460 Detection device, 470 Display, 480 Controller, 486 Sensor, 500 Computer, 501 Processor, 502 Memory, 503 Storage, 504 Input/Output interface, 505 Communication interface, 510 Control unit, 511 Storage unit, 600 Server, 601 Processor, 602 Memory, 603 Storage, 604 Input/Output interface, 605 Communication interface, 610 Control unit, 611 Storage unit, 700 External device, 810 Virtual space generation unit, 812 Coordinate definition unit, 816 MR side input reception unit, 818 User information acquisition unit, 820 MR side object control unit, 840 Display control unit, 845 Sound control unit, 850 Communication control unit, 916 VR side input reception unit, 920 VR side object control unit, 930 Virtual camera control unit, 940 Display control unit, 945 Sound control unit, 950 Communication control section

Claims (7)

a first display means for providing a first user with a first view, the first view being a view of a virtual space simulating a predetermined real space;
a second display means for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object corresponding to the first virtual object in the virtual space is arranged in the predetermined real space;
a first input receiving means for receiving an input from the first user;
a first control means for controlling a virtual object in the virtual space;
a second control means for controlling a virtual object in the mixed reality space,
the first control means controls the first virtual object in the virtual space based on an input from the first user accepted by the first input accepting means;
The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on input from the first user accepted by the first input accepting means. The information processing system according to claim 1 , wherein the first virtual object and the second virtual object are avatars of the first user. the first input receiving means receives an output of a detection means for detecting a movement of the first user as an input from the first user;
The information processing system according to claim 2 , wherein the second control means controls the second virtual object so that the movement of the avatar of the first user in the mixed reality space mimics the movement of the first user detected by the detection means. The first control means is
placing a specific virtual object in the virtual space, the specific virtual object corresponding to a specific object that actually exists in the predetermined real space;
imparting a predetermined change to the specific virtual object based on the input from the first user received by the first input receiving means;
The information processing system according to claim 1 , further comprising: a reflecting unit that reflects the input from the first user accepted by the first input accepting unit, on the specific object in the mixed reality space. a second input receiving means for receiving an input from the second user;
The information processing system according to claim 1 , wherein the first control means applies a predetermined change to the first virtual object based on the input from the second user received by the second input receiving means. a second input receiving means for receiving an input from the second user;
the second input receiving means receives an output from a position detection means that detects a position of the second user in the mixed reality space as an input from the second user;
The information processing system according to any one of claims 1 to 5, wherein the first control means places an avatar of the second user at a position in the virtual space corresponding to the position of the second user in the mixed reality space detected by the position detection means. a first display means for providing a first user with a first view, the first view being a view of a virtual space simulating a predetermined real space;
a second display means for providing a second user with a second view, which is a view of a mixed reality space in which a second virtual object corresponding to the first virtual object in the virtual space is arranged in the predetermined real space;
a first input receiving means for receiving an input from the first user;
a first control means for controlling a virtual object in the virtual space, and for controlling the first virtual object in the virtual space based on an input from the first user received by the first input receiving means;
a second control means for controlling a virtual object in the mixed reality space,
The second control means is a program that places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and controls the second virtual object in the mixed reality space based on input from the first user accepted by the first input accepting means.

Images