JP7412497B1 - information processing system - Google Patents

Abstract

[Problem] Expand the range of uses of virtual space. The information processing system includes a first display unit that provides a first user with a first view that is a view of a virtual space imitating a predetermined real space; a second display means that provides a second user with a second view that is a view of a mixed reality space in which a second virtual object corresponding to the first virtual object is arranged; and a first input reception that receives input from the first user. means, a first control means, and a second control means, the first control means controlling the first virtual object in the virtual space based on the input from the first user received by the first input receiving means. The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and receives the input from the first user received by the first input reception means. A second virtual object in the mixed reality space is controlled based on the input. [Selection diagram] Figure 8

Description

The present invention relates to an information processing system and a program.

BACKGROUND ART Conventionally, techniques for providing mixed reality (MR) spaces and virtual reality (VR) spaces to users have been known (for example, see Patent Document 1 and Patent Document 2).

Patent No. 6860488 Patent No. 6800599

Incidentally, there has been a demand for expanding the range of uses of virtual space.

The present invention has been made in view of the above circumstances, and aims to widen the range of uses of virtual space.

According to one embodiment presented in the present disclosure,
a first display means for providing a first user with a first view that is a view of a virtual space imitating a predetermined real space;
a second display means for providing a second user with a second view that is a view of a mixed reality space in which a second virtual object corresponding to the first virtual object in the virtual space is arranged in the predetermined real space; ,
a first input receiving means for receiving input from the first user;
a first control means for controlling a virtual object in the virtual space;
a second control means for controlling the virtual object in the mixed reality space,
The first control means controls the first virtual object in the virtual space based on the input from the first user received by the first input reception means,
The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and An information processing system is provided that controls the second virtual object in the mixed reality space based on input from one user.

According to the present invention, the range of uses of virtual space can be expanded.

1 is a diagram showing a schematic configuration of an information processing system. FIG. 1 is a diagram showing a schematic configuration of a VR system. 1 is a diagram showing a schematic configuration of an MR system. FIG. 2 is a diagram conceptually showing a virtual space. FIG. 3 is a diagram showing a YZ cross section of a viewing area viewed from the X direction in virtual space. FIG. 3 is a diagram showing an XZ cross section of a viewing area viewed from the Y direction in virtual space. FIG. 2 is a diagram showing a schematic configuration of a controller. FIG. 2 is a diagram for explaining mixed reality space and virtual space. FIG. 1 is a diagram showing a functional configuration of an information processing system. FIG. 3 is a diagram for explaining a real object and a virtual object corresponding to the real object. It is a flowchart which shows an example of the process performed by a VR system. 3 is a flowchart illustrating an example of processing executed by the MR system. 12 is a flowchart illustrating an example of a process for reflecting an MR user's behavior in a virtual space. 12 is a flowchart illustrating an example of a process for reflecting a VR user's behavior in a mixed reality space.

Embodiments of the present invention will be described below with reference to the drawings.

<Configuration of information processing system>
As shown in FIG. 1, the information processing system 100 of this embodiment includes a VR (Virtual Reality) system 200, an MR (Mixed Reality) system 400, a server 600, and an external device 700. The VR system 200 is configured to be able to communicate with the server 600, the MR system 400, and the external device 700 via the network 2. Further, the MR system 400 is configured to be able to communicate with the server 600, the VR system 200, and the external device 700 via the network 2. The number of MR systems 400 that constitute the information processing system 100 is not limited to one, and a plurality of MR systems may exist. Further, the number of VR systems 200 configuring the information processing system 100 is not limited to one, and a plurality of VR systems 200 may exist. Note that in this embodiment, communication between the VR system 200 and the MR system 400 via the network 2 is performed via the server 600, but may be performed without using the server 600. Note that the network 2 includes, for example, the Internet, a mobile communication system (for example, 3G, 4G, 5G, LTE (Long Term Evolution), etc.), Wi-Fi (registered trademark), Bluetooth (registered trademark), other communication lines, Alternatively, it may be configured by any combination of these.

(VR system configuration)
As shown in FIG. 2, the VR system 200 includes a VR device 210, a computer 300, a detection device 260, a display 270, and a controller 280. The VR device 210 includes a display 211, a gaze sensor 212, a first camera 213, a second camera 214, a microphone 215, a speaker 216, and a sensor 217. Hereinafter, the user who uses the VR device 210 will be referred to as a VR user.

The computer 300 can be connected to the Internet or other network 2, and can communicate with, for example, the server 600, the computer of the MR system 400, and other computers connected to the network 2. Examples of other computers include computers of other VR systems 200, external equipment 700, and the like.

The VR device 210 can be worn on the head of a VR user and provide a virtual space to the VR user during operation. The VR device 210 may be, for example, a so-called head-mounted display including a display, or may be a head-mounted device equipped with a smartphone or other terminal having a display. For example, the VR device 210 displays an image for the right eye and an image for the left eye on the display 211. When each eye of a VR user views a respective image, the VR user can recognize the image as a three-dimensional image based on the parallax between the eyes.

The display 211 is realized, for example, as a non-transmissive display device. The display 211 is arranged, for example, on the main body of the VR device 210 so as to be located in front of both eyes of the VR user. Therefore, when the VR user views the three-dimensional image displayed on the display 211, the VR user can immerse himself in the virtual space. Note that the display 211 may be realized by a display included in a so-called smartphone or other terminal.

Detection device 260 detects the movement of a VR user using VR device 210. The detection device 260 may have a position tracking function for detecting the movement of the VR device 210, thereby detecting the movement of the VR user. Specifically, the detection device 260 has a sensor that reads light (for example, infrared light) from the VR device 210 as a sensor for detecting the movement of the VR user, and detects the movement of the VR device in the real space. It may be possible to detect the position, inclination, etc. of 210. In this case, the VR device 210 may include a plurality of light sources (not shown). Further, each light source may be realized by, for example, an LED (Light Emitting Diode) that emits infrared light.

Further, the detection device 260 may be realized by, for example, a camera. Specifically, the detection device 260 has an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor) as a sensor for detecting the movement of the VR user. It may be. In other words, the VR user's movements may be detected by the camera. The detection device 260 is, for example, a depth camera that includes a device that emits predetermined light such as infrared light or light with a predetermined pattern, and an image sensor (for example, a depth sensor), and detects the light emitted from the device. The image sensor may detect the reflected light of the image sensor, and the position, posture, etc. of the VR user may be detected based on the output from the image sensor. Further, the detection device 460 includes such a depth camera and a camera capable of acquiring RGB images, and detects the position, posture, etc. of the VR user based on the output from these cameras. There may be. Further, the detection device 260 may be, for example, a stereo camera including a plurality of image sensors, and may be one that detects the position, posture, etc. of the VR user based on outputs from the plurality of image sensors. Note that the detection of the position, posture, etc. of the VR user may be the detection of the position, posture, etc. of the VR user's body, or may be the detection of the position, tilt, etc. of the VR device 210.

Note that the VR system 200 may have one or more types of detection devices as the detection device 260, or may have a plurality of each type of detection device. Furthermore, the VR system 200 does not need to include the detection device 260. Note that a part of the detection device 260 may be configured by the computer 300 or the like. For example, analysis of the output from the image sensor (eg, image recognition, etc.) may be performed by the computer 300 or the like.

In addition to the various sensors described above, the detection device 260 may include a sensor capable of detecting the position, inclination, etc. of the detection device 260 itself. Specifically, for example, the detection device 260 includes an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. You can. Further, the outputs from these sensors may be sent to the computer 300 or the like, and may be used, for example, when predetermined processing is performed based on the outputs from the image sensor included in the detection device 260.

Further, the VR device 210 may include a sensor 217 instead of or in addition to the detection device 260 as a detection means for detecting the movement of the VR user. The VR device 210 can use the sensor 217 to detect the position and tilt of the VR device 210 itself. The sensor 217 may be, for example, an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. Further, the VR device 210 may have one or more types of sensors as the sensor 217, or may have a plurality of each type of sensor. As an example, when an angular velocity sensor is used as the sensor 217, the VR device 210 can detect the angular velocity around three axes of the VR device 210 in real space over time. Then, the VR device 210 calculates the temporal change in the angle of the VR device 210 around the three axes based on each angular velocity, and further calculates the tilt of the VR device 210 based on the temporal change in the angle. etc. can be done. Further, the sensor 217 may be, for example, an image sensor. Then, the position, posture, etc. of the VR user may be detected based on the output from the image sensor. In other words, the position, posture, etc. of the VR user may be detected based on information etc. from a camera included in the VR device 210 that photographs the surroundings of the VR device 210. In other words, tracking of the VR device 210 may be performed using an outside-in method or an inside-out method.

Gaze sensor 212 detects the direction in which the VR user's right and left eyes are directed. That is, the gaze sensor 212 detects the VR user's line of sight (in other words, eye movement). Detection of the direction of the line of sight is realized, for example, by a known eye tracking function. The gaze sensor 212 is realized by a sensor having the eye tracking function. Gaze sensor 212 may include a right eye sensor and a left eye sensor. The gaze sensor 212 may be, for example, a sensor that irradiates infrared light to the right and left eyes of the VR user and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. . In this case, the gaze sensor 212 can detect the VR user's line of sight based on each detected rotation angle.

The first camera 213 photographs the lower part of the VR user's face. More specifically, the first camera 213 photographs the nose, mouth, etc. of the VR user. The second camera 214 photographs the eyes, eyebrows, etc. of the VR user. Here, the VR user side of the housing of the VR device 210 is defined as the inside of the VR device 210, and the side of the housing of the VR device 210 opposite to the VR user is defined as the outside of the VR device 210. The first camera 213 may be placed outside the VR device 210, and the second camera 214 may be placed inside the VR device 210. Images captured by the first camera 213 and the second camera 214 are input to the computer 300. Note that the first camera 213 and the second camera 214 may be implemented as one camera, and the face of the VR user may be photographed with this one camera.

The microphone 215 serving as a sound input means converts the voice of the VR user into a sound signal (in other words, an electrical signal) and outputs it to the computer 300. The speaker 216 as a sound output means converts the sound signal into sound and outputs it to the VR user. Note that the VR device 210 may include earphones instead of the speaker 216 as a sound output means.

Display 270 displays an image similar to the image displayed on display 211. This allows users other than the VR user wearing the VR device 210 to view the same images (in other words, virtual space) as the VR user. The image displayed on the display 270 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the VR device 210. Examples of the display 270 include a liquid crystal display and an organic EL display.

Controller 280 is connected to computer 300 by wire or wirelessly. The controller 280 accepts input operations related to instructions from the VR user to the computer 300. Further, the controller 280 accepts input operations by the VR user for controlling the position and movement of virtual objects placed in the virtual space. The controller 280 may be configured to be graspable by a VR user, for example. Further, the controller 280 may be configured to be able to be attached to a part of the VR user's body or clothing, for example. Specifically, the controller 280 may be, for example, glove-shaped. Further, the controller 280 may be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 300.

Further, the controller 280 may include multiple light sources. Each light source may be realized by, for example, an LED that emits infrared light. Then, the detection device 260 may read the infrared light from the controller 280 and detect the position, tilt, etc. of the controller 280 in the real space. In other words, the detection device 260 may have a position tracking function that detects the movement of the controller 280, thereby detecting the movement of the VR user.

Note that the VR system 200 does not need to include the controller 280.

Furthermore, the information processing system 100 may include a sensor 286 that detects the movement of the VR user. Sensor 286 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. Further, the sensor 286 may be provided in the controller 280, for example (see FIG. 7). Note that the information processing system 100 may have one or more types of sensors as the sensor 286, or may have a plurality of each type of sensor. Furthermore, a device including the sensor 286 (for example, the controller 280, a predetermined camera, etc.) and the computer 300 may be connected to each other wirelessly, for example. The information acquired by sensor 286 may then be transmitted to computer 300 via wireless communication, for example.

Further, the position and movement of the VR user can be determined by the VR device 210 equipped with a device that performs predetermined wireless communication (for example, short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB (Ultra Wide Band) communication). , the VR device 210 may be detected by performing wireless communication between the device and surrounding devices and acquiring position information of the VR device 210.

In addition, the position and movement of the VR user can be determined by using a device worn by the VR user (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). The detection may be performed using an external device 700 that is capable of tracking the user's movement (and a device that tracks the VR user's movement in cooperation with the device through short-range wireless communication or the like). Further, these external devices 700 may be used as the controller 280. Further, the position and movement of the VR user may be detected by a GPS sensor or the like.

(Configuration of computer 300)
As shown in FIG. 2, the computer 300 includes a processor 301, a memory 302, a storage 303, an input/output interface 304, and a communication interface 305 as main components. Each component is connected to each other by a bus.

Processor 301 controls the operation of VR device 210. The processor 301 reads the program from the storage 303 and expands it into the memory 302. Processor 301 executes the developed program. The processor 301 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 302 is a main storage device. The memory 302 is configured of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory 302 provides a work area to the processor 301 by temporarily storing programs and various data that the processor 301 reads from the storage 303. The memory 302 also temporarily stores various data generated while the processor 301 is operating according to a program, various data input to the computer 300, and the like.

Storage 303 is an auxiliary storage device. The storage 303 is configured of a storage device such as a flash memory or an HDD (Hard Disk Drive), for example. The storage 303 stores programs for providing services in the information processing system 100. Further, the storage 303 stores various data for providing services in the information processing system 100. The data stored in the storage 303 includes data for defining a virtual space, data regarding virtual objects, and the like.

Note that the storage 303 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 303 built into the computer 300, programs and data stored in an external storage device may be used.

The input/output interface 304 is an interface through which the computer 300 receives data input, and an interface through which the computer 300 outputs data. The input/output interface 304 can send or receive data to/from the VR device 210, the detection device 260, and the display 270. Furthermore, data may be transmitted to or received from the display 211, gaze sensor 212, first camera 213, second camera 214, microphone 215, speaker 216, and sensor 217 included in the VR device 210.

Note that the input/output interface 304 may transmit or receive data to/from the controller 280. For example, input/output interface 304 may receive input of signals output from controller 280 and sensor 286. Further, the input/output interface 304 may transmit instructions output from the processor 301 to the controller 280. The command may instruct the controller 280 to vibrate, output sound, emit light, or the like. Upon receiving the command, the controller 280 performs vibration, sound output, light emission, etc. in accordance with the command.

Note that each of the VR device 210, the detection device 260, the display 270, the controller 280, and the sensor 286 and the computer 300 may be connected by wire or wirelessly.

The communication interface 305 controls transmission and reception of various data via the network 2 with other computers (for example, the server 600, the computer 500, or the other computer 300, etc.).

The processor 301 accesses the storage 303, loads the program stored in the storage 303 into the memory 302, and executes a series of instructions included in the program. Further, the processor 301 sends a signal for providing a virtual space to the VR device 210 via the input/output interface 304.

Note that the computer 300 may be provided outside the VR device 210, or part or all of the computer 300 may be built into the VR device 210. Further, a portable terminal (for example, a smartphone) including the display 211 may function as at least a part of the computer 300. Note that when a plurality of VR apparatuses 210 exist, the computer 300 may be provided for each VR apparatus 210 or may be used in common for the plurality of VR apparatuses 210.

(Coordinate system)
In the VR system 200, a real coordinate system that is a coordinate system in real space is set in advance. The real coordinate system has three reference directions (axes) that are parallel to a vertical direction in real space, a horizontal direction perpendicular to the vertical direction, and a front-back direction perpendicular to both the vertical direction and the horizontal direction.

The position and tilt of the VR device 210 in real space can be detected by the detection device 260 and the sensor 217. The detected tilt of the VR device 210 corresponds to each tilt of the VR device 210 around three axes in the real coordinate system, for example. The computer 300 sets the uvw visual field coordinate system to the VR device 210 based on the inclination of the VR device 210 in the real coordinate system (see FIG. 4). The uvw visual field coordinate system set in the VR device 210 corresponds to the viewpoint coordinate system when a VR user wearing the VR device 210 views an object in the virtual space.

Here, the uvw visual field coordinate system will be explained. The computer 300 sets a three-dimensional uvw visual field coordinate system centered on the head of the VR user wearing the VR device 210 (origin). More specifically, the computer 300 tilts the horizontal direction, vertical direction, and front-rear direction that define the real coordinate system by the tilt of the VR device 210 around each axis within the real coordinate system. The three newly obtained directions are set as the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the uvw visual field coordinate system in the VR device 210.

For example, when a VR user wearing the VR device 210 is standing upright and viewing the front, the processor 301 sets a visual field coordinate system parallel to the real coordinate system in the VR device 210. In this case, the horizontal direction, vertical direction, and longitudinal direction in the real coordinate system coincide with the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the visual field coordinate system in the VR device 210. .

After the uvw visual field coordinate system is set in the VR device 210, the detection device 260 or sensor 217 can detect the tilt of the VR device 210 in the set uvw visual field coordinate system based on the movement of the VR device 210. In this case, the detection device 260 or the sensor 217 detects the pitch angle, yaw angle, and roll angle of the VR device 210 in the uvw visual field coordinate system as the inclination of the VR device 210, respectively. The pitch angle represents the tilt angle of the VR device 210 around the pitch axis in the uvw viewing coordinate system. The yaw angle represents the tilt angle of the VR device 210 around the yaw axis in the uvw viewing coordinate system. The roll angle represents the tilt angle of the VR device 210 around the roll axis in the uvw viewing coordinate system.

Based on the detected tilt of the VR device 210, the computer 300 sets in the VR device 210 a uvw visual field coordinate system in the VR device 210 after the VR device 210 moves. The relationship between the VR device 210 and the uvw visual field coordinate system of the VR device 210 is always constant regardless of the position and tilt of the VR device 210. When the position and inclination of the VR device 210 change, the position and inclination of the uvw visual field coordinate system of the VR device 210 in the real coordinate system change in conjunction with the change in the position and inclination.

Note that the detection device 260 may specify the position of the VR device 210 in the real space as a relative position with respect to the detection device 260. Furthermore, the processor 301 may determine the origin of the uvw visual field coordinate system of the VR device 210 in the real space (real coordinate system) based on the specified relative position.

(Virtual space)
The virtual space will be further explained with reference to FIG. 4. FIG. 4 is a diagram conceptually representing one aspect of representing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure that covers the entire 360 degree direction of the center 12. In FIG. 4, the celestial sphere in the upper half of the virtual space 11 is illustrated in order not to complicate the explanation. In the virtual space 11, each mesh is defined. The position of each mesh is predefined as a coordinate value in an XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 300 associates each partial image forming the panoramic image 13 (still image, moving image, etc.) that can be developed in the virtual space 11 with each corresponding mesh in the virtual space 11 .

In the virtual space 11, an XYZ coordinate system with a predetermined point as the origin 12 is defined. For example, the XYZ coordinate system is parallel to the real coordinate system. The horizontal direction, vertical direction (vertical direction), and front-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X axis of the XYZ coordinate system is parallel to the horizontal direction of the real coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the vertical direction of the real coordinate system, and the Z axis of the direction) is parallel to the front-back direction of the real coordinate system.

When the VR device 210 is started, that is, in the initial state of the VR device 210, the virtual camera 14 is placed at a predetermined position (for example, at the center) of the virtual space 11. Furthermore, the processor 301 displays the image captured by the virtual camera 14 on the display 211 of the VR device 210. The virtual camera 14 moves within the virtual space 11 in conjunction with the movement of the VR device 210 in the real space. Thereby, changes in the tilt and position of the VR device 210 in the real space can be similarly reproduced in the virtual space 11.

As in the case of the VR device 210, the uvw visual field coordinate system is defined for the virtual camera 14. The UVW visual field coordinate system of the virtual camera 14 in the virtual space 11 is defined to be linked to the UVW visual field coordinate system of the VR device 210 in the real space (real coordinate system). Therefore, when the tilt of the VR device 210 changes, the tilt of the virtual camera 14 also changes accordingly. Note that the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in the real space, but in this embodiment, even if the VR user moves in the real space, the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in the real space. It is designed not to move.

The processor 301 of the computer 300 defines the viewing area 15 in the virtual space 11 based on the position and tilt of the virtual camera 14. The viewing area 15 corresponds to an area of the virtual space 11 that is viewed by a VR user wearing the VR device 210. In other words, the position of the virtual camera 14 can be said to be the VR user's viewpoint in the virtual space 11.

The uvw visual field coordinate system of the VR device 210 is equal to the viewpoint coordinate system when the VR user views the display 211 . The uvw visual field coordinate system of the virtual camera 14 is linked to the uvw visual field coordinate system of the VR device 210. Therefore, the line of sight of the VR user detected by the gaze sensor 212 can be regarded as the line of sight of the VR user in the uvw visual field coordinate system of the virtual camera 14.

(Visual area)
The viewing area 15 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram showing a YZ cross section of the viewing area 15 viewed from the X direction in the virtual space 11. FIG. 6 is a diagram showing an XZ cross section of the viewing area 15 viewed from the Y direction in the virtual space 11.

As shown in FIG. 5, the viewing area 15 in the YZ cross section includes a region 18. The region 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 301 defines a range including the polar angle α centered on the reference line of sight 16 in the virtual space as a region 18 .

As shown in FIG. 6, the viewing area 15 in the XZ cross section includes a region 19. The region 19 is defined by the position of the virtual camera 14, the reference line of sight 16, and the XZ cross section of the virtual space 11. The processor 301 defines a range including the azimuth angle β centered on the reference line of sight 16 in the virtual space 11 as a region 19 . The polar angles α and β are determined depending on the position of the virtual camera 14 and the inclination (orientation) of the virtual camera 14.

The VR system 200 provides the VR user with a view in the virtual space 11 by displaying the view image 17 on the display 211 based on a signal from the computer 300 (see FIG. 4). The visibility image 17 is an image corresponding to a portion of the panoramic image 13 that corresponds to the visibility area 15. When the VR user moves the VR device 210 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the viewing area 15 in the virtual space 11 changes. Thereby, the view image 17 displayed on the display 211 is updated to an image of the panoramic image 13 that is superimposed on the view area 15 in the direction toward which the VR user faces in the virtual space 11. The VR user can visually recognize a desired direction in the virtual space 11.

While wearing the VR device 210, the VR user can view only the panoramic image 13 developed in the virtual space 11 without viewing the real world. Therefore, the information processing system 100 can provide the VR user with a highly immersive feeling in the virtual space 11.

Note that the virtual camera 14 may include two virtual cameras, that is, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In this case, appropriate parallax is set for the two virtual cameras so that the VR user can recognize the three-dimensional virtual space 11. Note that the virtual camera 14 may be realized by one virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by one virtual camera.

(controller)
An example of the controller 280 will be described with reference to FIG. 7.

As shown in FIG. 7, the controller 280 may include a right controller 280R and a left controller (not shown). In this case, the right controller 280R is operated with the VR user's right hand. The left controller is operated with the VR user's left hand. In one aspect, the right controller 280R and the left controller are configured as separate devices. Therefore, the VR user can freely move the right hand holding the right controller 280R and the left hand holding the left controller. In another aspect, controller 280 may be an integrated controller that accepts operation with both hands. The right controller 280R will be explained below.

The right controller 280R includes a grip 281, a frame 282, a top surface 283, buttons 284 and 285, a sensor (for example, a motion sensor) 286, an infrared LED 287, buttons 288 and 289, and an analog stick 290. Equipped with Grip 281 is configured to be held by the VR user's right hand. For example, the grip 281 may be held by the VR user's right hand palm and three fingers (middle finger, ring finger, little finger).

The button 284 is arranged on the side surface of the grip 281 and accepts operation by the middle finger of the right hand. Button 285 is arranged on the front surface of grip 281 and accepts operation by the index finger of the right hand. Further, the buttons 284 and 285 are provided with switches that detect the movement of the user who presses the buttons 284 and 285. Note that the buttons 284 and 285 may be configured as trigger-type buttons.

The sensor 286 is built into the housing of the grip 281. Additionally, the sensor 286 detects the movement of the VR user. Specifically, sensor 286 detects the movement of the VR user's hand. For example, the sensor 286 detects the rotation speed, number of rotations, etc. of the hand. Note that the controller 280 does not need to include the sensor 286.

A plurality of infrared LEDs 287 are arranged on the frame 282 along its circumferential direction. The infrared light LED 287 emits infrared light while a program using the controller 280 is being executed in accordance with the progress of the program. The infrared light emitted from the infrared light LED 287 can be used to detect the positions and postures (tilt, orientation) of the right controller 280R and the left controller.

The top surface 283 includes buttons 288 and 289 and an analog stick 290. Buttons 288 and 289 accept operations by the VR user's right thumb. Further, the buttons 288 and 289 are provided with switches that detect the movement of the user who presses the buttons 288 and 289. The analog stick 290 accepts operation in any direction within 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object placed in the virtual space 11. Further, the analog stick 290 includes a sensor that detects the movement of the user who operates the analog stick 290.

Information regarding the VR user's input operation (in other words, the VR user's movement) detected by the controller 280 is sent to the computer 300.

As shown in FIG. 7, the controller 280 is defined with yaw, roll, and pitch directions, for example, with respect to the VR user's right hand. Specifically, for example, when a VR user extends his thumb and index finger, the direction in which the thumb extends is the yaw direction, the direction in which the index finger extends is the roll direction, and a plane defined by the axis of the yaw direction and the axis of the roll direction. The direction perpendicular to is defined as the pitch direction.

(Configuration of MR system)
As shown in FIG. 3, the MR system 400 includes an MR device 410, a computer 500, a detection device 460, a display 470, and a controller 480. MR device 410 includes a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417. Hereinafter, the user who uses the MR device 410 will be referred to as the MR user 6.

The computer 500 can be connected to the Internet or other networks 2, and can communicate with, for example, the server 600, the computer of the VR system 200, and other computers connected to the network 2. Examples of other computers include computers of other MR systems 400, external equipment 700, and the like.

The MR device 410 is attached to the head of the MR user 6 and can provide the MR user 6 with a mixed reality space during operation. The MR device 410 may be, for example, a glasses-type device (for example, so-called MR glasses) including a display. Further, the MR device 410 may be a contact lens type device or the like.

The display 411 is realized, for example, as a transmissive display device. Note that the transmissive display 411 may temporarily function as a non-transmissive display device by adjusting its transmittance. The display 411 is arranged on the main body of the MR apparatus 410, for example, so as to be located in front of both eyes of the MR user 6. Therefore, the MR user 6 is provided with a display in which the virtual object displayed by the display 411 is superimposed on the real space visible through the transparent display 411. That is, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which virtual objects are placed in the real space. In other words, the MR device 410 allows the MR user 6 to simultaneously view a virtual object and an object existing in real space (hereinafter referred to as a "real object").

Detection device 460 detects the movement of MR user 6 using MR device 410. The detection device 460 may have a position tracking function for detecting the movement of the MR device 410, thereby detecting the movement of the MR user 6. Specifically, the detection device 460 has a sensor that reads light (for example, infrared light) from the MR device 410 as a sensor for detecting the movement of the MR user 6, and detects the movement of the MR user 6 in the mixed reality space. It may also be a device that detects the position, inclination, etc. of the MR device 410. In this case, the MR device 410 may include a plurality of light sources (not shown). Further, each light source may be realized by, for example, an LED (Light Emitting Diode) that emits infrared light.

Further, the detection device 460 may be realized by, for example, a camera. Specifically, the detection device 460 includes an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor) as a sensor for detecting the movement of the MR user 6. It may be something. In other words, the movement of the MR user 6 may be detected by the camera. The detection device 460 is, for example, a depth camera that includes a device that emits predetermined light such as infrared light or light with a predetermined pattern, and an image sensor (for example, a depth sensor), and detects the light emitted from the device. The image sensor may detect the reflected light of the image sensor, and the position, posture, etc. of the MR user 6 may be detected based on the output from the image sensor. Further, the detection device 460 includes such a depth camera and a camera capable of acquiring RGB images, and detects the position, posture, etc. of the MR user 6 based on the output from these cameras. It may be. Further, the detection device 460 may be, for example, a stereo camera including a plurality of image sensors, and may be one that detects the position, posture, etc. of the MR user 6 based on outputs from the plurality of image sensors. . Note that the detection of the position, posture, etc. of the MR user 6 may be the detection of the body position, posture, etc. of the MR user 6, or may be the detection of the position, tilt, etc. of the MR device 410.

Note that the MR system 400 may include one or more types of detection devices as the detection device 460, or may include a plurality of each type of detection device. For example, the MR system 400 may include multiple cameras (in other words, multiple sensors), and the multiple cameras may function as the detection device 460. In this embodiment, the MR system 400 will be described as including three detection devices 460, and each detection device 460 includes a depth camera and a camera capable of acquiring RGB images. Note that a part of the detection device 460 may be configured by the computer 500 or the like. For example, analysis of the output from the image sensor (eg, image recognition, etc.) may be performed by the computer 500 or the like. Note that the MR system 400 does not need to include the detection device 460.

In addition to the various sensors described above, the detection device 460 may include a sensor capable of detecting the position, inclination, etc. of the detection device 460 itself. Specifically, for example, the detection device 460 includes an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. You can. Further, the outputs from these sensors may be sent to the computer 500 or the like, and may be used, for example, when predetermined processing is performed based on the outputs from the image sensor included in the detection device 460.

Further, as understood from the configuration of the MR device 410 described later, the MR device 410 of another MR user 6 may be used as the detection device 460.

Further, the MR device 410 may include a sensor 417 instead of or in addition to the detection device 460 as a detection means for detecting the movement of the MR user 6. MR device 410 can use sensor 417 to detect the position and tilt of MR device 410 itself. The sensor 417 may be, for example, an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. Furthermore, the MR device 410 may have one or more types of sensors as the sensor 417, or may have a plurality of each type of sensor.

Gaze sensor 412 detects the direction in which the right and left eyes of MR user 6 are directed. That is, the gaze sensor 412 detects the line of sight (in other words, the movement of the eyes) of the MR user 6. Detection of the direction of the line of sight is realized, for example, by a known eye tracking function. The gaze sensor 412 is realized by a sensor having the eye tracking function. Gaze sensor 412 may include a right eye sensor and a left eye sensor. For example, the gaze sensor 412 may be a sensor that irradiates infrared light to the right and left eyes of the MR user 6 and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. good. In this case, the gaze sensor 412 can detect the line of sight of the MR user 6 based on each detected rotation angle.

The camera 413 photographs the surroundings (for example, in front) of the MR user 6. The MR device 410 may include, for example, a depth camera and a camera capable of acquiring RGB images as the camera 413. Then, based on the output from the camera 413, the computer 500 detects the shapes of objects around the MR device 410 (in other words, the MR user 6), the relative distances between the MR device 410 and the surrounding objects, etc. You can. Further, the position and tilt of the MR device 410 may be detected based on the output from the camera 413. In other words, tracking of the MR device 410 may be performed using an outside-in method or an inside-out method.

The microphone 415 serving as a sound input means converts the voice of the MR user 6 into a sound signal (in other words, an electrical signal) and outputs it to the computer 500. A speaker 416 serving as a sound output means converts the sound signal into sound and outputs it to the MR user 6. Note that the MR device 410 may include earphones instead of the speaker 416 as a sound output means.

Display 470 displays an image similar to the image displayed on display 411, for example. This allows users other than the MR user 6 wearing the MR device 410 to view the same images as the MR user 6. The image displayed on the display 470 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the MR device 410. Examples of the display 470 include a liquid crystal display and an organic EL display. Note that the display 470 may display an image showing a view of the mixed reality space that the MR device 410 provides to the MR user 6.

Controller 480 is connected to computer 500 by wire or wirelessly. Controller 480 accepts input operations related to instructions from MR user 6 to computer 500. Further, the controller 480 accepts input operations by the MR user 6 to control the position and movement of virtual objects placed in the mixed reality space. The controller 480 may be configured to be graspable by the MR user 6, for example. Furthermore, the controller 480 may be configured to be attachable to the body or part of clothing of the MR user 6, for example. Further, the controller 480 may be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 500.

Further, the controller 480 may include multiple light sources. Each light source may be realized by, for example, an LED that emits infrared light. The detection device 460 may read the infrared light from the controller 480 and detect the position, tilt, etc. of the controller 480 in the mixed reality space. In other words, the detection device 460 may have a position tracking function that detects the movement of the controller 480, thereby detecting the movement of the MR user 6.

Note that the MR system 400 does not need to include the controller 480. Moreover, the controller 480 may have part or all of the configuration of the controller 280, for example.

Note that the information processing system 100 may include a sensor 486 that detects the movement of the MR user 6. Sensor 486 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. Further, the sensor 486 may be provided in the controller 480, for example. Note that the information processing system 100 may have one or more types of sensors as the sensor 486, or may have a plurality of each type of sensor. Further, a device including the sensor 486 (for example, the controller 480, a predetermined camera, etc.) and the computer 500 may be connected to each other wirelessly, for example. The information acquired by sensor 486 may then be transmitted to computer 300 via wireless communication, for example.

Further, the position and movement of the MR user 6 can be determined by the MR device 410, which is equipped with a device that performs predetermined wireless communication (for example, short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB communication), and is connected to the device. The VR device 210 may be detected by wirelessly communicating with surrounding devices and acquiring position information of the VR device 210.

In addition, the position and movement of the MR user 6 can be determined by a device worn by the MR user 6 (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). , may be detected using an external device 700 that is capable of tracking the movement of the MR user 6 (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication or the like). Further, these external devices 700 may be used as the controller 480. Furthermore, the position and movement of the MR user 6 may be detected by a GPS sensor or the like.

(Configuration of computer 500)
As shown in FIG. 3, the computer 500 includes a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505 as main components. Each component is connected to each other by a bus.

Processor 501 controls the operation of MR device 410. Processor 501 reads a program from storage 503 and expands it into memory 502. Processor 501 executes the developed program. The processor 501 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 502 is a main storage device. The memory 502 is configured of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory 502 provides a work area for the processor 501 by temporarily storing programs and various data that the processor 501 reads from the storage 503. The memory 502 also temporarily stores various data generated while the processor 501 is operating according to a program, various data input to the computer 500, and the like.

Storage 503 is an auxiliary storage device. The storage 503 is configured of a storage device such as a flash memory or a hard disk drive (HDD), for example. The storage 503 stores programs for providing services in the information processing system 100. Further, the storage 503 stores various data for providing services in the information processing system 100. The data stored in the storage 503 includes data for defining a mixed reality space, data regarding virtual objects, and the like.

Note that the storage 503 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 503 built into the computer 500, programs and data stored in an external storage device may be used.

The input/output interface 504 is an interface through which the computer 500 receives data input, and an interface through which the computer 500 outputs data. Input/output interface 504 may send data to or receive data from MR device 410, detection device 460, and display 470. In addition, the input/output interface 504 may transmit or receive data with a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417 included in the MR apparatus 410.

Note that the input/output interface 504 may transmit or receive data to/from the controller 480. For example, input/output interface 504 may receive input of signals output from controller 480 and sensor 486. Further, the input/output interface 504 may transmit instructions output from the processor 501 to the controller 480. The command may instruct the controller 480 to vibrate, output sound, emit light, or the like. When the controller 480 receives the command, it vibrates, outputs sound, emits light, etc. in accordance with the command.

Note that each of the MR device 410, the detection device 460, the display 470, the controller 480, and the sensor 486 and the computer 500 may be connected by wire or wirelessly.

The communication interface 505 controls transmission and reception of various data via the network 2 with other computers (for example, the server 600, the computer 300, or the other computer 500, etc.).

The processor 501 accesses the storage 503, loads the program stored in the storage 503 into the memory 502, and executes a series of instructions included in the program. Further, the processor 501 sends a signal for providing mixed reality space to the MR device 410 via the input/output interface 504.

Note that the computer 500 may be provided outside the MR apparatus 410, or part or all of the computer 500 may be built into the MR apparatus 410. Furthermore, when there are a plurality of MR apparatuses 410, the computer 500 may be provided for each MR apparatus 410, or may be used in common for the plurality of MR apparatuses 410. Further, in this case, a part of the computer 500 may be built into each of the plurality of MR apparatuses 410. In this embodiment, a case where a plurality of MR apparatuses 410 are connected to a computer 500 will be described as an example.

The method of acquiring the position and inclination of the MR device 410 in the real space (in other words, the mixed reality space) is well known and can be detected by a method similar to the method used in the VR device 210 described above, so a description thereof will be omitted.

(Server configuration)
Server 600 may send programs to computer 300. Additionally, server 600 may send programs to computer 500. Further, the server 600 enables communication between the computer 300 and the computer 500. Additionally, the server 600 enables communication between the computer 300 and other computers 300 . For example, when the information processing system 100 provides a service in which multiple VR users can participate, each computer 300 of the VR users communicates with other computers 300 via the server 600, thereby allowing multiple VR users to participate in the same virtual space. VR users may be able to share their experiences. Note that each computer 300 may communicate with other computers 300 without going through the server 600.

As shown in FIG. 1, the server 600 includes a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605. Each component is connected to each other by a bus.

Processor 601 controls the overall operation of server 600. The processor 601 reads the program from the storage 603 and expands it into the memory 602. Processor 601 executes the developed program. The processor 601 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 602 is a main storage device. The memory 602 is configured by a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 602 provides a work area to the processor 601 by temporarily storing programs and various data that the processor 601 reads from the storage 603. The memory 602 also temporarily stores various data generated while the processor 601 is operating according to a program, various data input to the server 600, and the like.

Storage 603 is an auxiliary storage device. The storage 603 is configured of a storage device such as a flash memory or a hard disk drive (HDD), for example. The storage 603 stores programs for providing services in the information processing system 100. Further, the storage 603 stores various data for providing services in the information processing system 100. The data stored in the storage 603 includes data for defining a virtual space, data for defining a mixed reality space, data regarding virtual objects, and the like.

Note that the storage 603 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 603 built into the server 600, programs and data stored in an external storage device may be used.

The input/output interface 604 is an interface through which the server 600 receives data input, and an interface through which the server 600 outputs data. The input/output interface 604 can send or receive data to/from input devices such as a mouse and keyboard, and output devices such as a display.

The communication interface 605 controls transmission and reception of various data via the network 2 with other computers (for example, the computer 300, the computer 500, etc.).

External device 700 may be any device that can communicate with computer 300, computer 500, or server 600. The external device 700 may be, for example, a device that can communicate with the computer 300 via the network 2, or a device that can communicate with the computer 300 through short-range wireless communication or a wired connection. Furthermore, the external device 700 may be, for example, a device that can communicate with the computer 500 via the network 2, or a device that can communicate with the computer 500 through short-range wireless communication or a wired connection. Further, the external device 700 may be a device that can communicate with the server 600 via the network 2, for example. Examples of the external device 700 include, but are not limited to, a smart device, a PC (Personal Computer), and peripheral devices of the computer 300 and the computer 500.

<Control of information processing system>
In the MR system 400 of this embodiment, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which virtual objects are arranged in the real space, similar to well-known MR devices. Furthermore, in the VR system 200 of this embodiment, the VR device 210 provides a view of the virtual space to the VR user, similar to well-known VR devices. Furthermore, in the MR system 400 of this embodiment, the MR user 6 is able to operate virtual objects in the mixed reality space. Furthermore, in the VR system 200 of this embodiment, the VR user can operate virtual objects in the virtual space. In addition, in the information processing system 100 of this embodiment, the movements of the VR user using the VR system 200 are reflected in the virtual objects included in the view of the mixed reality space provided by the MR system 400. There is. Furthermore, in the information processing system 100 of this embodiment, the movements of the MR user 6 using the MR system 400 are reflected in virtual objects included in the view of the virtual space provided by the VR system 200. . Note that in this embodiment, the movement of the VR user, etc. is reflected in the virtual object in the virtual space and the mixed reality space in real time. Furthermore, in this embodiment, the movement of the MR user 6 and the like is reflected in the virtual object in the virtual space and mixed reality space in real time.

In the following, an example will be described in which the configuration according to the present embodiment is applied to a service (in other words, an application) that enables communication between the MR user 6 and the VR user. Specifically, an MR user 6 who is in a conference room as a predetermined real space and a VR user who is located away from the conference room have a meeting while simultaneously viewing an object such as a product mockup as a virtual object. An example will be described in which the configuration according to the present embodiment is applied to a service that enables the following. Note that the configuration according to this embodiment can also be applied to, for example, a system that allows students to take school classes from home. Furthermore, the configuration according to the present embodiment can also be applied to a system that allows a user to participate in a home party held at a specific house from another location. Note that the application of the configuration according to this embodiment is not limited to these services. Further, in the following description, it is assumed that a plurality of detection devices 460 are installed in advance at predetermined positions in the conference room, as shown in FIG. 8 . Further, the description will be made assuming that three MR users 6, MR user 6A, MR user 6B, and MR user 6C, exist in the conference room. Note that, hereinafter, the avatar object of the VR user will be referred to as the VR avatar 25. Further, the avatar object of the MR user 6 is called an MR avatar 26. Specifically, the avatar object of MR user 6A is called MR avatar 26A, the avatar object of MR user 6B is called MR avatar 26B, and the avatar object of MR user 6C is called MR avatar 26C.

Referring to FIG. 8, the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 The field of view provided to the VR user entering the virtual space 11 will be explained. FIG. 8 is a schematic diagram showing the state of the actual conference room 21 and the state of the virtual space 11 when the MR users 6A, 6B, 6C and the VR user are using the service. The upper part of FIG. 8 shows the state of the actual conference room 21, and the lower part of FIG. 8 shows the state of the virtual space 11. In addition, in FIG. 8, the VR avatar 25 is wearing the VR device 210, but this is to make the explanation easier to understand, and in reality, the VR avatar 25 is wearing the VR device 210 (virtual VR device 210) may not be attached. Further, in FIG. 8, the MR avatar 26 is not wearing the MR device 410 (virtual MR device 410), but the MR avatar 26 may be wearing the MR device 410.

As shown in FIG. 8, a desk 8 and the like as real objects are arranged in the conference room 21. Furthermore, three detection devices 460 are arranged in the conference room 21. Furthermore, three MR users 6A, 6B, and 6C are present in the conference room 21. On the other hand, there are no VR users in the conference room 21. Here, on the display 411 of the MR apparatus 410 used by each of the MR users 6A, 6B, and 6C, a VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as "virtual model 30") are displayed as virtual objects. ) is displayed. That is, in the MR device 410, the VR avatar 25 and the virtual model 30 are displayed in the real conference room 21 that can be seen through the transparent display 411. Therefore, the MR users 6A, 6B, and 6C are provided with a view as if the VR avatar 25 and virtual model 30, which do not actually exist in the conference room 21, were present in the conference room 21.

Further, the virtual space 11 that the VR user views via the VR device 210 imitates the conference room 21 and includes virtual objects 31 such as the desk 8 that exist in the actual conference room 21 . Further, in the virtual space 11, MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, a virtual model 30, etc. are arranged. Therefore, the VR user is provided with a view as if he or she were in the conference room 21 where the MR users 6A, 6B, and 6C are present.

In this way, in the information processing system 100 of the present embodiment, the MR user 6 can visually recognize other MR users 6 and real objects actually existing at the same location through the transparent display 411. Further, from the viewpoint of the MR user 6, the VR avatar 25 of the VR user appearing as a virtual object can be visually recognized. Therefore, the MR user 6 can communicate with the VR user, other MR users 6, etc. without losing the feeling of being in the real world. Further, the VR user is in a different location from the MR user 6 in the real world, and in such a case, the shape of the room etc. is often different between the VR user's location and the MR user 6's location. For this reason, even on the VR user side, if the MR device 410 or the like is used to display the MR avatar 26 in real space, there is a risk that the display will look unnatural. In this embodiment, the VR user can enter the virtual space 11 and communicate with the MR user 6 and the like, so it is possible to prevent such discomfort from occurring. In addition, since the virtual space 11 provided to the VR user is a simulation of the location where the MR user 6 is, when the VR user and the MR user 6 move around in the virtual space 11 or the real space, respectively, or move virtual objects, In addition, the movements of the avatar, virtual model 30, etc. displayed to the other party become natural. Furthermore, when operating a virtual object such as the virtual model 30, the VR user or MR user 6 can easily imagine how it will appear to the other party, thereby facilitating smooth communication.

The functional configuration of the information processing system 100 will be described below with reference to FIG. 9. Note that the functional configuration shown in FIG. 9 is only an example. Each of the VR system 200, the computer 500, and the server 600 may have at least some of the functions that other devices have. In other words, the computer 300, the computer 500, the server 600, or another device may include some or all of the functional blocks included in each of the computer 300, the computer 500, and the server 600 in this embodiment. Further, each device such as the computer 300, the computer 500, and the server 600 does not need to be realized by an integrated device, and may be realized by, for example, a plurality of devices connected via a network or the like. .

Further, in this embodiment, the processor 301, the processor 501, or the processor 601 will be described as performing each process described below by executing a program stored in the information processing system 100. However, at least a portion of the processing to be described later and performed by the processor 301 may be executed by a processor other than the processor 301. Further, at least a portion of the processing to be described later which is performed by the processor 501 may be executed by a processor other than the processor 501. Further, at least a part of the processing to be described later which is performed by the processor 601 may be executed by a processor other than the processor 601. In other words, the computer that executes the program in this embodiment may be any computer including the computer 300, the computer 500, and the server 600, or may be realized by a combination of multiple devices.

FIG. 9 is a block diagram showing the functional configuration of the information processing system 100. As shown in FIG. 9, the computer 300 (in other words, the VR system 200) has a control unit 310, a storage unit 311, functions as Further, the computer 500 (in other words, the MR system 400) functions as a control unit 510 and a storage unit 511 through the cooperation of a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505. Further, the server 600 functions as a control unit 610 and a storage unit 611 through cooperation of a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605.

The control unit 510 of the MR system 400 includes a virtual space generation unit 810, a coordinate definition unit 812, an MR side input reception unit 816, a user information acquisition unit 818, an MR side object control unit 820, and a display control unit 840. , a sound control section 845, and a communication control section 850.

The virtual space generation unit 810 generates a virtual space (in other words, virtual space data representing the virtual space) based on a predetermined real space. In other words, the virtual space generation unit 810 generates a virtual space imitating a predetermined real space. Here, the predetermined real space is a predetermined place that exists in the real world, and may be, for example, a conference room or a specific room such as a school classroom. Further, the predetermined real space may be, for example, a school or a specific building such as a house. Further, the predetermined real space may be, for example, a specific town. In other words, the predetermined real space does not have to be a space partitioned off by a wall or the like. Moreover, imitating a predetermined real space may mean that the generated virtual space has a structure similar to the basic structure of the real space. Specifically, for example, in a virtual space that imitates a conference room in the real world, the shape of the virtual room as a virtual space (for example, the shape of walls, floors, etc.) approximately matches the shape of the conference room in the real world. It may be a virtual object in which virtual objects corresponding to real objects such as desks and chairs existing in a conference room in the real world are arranged. At this time, virtual objects such as desks, chairs, walls, floors, etc. do not need to be exact copies of the form of real objects; for example, they may have simplified shapes, patterns, colors, etc. Good too. Furthermore, for example, real objects may include objects that do not appear in the virtual world (real objects for which there is no corresponding virtual object), such as posters pasted on walls or small items placed on desks, floors, etc. .

In this embodiment, as described above, the predetermined real space is a specific conference room 21, and the virtual space 11 imitating the specific conference room 21 is generated.

As for the method of generating a virtual space imitating a real space, a known method can be used, and the method is not particularly limited, but for example, the following method may be used. That is, the virtual space generation unit 810 generates the virtual space 11 based on information from a sensor that can detect the shapes and positions of real objects such as walls, floors, desks, and chairs that make up the real space. Specifically, the virtual space generation unit 810 generates data from an image sensor included in the detection device 460 or the MR device 410 (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor). A virtual space may be generated by acquiring three-dimensional information of the real space based on the information. Further, the virtual space generation unit 810 may use information from an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, or the like to generate the virtual space 11, for example.

The virtual space generation unit 810 stores data indicating the generated virtual space 11 (hereinafter referred to as “virtual space data”) in the storage unit 611 as a virtual space data storage unit, for example.

Note that the virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in advance before the VR user or MR user 6 starts using the service. Furthermore, the virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in real time while the VR user or MR user 6 is using the service, for example.

The coordinate definition unit 812 defines coordinates in the real space (in other words, coordinates in the mixed reality space 21; hereinafter referred to as "mixed reality coordinates"). Further, the coordinate definition unit 812 defines coordinates of the virtual space 11 (hereinafter referred to as "virtual space coordinates").

Mixed reality coordinates have, for example, three mutually orthogonal axes: x-axis, y-axis, and z-axis. The virtual space coordinates have, for example, three mutually orthogonal axes: an X-axis, a Y-axis, and a Z-axis. For each point in the virtual space 11 (in other words, the virtual conference room) that corresponds to (in other words, indicates the same point) each point in the real space (in other words, the real conference room 21), the coordinate definition unit 812 Each coordinate is defined (in other words, they are associated) so that the mixed reality coordinates indicating each point in the real space and the virtual space coordinates indicating each point in the virtual space have a one-to-one correspondence. The coordinate definition unit 812 defines virtual space coordinates associated with mixed reality coordinates, for example, when the virtual space generation unit 810 generates a virtual space imitating a real space. At this time, the coordinate definition unit 812 may associate the mixed reality coordinates and the virtual space coordinates based on the feature points of the conference room detected by the detection device 460, the MR device 410, or the like, for example. Further, the coordinate definition unit 812 may associate the mixed reality coordinates and the virtual space coordinates based on, for example, the position of the detection device 460 or the position of a marker installed in advance in the conference room.

Further, the coordinate definition unit 812 stores information regarding the correspondence between mixed reality coordinates and virtual space coordinates (in other words, the correspondence between each point in the real space and each point in the virtual space) as a positional relationship storage unit. The information is stored in the storage unit 611.

Note that a known method can be used as a method for defining mixed reality coordinates and virtual space coordinates, and there are no particular limitations.

Note that the timing of defining mixed reality coordinates and the timing of defining virtual space coordinates (in other words, the timing of associating mixed reality coordinates and virtual space coordinates) may be, for example, when the virtual space generation unit 810 generates a virtual The timing may be when a space is generated, or the timing when the display of a virtual object on the MR device 410 is started.

Note that the generation of the virtual space, the definition of each coordinate, etc. are performed in advance by a dedicated device as the external device 700 that can scan the real space with high precision, and the generated data is stored in the storage unit 611. You can also leave it there.

The MR side input receiving unit 816 receives input from the MR user 6. In other words, the MR side input reception unit 816 acquires input information from the MR user 6. Specifically, the MR side input reception unit 816 includes the detection device 460, the controller 480, the camera 413, the device worn by the MR user, the gaze sensor 412, the sensor 417, the sensor 486, and the detection device 460 and the MR device 410. The output of the provided image sensor or the like is accepted as input from the MR user 6. In other words, the MR-side input reception unit 816 uses detection means for detecting the movement of the MR user 6 (for example, the detection device 460, the controller 480, the camera 413, the gaze sensor 412, the sensor 417, the sensor 486, and the detection device 460 or the MR The output from an image sensor (such as an image sensor included in the device 410) is received as an input from the MR user 6. These devices and sensors are for acquiring information regarding the movement of the MR user 6, which is used to control virtual objects in the virtual space 11 and the mixed reality space 21, as will be described later. The user 6 can input information for moving the virtual object etc. to the MR system 400 by his/her own movements.

Specifically, the MR-side input receiving unit 816 may receive, for example, a camera serving as the detection device 460 or an output from the camera 413 as an input from the MR user 6. More specifically, the MR side input reception unit 816 indicates the position of the MR user 6 in the mixed reality space 21, which is detected by image recognition from images taken by image sensors included in each of these cameras, for example. The position information may be received as input from the MR user 6. In other words, the MR side input receiving unit 816 may receive an output from a position detection means for detecting the position of the MR user 6 in the mixed reality space 21 as an input from the MR user 6. Note that the position detecting means may, for example, perform wireless communication (for example, Wi-Fi communication) between the MR device 410 and a predetermined device (for example, the detection device 460, a beacon (not shown), etc.) of the MR system 400. , Bluetooth communication, or UWB communication), or positioning using a GPS (Global Positioning System) sensor included in the MR device 410, to detect the position of the MR user 6 and obtain the position information. etc. may be used. Note that the position information indicating the position of the MR user 6 can also be said to be information regarding the movement of the MR user 6.

Further, the MR-side input receiving unit 816 receives information regarding the movement of the MR user 6 from the MR user 6, which is detected by image recognition from an image taken by a camera as the detection device 460 or an image sensor included in the camera 413, for example. may be accepted as input. For example, the MR side input reception unit 816 receives information regarding the hand movement of the MR user 6 detected by image recognition, information regarding the inclination of the MR user 6, and the direction in which the MR user 6 is facing, as input from the MR user 6. Good too.

Further, the MR side input reception unit 816 may be configured to include, for example, a device worn by the MR user 6 (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). Information regarding the movement of the MR user 6 obtained using a device that is capable of tracking the movement of the MR user 6 (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication, etc.) may be accepted as an input from the MR user 6.

Further, the MR side input reception unit 816 may accept, for example, information indicating the operation of the MR user 6 detected by the controller 480 (in other words, information regarding the movement of the MR user 6) as an input from the MR user 6. good.

The MR side input receiving unit 816 also receives outputs related to the inclination and facing direction of the MR device 410 from the detection device 460 or the sensors (for example, acceleration sensor, angular velocity sensor, geomagnetic sensor, etc.) included in the MR device 410. Data (in other words, information regarding the movements of the MR user 6) may be accepted as input from the MR user 6.

The user information acquisition unit 818 acquires information regarding the MR user 6 existing in the mixed reality space 21. Specifically, for example, the user information acquisition unit 818 acquires identification information that allows each MR user 6A, 6B, and 6C in the mixed reality space 21 to be identified (in other words, identification information that allows each MR device 410 to be identified). ) to obtain. For example, the user information acquisition unit 818 communicates with the MR device 410 worn by each user, and receives information from the MR device 410 about the user using the MR device 410 (for example, who is logged in to the MR device 410? account information, etc.) may be acquired as the identification information. Further, for example, the user information acquisition unit 818 communicates with the MR device 410 worn by each user, and acquires information unique to each MR device 410 that allows each MR device 410 to be identified from the MR device 410. It may be obtained as information. Further, for example, the user information acquisition unit 818 identifies the MR user 6 in the mixed reality space 21 by image recognition from an image captured by an image sensor included in the detection device 460 or the MR device 410, and obtains the identification information. You may obtain it. Further, the user information acquisition unit 818 associates the information acquired by the MR side input reception unit 816 with the identification information. In other words, the user information acquisition unit 818 performs control so that when information regarding the movement of the MR user 6 is used in controlling a virtual object, which will be described later, it is possible to determine which MR user 6's information belongs.

The MR side object control unit 820 controls virtual objects in the mixed reality space 21.

The MR side object control unit 820 arranges a virtual object in the mixed reality space 21. Specifically, the MR side object control unit 820 arranges, for example, the VR user's VR avatar 25, an object to which a predetermined change is given by at least one of the VR user or the MR user 6, and the like. In this embodiment, as shown in FIG. ) are placed in the mixed reality space 21.

For example, the MR-side object control unit 820 places a virtual object corresponding to the virtual object at a position in the mixed reality space 21 that corresponds to a position in the virtual space 11 where the virtual object is placed. Here, the virtual object corresponding to the virtual object, that is, the virtual object in the mixed reality space 21 corresponding to the virtual object in the virtual space 11, is, for example, a virtual object having the same form as the virtual object in the virtual space 11 ( In other words, they may represent the same object) or may have different forms. In this embodiment, a virtual model 30 having the same form as the virtual model 30 in the virtual space 11 is arranged in the mixed reality space 21 . Further, in the mixed reality space 21, a VR avatar 25 having the same form as the VR avatar 25 in the virtual space 11 is arranged.

Note that the arrangement of the VR avatar 25 in the mixed reality space 21 can be said as follows. That is, it can be said that the MR-side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR user in the virtual space 11. Here, the position of the VR user in the virtual space 11 is, for example, the position of the VR avatar 25 in the virtual space 11. In addition, in content where the VR user enters the virtual space 11 from a first-person perspective, if the VR user cannot see himself (VR avatar 25) (specifically, if only a part of the hand, etc. However, even in such a case, the VR user in the virtual space 11 recognized by the computer 300 can be said to be the VR avatar 25, and the position of the VR user in the virtual space 11 recognized by the computer 300 is This can be said to be the position of the VR avatar 25 in the virtual space.

Note that the appearance (in other words, form) of the VR avatar 25 may or may not imitate the appearance of the VR user. For example, the appearance of the VR avatar 25 may be one that imitates the appearance of a predetermined animal. Further, the VR avatar 25 placed in the virtual space 11 and the VR avatar 25 placed in the mixed reality space 21 may have different appearances. According to such a configuration, the VR user can display his/her own avatar in the mixed reality space 21 while keeping the form of his or her avatar in the virtual space 11 hidden. Note that the appearance of the VR avatar 25 placed in the virtual space 11 may be the appearance visually recognized by the VR user who operates the VR avatar 25, and the appearance of the VR avatar 25 placed in the virtual space 11 may be the appearance visually recognized by the VR user who operates the VR avatar 25. It may also be an appearance that is visible to other VR users operating other VR avatars 25 that exist in the VR avatar 25 .

Further, the MR-side object control unit 820 may arrange the virtual object in the mixed reality space 21 as follows, for example.

That is, for example, when placing the VR avatar 25, the MR-side object control unit 820 arranges the mixed reality space 21 based on position information indicating the position of the VR user (in other words, the VR avatar 25) in the virtual space 11. Decide where to place it inside. Specifically, for example, when the VR avatar 25 is placed in the virtual space 11, the VR-side object control unit 920, which will be described later, determines the position of the VR avatar 25 in the virtual space 11 (for example, coordinate values of virtual space coordinates). The information is transmitted to the MR side object control unit 820 via the server 600. Based on the information, the MR side object control unit 820 determines the position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11 (for example, the mixed reality corresponding to the coordinate values of the virtual space coordinates). The VR avatar 25 is placed at the coordinate value position). As a result, the VR avatar 25 is placed at the same position in the conference room in the virtual space 11 and the mixed reality space 21.

In other words, when placing a virtual object (for example, the VR avatar 25 or the virtual model 30, etc.) in the mixed reality space 21, the MR side object control unit 820 controls the virtual object ( For example, the position in the mixed reality space 21 to place the VR avatar 25 or the virtual model 30, etc.) is determined based on the position information indicating the position (for example, the position information transmitted by the VR-side object control unit 920). You may.

Further, for example, when placing a virtual object in the mixed reality space 21, the MR side object control unit 820 determines which position in the mixed reality space 21 it should be placed based on information detected by a predetermined sensor of the MR system 400. You may decide where to place it. Specifically, for example, a predetermined marker is installed in advance in the real space, and the MR side object control unit 820 places the marker at the position photographed by the image sensor included in the detection device 460 or the MR device 410. A virtual model 30 as a virtual object may be arranged. Further, the MR-side object control unit 820 may place the virtual model 30 as a virtual object at a position specified by the MR user 6. The instruction may be given by a predetermined gesture or the like (for example, a gesture pointing to a point where the virtual model 30 is to be placed), or by placing the marker. Further, the MR side object control unit 820 detects a flat surface in the mixed reality space 21 from an image captured by an image sensor included in the detection device 460 or the MR device 410, and arranges the virtual model 30 on the flat surface. You may do something like this.

Note that when placing a virtual object in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the virtual object, and places the virtual object based on the object data. Here, the MR-side object control unit 820 may receive object data from, for example, the control unit 610 of the server 600 or the VR-side object control unit 920 of the VR system 200. Specifically, when arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the VR avatar 25 from the control unit 610, and receives the object data indicating the form of the object data. The VR avatar 25 may be placed in the mixed reality space 21. Furthermore, when arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the VR avatar 25 from the VR side object control unit 920, and receives the form indicated by the object data. The VR avatar 25 may be placed in the mixed reality space 21.

Display control unit 840 controls image display on display 411 of MR apparatus 410. The display control unit 840 generates an image for displaying the virtual object in the mixed reality space 21 placed by the MR side object control unit 820 at the position placed by the MR side object control unit 820. Furthermore, the display control unit 840 causes the display 411 to display the image. As a result, the MR user 6 is provided with a view of the mixed reality space in which the virtual object is placed at a desired position in the real space.

Further, the display control unit 840 controls image display on the display 470. For example, the display control unit 840 causes the display 470 to display an image similar to the image displayed on the display 411. Note that the display control unit 840 may cause the display 470 to display an image showing a view of the mixed reality space that the MR device 410 provides to the MR user 6.

When the utterance of the MR user 6 is detected by the microphone 415 of the MR device 410, the sound control unit 845 acquires sound data corresponding to the utterance. Further, the sound control unit 845 transmits the acquired sound data to the computer 300 etc. via the network 2. Furthermore, upon receiving sound data from the computer 300 via the network 2, the sound control unit 845 outputs sound (speech) corresponding to the sound data from the speaker 416. This allows the MR user 6 to communicate with the VR user by telephone, for example. Note that the MR-side input receiving unit 816 may receive sound data related to the utterance of the MR user 6 detected by the microphone 415 as an input from the MR user 6.

The communication control unit 850 can communicate with the server 600, the computer 300, and other information communication devices via the network 2. Communication control unit 850 transmits information used by server 600 or computer 300 to server 600 or computer 300, for example. Further, the communication control unit 850 receives information used by the computer 500 from the server 600 or the computer 300, for example.

The control unit 310 of the VR system 200 includes a VR side input reception unit 916, a VR side object control unit 920, a virtual camera control unit 930, a display control unit 940, a sound control unit 945, a communication control unit 950, Equipped with

The VR side input reception unit 916 receives input from a VR user. In other words, the VR side input reception unit 916 acquires input information from the VR user. Specifically, the VR side input reception unit 916 includes the detection device 260, the controller 280, the first camera 213, the second camera 214, the wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and The output of the detection device 260, the image sensor, etc. included in the VR device 210 is accepted as input from the VR user. In other words, the VR side input reception unit 916 uses detection means for detecting the movement of the VR user (for example, the detection device 260, the controller 280, the first camera 213, the second camera 214, the wearable device worn by the VR user, the gaze The outputs of the sensors 212, 217, 286, image sensors included in the detection device 260 and the VR device 210, etc.) are accepted as inputs from the VR user. As described later, these devices and sensors are for acquiring information regarding the movements of the VR user, which is used to control virtual objects, etc. in the virtual space 11 and the mixed reality space 21. It is possible to input information for moving the virtual object etc. to the VR system 200 by the user's own movements.

Specifically, the VR-side input reception unit 916 may accept, for example, an output from a camera as the detection device 260 or a camera included in the VR device 210 that photographs the surroundings of the VR device 210 as an input from a VR user. good. More specifically, the VR side input reception unit 916 inputs position information indicating the position of the VR user in real space, which is detected by image recognition from images taken by image sensors included in each of these cameras, to the VR user. It may also be accepted as input from In other words, the VR-side input reception unit 916 may receive an output from a position detection means that detects the position of the VR user in real space as an input from the VR user. Note that the position detection means, for example, performs wireless communication (for example, Wi-Fi communication) between the VR device 210 and a predetermined device (for example, the detection device 260, a beacon (not shown), etc.) of the VR system 200. , Bluetooth communication, or UWB communication), or one that detects the position of the VR user based on positioning using a GPS (Global Positioning System) sensor included in the VR device 210, and acquires the position information. It may be. Note that the position information indicating the position of the VR user can also be said to be information regarding the movement of the VR user.

Further, the VR-side input reception unit 916 may be configured to receive a VR image detected by image recognition from an image captured by a camera as the detection device 260 or an image sensor included in a camera that captures the surroundings of the VR device 210 included in the VR device 210. Information regarding the user's movements may be accepted as input from the VR user. For example, the VR-side input reception unit 916 may receive information regarding the movement of the VR user's hand detected by image recognition, or information regarding the tilt or direction of the VR user as input from the VR user.

Further, the VR-side input reception unit 916 is configured to input, for example, a device worn by a VR user (for example, a wearable device such as a watch, wristband, ring, or clothing type, or a device implanted in the body). , information regarding the user's movement obtained using a device that is capable of tracking the movement of the VR user (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication, etc.) is transmitted to the VR user. It may also be accepted as input from

Further, the VR-side input receiving unit 916 may receive, for example, information indicating the VR user's operation detected by the controller 280 (in other words, information regarding the VR user's movement) as an input from the VR user.

Further, the VR side input reception unit 916 may output, for example, an output from the detection device 260 or a sensor (for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) included in the VR device 210 regarding the inclination or facing direction of the VR device 210. Data (in other words, information regarding the movements of the VR user) may be accepted as input from the VR user.

The control unit 310 controls the virtual space 11. The control unit 310 acquires virtual space data indicating the virtual space 11 stored in the storage unit 611, and defines the virtual space 11 to be provided to the VR user based on the virtual space data.

The VR-side object control unit 920 controls virtual objects in the virtual space 11.

The VR-side object control unit 920 places a virtual object in the virtual space 11 indicated by the virtual space data. Object data indicating a virtual object is stored in, for example, the storage unit 611 or the storage unit 311 as an object data storage unit. Then, the VR-side object control unit 920 places the virtual object in the virtual space 11 using the object data.

Specifically, the VR-side object control unit 920 controls, for example, the MR avatars 26A, 26B, and 26C of the MR users 6A, 6B, and 6C, and at least one of the VR users or the MR users 6 to A virtual object etc. that can be given a change is placed. In the present embodiment, the VR-side object control unit 920 controls a mock-up virtual object 30 (hereinafter also referred to as "virtual model 30") that is moved by the VR avatar 25, the MR avatar 26, the VR user, and the MR user 6. ) are placed in the virtual space 11. Note that virtual objects 31 and the like corresponding to the desk 8 as a real object are also arranged in the virtual space 11.

Note that, for example, when the VR system 200 (in other words, computer 300) used by a VR user is connected to the VR system 200 (in other words, computer 300) used by another VR user via the network 2, the VR The side object control unit 920 may arrange the VR avatar 25 of the other VR user in the virtual space 11. The other VR user's VR avatar 25 can be operated by the other VR user.

The VR-side object control unit 920 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21, for example. That is, the VR-side object control unit 920 determines the positions of the MR users 6A, 6B, and 6C in the real conference room 21, and the positions of the MR avatars 26A, 26B, and 26C in the virtual conference room. The MR avatars 26A, 26B, and 26C are arranged so that they are at the same position.

Note that the appearance (in other words, form) of the MR avatar 26 may or may not be an imitation of the appearance of the MR user 6. For example, the appearance of the MR avatar 26 may be one that imitates the appearance of a predetermined animal. When placing a virtual object in the mixed reality space 21, the VR side object control unit 920 receives object data indicating the form of the virtual object, and places the virtual object based on the object data. Here, the VR-side object control unit 920 may receive object data from the control unit 610 of the server 600 or from the MR-side object control unit 820 of the MR system 400. Specifically, for example, when placing the MR avatar 26 in the virtual space 11, the VR side object control unit 920 receives object data indicating the form of the MR avatar 26 from the control unit 610 or the MR side object control unit 820. After receiving the object data, the MR avatar 26 having the form indicated by the object data may be placed in the virtual space 11. The appearance of the MR avatar 26 can be set in advance by the MR user 6, and when the MR avatar 26 is displayed on the VR device 210, it is displayed with the appearance set by the MR user 6. It may be possible to do so. Further, when the appearance of the MR avatar 26 is imitated to the appearance of the MR user 6, the VR-side object control unit 920 detects the MR user's appearance obtained by a predetermined camera such as the camera as the detection device 460 or the camera 413. The MR avatar 26 generated based on the image No. 6 (for example, generated by the control unit 510) may be placed in the virtual space 11.

The virtual camera control unit 930 arranges the virtual camera 14 in the virtual space 11. Further, the virtual camera control unit 930 controls the position of the virtual camera 14 in the virtual space 11 and the inclination (orientation) of the virtual camera 14. Note that in this embodiment, the virtual camera control unit 930 arranges the virtual camera 14 at the eye position of the VR avatar 25 in the virtual space 11. In other words, the virtual camera control unit 930 links the position of the virtual camera 14 with the position of the VR avatar 25, and moves the virtual camera 14 within the virtual space 11 when the VR avatar 25 moves within the virtual space 11. Note that the position of the virtual camera 14 does not need to be linked to the position of the VR avatar 25. For example, it may be possible to move the VR avatar 25 within the virtual space 11 while keeping the virtual camera 14 fixed at a predetermined position.

The display control unit 940 controls image display on the display 211 of the VR device 210. The display control unit 940 defines the viewing area 15 according to the position and tilt of the virtual camera 14 (in other words, the tilt of the head of the VR user wearing the VR device 210). Further, the display control unit 940 generates the visual field image 17 displayed on the display 211 based on the defined visual field area 15. The view image 17 generated by the display control unit 940 is output to the VR device 210.

Further, the display control unit 940 controls image display on the display 270. For example, the display control unit 940 causes the display 270 to display an image similar to the image displayed on the display 211.

When the sound control unit 945 detects an utterance from the VR user using the microphone 215 from the VR device 210, it acquires sound data corresponding to the utterance. Further, the sound control unit 945 transmits the acquired sound data to the computer 500, the computer 300 of another VR user, etc. via the network 2. Furthermore, upon receiving sound data from the computer 500 or another user's computer 300 via the network 2, the sound control unit 945 outputs sound (speech) corresponding to the sound data from the speaker 216. This allows the VR user to communicate with the MR user 6 by telephone, for example. Note that the VR-side input reception unit 916 may accept sound data related to the VR user's utterances detected by the microphone 215 as input from the VR user.

The communication control unit 950 can communicate with the server 600, the computer 500, the computer 300 of another VR user, and other information communication devices via the network 2. The communication control unit 950 transmits information used by the server 600, the computer 500, or the other VR user's computer 300 to the server 600, the computer 500, or the other VR user's computer 300, for example. Further, the communication control unit 850 receives information used by the computer 300, for example, from the server 600, the computer 500, or another VR user's computer 300.

(Fusion of mixed reality space and virtual space)
In this embodiment, the virtual object displayed by the MR device 410 and the virtual object displayed by the VR device 210 are controlled based on the input from the MR user 6 acquired by the MR side input reception unit 816. ing.

Specifically, the MR-side object control unit 820 controls the virtual object in the mixed reality space 21, specifically, the virtual model 30, based on the input from the MR user 6 received by the MR-side input reception unit 816. give a change of Here, predetermined changes include complex changes such as moving the virtual object, changing the inclination or direction of the virtual object, and changing the relative positional relationship between the parts of the virtual object. It may also be something that moves a virtual object in the real space 21. Furthermore, the predetermined change may include changing the color of the virtual object, changing information displayed by the virtual object, or changing the form of the virtual object.

More specifically, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on, for example, information regarding the movement of the MR user 6 received by the MR-side input reception unit 816. Specifically, for example, when a gesture of the MR user 6 carrying the virtual model 30 is detected by the detection means (for example, the detection device 460 or the camera 413, etc.), and the detection means outputs an output related to the gesture, the MR side The input receiving unit 816 receives the output as an input from the MR user 6. Further, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on the input received by the MR-side input reception unit 816 and in accordance with the instruction from the MR user 6 using the gesture. Note that the gesture of the MR user 6 that causes a predetermined change to the virtual model 30 as a virtual object does not have to be a gesture that touches the virtual model 30 in the mixed reality space 21 . For example, it may be possible to rotate the virtual object by holding up a hand toward the virtual model 30 and waving the hand sideways.

Further, the MR-side object control unit 820 controls the virtual object based on, for example, information regarding the operation of the MR user 6 on the controller 480 (in other words, information regarding the movement of the MR user 6) received by the MR-side input reception unit 816. It is also possible to give a predetermined change to. Further, the MR-side object control unit 820 may, for example, apply a predetermined change to the virtual object based on the sound data related to the utterance of the MR user 6, which is received by the MR-side input reception unit 816.

Further, the VR-side object control unit 920 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 that the MR-side input reception unit 816 receives.

Specifically, the VR-side object control unit 920 controls, for example, a change similar to the change given to the virtual model 30 in the mixed reality space 21 based on the input from the MR user 6 received by the MR-side input reception unit 816. is given to the virtual model 30 in the virtual space 11. That is, for example, as described above, when the virtual model 30 in the mixed reality space 21 moves due to the gesture of the MR user 6 carrying the virtual model 30, the VR side object control unit 920 moves the virtual model 30 in the virtual space 11. The model 30 is moved in the same manner.

Note that the method of applying a change similar to the change applied to the virtual model 30 in the mixed reality space 21 to the virtual model 30 in the virtual space is not particularly limited, but may be, for example, as follows. . That is, for example, when the MR side object control unit 820 moves the virtual model 30 (in other words, the virtual object) in the mixed reality space 21 based on the input from the MR user 6 (in other words, at a predetermined timing), the VR Information indicating the position and inclination of the virtual model 30 after movement (in other words, information regarding the state of the virtual model 30) may be sent to the side object control unit 920. Then, the VR-side object control unit 920 may arrange the virtual model 30 at a position indicated by the information in the virtual space 11, or may arrange the virtual model 30 at an inclination indicated by the information. For example, the communication control unit 850 may also transmit information based on input from the MR user 6 (for example, the amount of movement, movement trajectory, and inclination of the virtual model 30 calculated (in other words, acquired) by the control unit 510 based on the input). etc.) to the communication control unit 950, and the VR-side object control unit 920 may move a virtual object such as the virtual model 30 based on the information. That is, in the present embodiment, the VR side object control unit 920 gives a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 accepted by the MR side input reception unit 816. "Based on the input from the MR user 6" does not mean that the VR side object control unit 920 itself receives the input and controls the virtual object, but the virtual object in the virtual space 11 responds to the input as a result. Any material that makes a predetermined change may be used.

Further, the VR-side object control unit 920 moves the MR avatar 26 in the virtual space 11 based on the input from the MR user 6 received by the MR-side input reception unit 816, for example. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information regarding the movement of the MR user 6 received by the MR-side input reception unit 816. For example, the VR-side object control unit 920 controls the MR avatar 26 based on position information indicating the position of the MR user 6 in the mixed reality space 21, which the MR-side input reception unit 816 receives as input from the MR user 6. move. That is, the VR side object control unit 920 places the MR avatar 26 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21 based on the position information, and the MR user 6 When moving in the real space 21, the MR avatar 26 is placed at a position in the virtual space 11 corresponding to the position after movement.

Further, the VR side object control unit 920 also controls the movement of each part of the body of the MR user 6 (for example, hand movement, head movement) that the MR side input reception unit 816 receives as input from the MR user 6. The MR avatar 26 is moved based on information regarding the MR user 6's inclination and the direction in which the MR user 6 is facing. That is, the VR-side object control unit 920 controls the movement of each part of the body of the MR user 6 (in other words, the posture), the inclination and direction of the MR user 6 in the virtual space, based on these pieces of information. Reflect it on MR avatar 26 in 11. That is, in the present embodiment, the VR-side object control unit 920 determines that the movement of the MR avatar 26 in the virtual space 11 imitates the movement of the MR user 6 detected by the detection means that detects the movement of the MR user 6. The movement of the MR avatar 26 is controlled so that the movement of the MR avatar 26 is controlled. In other words, the VR-side object control unit 920 causes the movement of the MR avatar 26 to be linked to the movement of the MR user 6 for at least part of the body (for example, hands, feet, head, eyes, mouth, etc.). The movement of the MR avatar 26 is controlled as follows.

Note that at least part of the movement of the MR avatar 26 may be realized by the MR user 6's operation on the controller 480 (for example, operation on an analog stick, button, etc.). That is, for example, the VR side object control unit 920 may move a part of the body of the MR avatar 26 based on information indicating the operation of the MR user 6 on the controller 480 as an input from the MR user 6. good.

Note that at least one of the position or movement of the MR avatar 26 in the virtual space 11 may not be linked to the position or movement of the MR user 6 in the mixed reality space 21. That is, for example, even if the MR user 6 moves in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move accordingly. Further, for example, even if the MR user 6 moves his hand in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not make the same movement.

Furthermore, in this embodiment, the virtual object displayed by the VR device 210 and the virtual object displayed by the MR device 410 are controlled based on the input from the VR user acquired by the VR side input reception unit 916. It has become.

Specifically, the VR-side object control unit 920 causes the virtual object in the virtual space 11, specifically, the virtual model 30, to undergo a predetermined change based on the input from the VR user that the VR-side input reception unit 916 has received. give.

More specifically, the VR-side object control unit 920, for example, based on information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movement) received by the VR-side input receiving unit 916. , gives a predetermined change to the virtual model 30 in the virtual space 11. Specifically, for example, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 in accordance with an instruction from the VR user by operating the controller 280.

Further, the VR-side object control unit 920 may, for example, give a predetermined change to the virtual object based on the gesture of the VR user detected by the detection means. Further, the VR-side object control unit 920 may, for example, give a predetermined change to the virtual object based on the sound data related to the VR user's utterances, which is received by the VR-side input reception unit 916.

Furthermore, the MR-side object control unit 820 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user that the VR-side input reception unit 916 has received.

Specifically, the MR side object control unit 820, for example, causes a change similar to the change given to the virtual model 30 in the virtual space 11 based on the input from the VR user accepted by the VR side input reception unit 916. It is given to the virtual model 30 in the mixed reality space 21. That is, for example, as described above, when the virtual model 30 in the virtual space 11 is moved by the VR user's operation on the controller 280, the MR side object control unit 820 moves the virtual model 30 in the mixed reality space 21 to the same position. Move it like this.

Note that there is no particular limitation on the method of applying the same change to the virtual model 30 in the virtual space 11 to the virtual model 30 in the mixed reality space 21, but for example, the following method may be used. good. That is, for example, when the VR side object control unit 920 moves the virtual model 30 (in other words, a virtual object) in the virtual space 11 based on an input from a VR user (in other words, at a predetermined timing), the MR side object Information indicating the position and inclination of the virtual model 30 after movement (in other words, information regarding the state of the virtual model 30) may be sent to the control unit 820. Then, the MR side object control unit 820 may arrange the virtual model 30 at a position indicated by the information in the mixed reality space 21, or may arrange the virtual model 30 at an inclination indicated by the information. For example, the communication control unit 950 may also transmit information based on input from the VR user (for example, the amount of movement, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 310 based on the input). The MR-side object control unit 820 may move a virtual object such as the virtual model 30 based on the information. That is, in the present embodiment, the MR side object control unit 820 gives a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user accepted by the VR side input reception unit 916. "Based on the input from the VR user" does not mean that the MR side object control unit 820 itself receives the input and controls the virtual object, but the virtual object in the mixed reality space 21 responds to the input as a result. Any material that makes a predetermined change may be used.

Furthermore, the VR-side object control unit 920 gives a predetermined change to the VR avatar 25 in the virtual space 11 based on the input from the VR user that the VR-side input reception unit 916 has received. Specifically, the VR-side object control unit 920, for example, based on information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movement) received by the VR-side input receiving unit 916, A predetermined change is given to the VR avatar 25 in the virtual space 11. Specifically, for example, the VR-side object control unit 920 moves the VR avatar 25 in the virtual space 11 in accordance with an instruction from the VR user by operating the controller 280.

In addition, the VR-side object control unit 920 also controls the movement of each part of the VR user's body (for example, hand movement, head movement, The VR avatar 25 is moved based on information regarding eye movements, changes in facial expressions, etc., and information regarding the tilt and direction of the VR user. That is, the VR-side object control unit 920 controls the movement of each part of the VR user's body (in other words, the posture), the inclination of the VR user, and the direction in which the VR user is facing in the virtual space 11 based on each of these pieces of information. reflected on the VR avatar 25. That is, in the present embodiment, the VR-side object control unit 920 makes the movement of the VR avatar 25 in the virtual space 11 imitate the movement of the VR user detected by the detection means that detects the movement of the VR user. Then, the movement of the VR avatar 25 is controlled. In other words, the VR-side object control unit 920 controls the movement of the VR avatar 25 to be linked to the movement of the VR user for at least part of the body (for example, hands, feet, head, eyes, mouth, etc.). Then, the movement of the VR avatar 25 is controlled.

Further, the MR-side object control unit 820 may, for example, make changes similar to changes given to the VR avatar 25 in the virtual space 11 based on the input from the VR user received by the VR-side input reception unit 916 in the mixed reality space. Give to VR avatar 25 out of 21. That is, for example, as described above, when the VR avatar 25 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR side object control unit 820 moves the VR avatar 25 in the mixed reality space 21 to the same position. Move it like this. Note that there is no particular limitation on the method of applying the same change to the VR avatar 25 in the virtual space 11 to the VR avatar 25 in the mixed reality space 21, but for example, the above-mentioned virtual model 30 may be used. You can do it as well.

Note that the position of the VR avatar 25 in the virtual space 11 and the position in the mixed reality space 21 may move as the VR user moves in the real space. That is, for example, the VR-side object control unit 920 may move the VR avatar 25 based on position information indicating the position of the VR user in real space, which is input from the VR user.

Further, movement of a part of the VR avatar 25 in the virtual space 11 and the VR avatar 25 in the mixed reality space 21 may be realized by operating the controller 280. That is, for example, the VR side object control unit 920 and the MR side object control unit 820 control a part of the body of the VR avatar 25 based on information indicating the VR user's operation on the controller 280, which is input from the VR user. You may also move it.

In this embodiment, the results of actions taken on real objects that actually exist in the conference room 21 as a predetermined real space can also be shared between the VR user and the MR user 6. This point will be explained below by taking as an example the control of an object (for example, a blackboard, a whiteboard, etc.) on which a predetermined entry can be made as a real object. As shown in FIG. 8, a blackboard 35 as a real object is arranged in the mixed reality space 21, and a virtual object as a virtual object corresponding to the blackboard 35 is arranged in the virtual space 11 at a position corresponding to the blackboard 35. It is assumed that a virtual blackboard 36 is placed.

In this embodiment, the action (in other words, the result of the action) performed by the MR user 6 on the blackboard 35 as a real object in the mixed reality space 21 is reflected on the blackboard 35 in the mixed reality space 21. ing. Further, in this embodiment, the action is reflected on the virtual blackboard 36 in the virtual space 11.

In the mixed reality space 21, the MR user 6 can make predetermined entries on the blackboard 35. Further, the writing operation by the MR user 6 on the blackboard 35 is detected by a detection means for detecting the writing operation (hereinafter referred to as "MR side writing detection means"), and the output from the MR side writing detection means (in other words, information regarding the entry operation) is received by the MR side input reception unit 816 as input from the MR user 6. The MR side entry detection means may be, for example, a detection means capable of detecting the hand movement of the MR user 6. Specifically, the MR side entry detection means may be, for example, a pen-shaped controller as the controller 480. Further, the MR side entry detection means may be a camera as the detection device 460, the camera 413, or the like. Furthermore, the MR side entry detection means may be a device worn by the MR user 6 or the like. Further, the MR side entry detection means may be a sensor provided on the blackboard 35 (for example, a touch sensor, a sensor that reads reflected light of infrared light from a device provided on the blackboard 35 that emits infrared light, etc.). In other words, the blackboard 35 is capable of communicating with the computer 500, and information regarding the writing operation of the MR user 6 detected by the sensor included in the blackboard 35 is sent to the MR side input receiving unit 816 as an input from the MR user 6. It may be accepted.

The writing actions of the MR user 6 are reflected on the blackboard 35 in the mixed reality space 21 . In other words, when the MR user 6 performs a writing operation, the blackboard 35 displays the contents written by the MR user 6 (for example, the picture of the moon in FIG. 10(a), The state changes to displaying a picture of a heart and the words "Sample". For example, the blackboard 35 is an electronic blackboard that electronically displays (in other words, reflects) the contents filled in by the MR user 6, and when the MR user 6 performs a writing operation, the contents are displayed. It may be changed to . Specifically, for example, the writing surface of the blackboard 35 itself serves as a display, and the display may display the written contents. Further, the blackboard 35 may include a projector that projects an image onto the writing surface, and the projector may project the written contents onto the writing surface. In other words, the reflection of the user's actions on the mixed reality space 21 (in other words, the changes in the mixed reality space 21 seen through the MR device 410 caused by the actions of the MR user 6) is caused by some change ( For example, this may be realized by a change in display content). In addition, when displaying the entry electronically, the computer of the blackboard 35 may control the display based on the detection result of the sensor provided on the blackboard, and the computer 500 may control the display based on the detection result of the MR side entry detection means. The display may be controlled based on. That is, for example, the control unit 510 or the like of the computer 500 may control the display of the blackboard 35 based on the input from the MR user 6 received by the MR side input receiving unit 816 (hereinafter referred to as "blackboard display control means"). ). Specifically, the blackboard display control means may control, based on input from the MR user 6, so that the image displayed by the display or projector of the blackboard 35 becomes an image showing the written content. Note that the writing operation of the MR user 6 may be reflected on the blackboard 35 in the mixed reality space 21 by the MR user 6 actually writing letters or pictures on the blackboard 35 using a writing instrument such as chalk. In other words, the action of actually writing characters or pictures on the blackboard 35 with a writing instrument may be detected by the MR side writing detection means as the writing action of the MR user 6.

Further, for example, the MR-side object control unit 820 generates a virtual object (hereinafter referred to as "input object 37") indicating the entry content entered by the MR user 6, and places it at the position of the blackboard 35 in the mixed reality space 21. (See FIG. 10(a)). Specifically, the MR-side object control unit 820 may generate the entry object 37 based on the input from the MR user 6 that the MR-side input reception unit 816 receives. When viewing the blackboard 35 through the MR device 410, the state in which the blackboard 35 displays the entry contents (in other words, the entry object 37) may be visually recognized. In other words, the reflection of the user's actions on the mixed reality space 21 (in other words, the changes in the mixed reality space 21 seen via the MR device 410 caused by the actions of the MR user 6) is based on a predetermined change in the real object. It may be realized by adding a virtual object.

Furthermore, the VR-side object control unit 920 applies a predetermined change to the virtual object corresponding to the real object based on the input from the MR user 6 that the MR-side input reception unit 816 has received. Specifically, as shown in FIG. 10(b), the VR side object control unit 920 causes the input from the MR user 6 to be reflected on the virtual blackboard 36, so that the virtual blackboard 36 reflects the input contents written by the MR user 6. control so that it is displayed. In other words, the VR-side object control unit 920 changes changes regarding the real object (for example, the blackboard 35) in the mixed reality space 21 based on the behavior of the MR user 6 into the virtual object (for example, the blackboard 35) in the virtual space 11 corresponding to the real object. For example, it is reflected on the virtual blackboard 36). Specifically, for example, when information regarding the entry operation of the MR user 6 is input to the MR side input receiving unit 816, the VR side object control unit 920 displays the entry object 38 indicating the entry content entered by the MR user 6. may be placed at the position of the virtual blackboard 36 in the virtual space 11 (see FIG. 10(b)). Further, the VR-side object control unit 920 may control the virtual blackboard 36 to be in a state in which written contents are displayed, for example, by changing the form of the virtual blackboard 36 itself. Further, the virtual blackboard 36 is a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, and the VR-side object control unit 920 causes the virtual blackboard 36 to display the image. You can leave it there. Further, the MR system 400 includes a camera that photographs the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 is a virtual object that displays an image photographed by the camera, and the VR-side object control unit 920 The image may be displayed on the virtual blackboard 36.

Further, in the present embodiment, an action (in other words, a result of the action) performed by the VR user on the virtual blackboard 36 as a virtual object in the virtual space 11 is reflected on the virtual blackboard 36 in the virtual space 11. It has become. Further, in this embodiment, the action is reflected on the blackboard 35 in the mixed reality space 21.

In the virtual space 11, the VR user can make predetermined entries on the virtual blackboard 36. Further, the writing operation on the virtual blackboard 36 by the VR user is detected by a detection means (hereinafter referred to as "VR side writing detection means") that detects the writing operation, and the output from the VR side writing detection means is detected by the VR user. The VR-side input receiving unit 916 receives the input as the input. The VR side entry detection means may be, for example, a detection means capable of detecting the movement of a VR user's hand. Specifically, the VR side entry detection means may be, for example, the controller 280 or the like. Further, the VR side entry detection means may be a camera or the like as the detection device 260. Furthermore, the VR side entry detection means may be a device worn by the VR user.

The VR user's writing actions are reflected on the blackboard 35 in the mixed reality space 21 . In other words, when the VR user performs a writing operation, the blackboard 35 changes to a state where the content written by the VR user is displayed. In other words, the blackboard 35 reflects the input from the VR user accepted by the VR-side input reception unit 916. For example, the input from the VR user may be reflected on the blackboard 35 as follows. That is, for example, the MR-side object control unit 820 may place the entry object 37 indicating the content entered by the VR user at the position of the blackboard 35 in the mixed reality space 21. Further, the blackboard display control means may control, based on input from the VR user, so that the image displayed on the display of the blackboard 35 or the projector becomes an image showing the content written by the VR user. That is, the MR side object control unit 820 or the blackboard display control unit can function as a reflection unit that reflects the input from the VR user received by the VR side input reception unit 916 on the real object in the mixed reality space 21.

Further, the VR user's writing operation is also reflected on the virtual blackboard 36 in the virtual space 11. In other words, when the VR user performs a writing operation, the virtual blackboard 36 changes to a state where the contents written by the VR user are displayed. In other words, the virtual blackboard 36 reflects the input from the VR user accepted by the VR-side input reception unit 916. Specifically, the VR-side object control unit 920 applies a predetermined change to the virtual blackboard 36 based on the input from the VR user that the VR-side input reception unit 916 receives. More specifically, the VR-side object control unit 920 controls the input from the VR user to be reflected on the virtual blackboard 36 so that the virtual blackboard 36 is in a state of displaying the content written by the VR user. Specifically, for example, when information regarding the entry operation of the VR user is input to the VR-side input reception unit 916, the VR-side object control unit 920 inputs the entry object 38 indicating the entry content entered by the VR user, It may be placed at the position of the virtual blackboard 36 in the virtual space 11. Further, the VR-side object control unit 920 may control the virtual blackboard 36 to be in a state in which written contents are displayed, for example, by changing the form of the virtual blackboard 36 itself. Further, the virtual blackboard 36 is a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, and the VR-side object control unit 920 causes the virtual blackboard 36 to display the image. You can leave it there. Further, the MR system 400 includes a camera that photographs the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 is a virtual object that displays an image photographed by the camera, and the VR-side object control unit 920 The image may be displayed on the virtual blackboard 36. In other words, the input from the VR user received by the VR side input reception unit 916 is reflected on the real object in the mixed reality space 21, so that the virtual blackboard 36 changes based on the input from the VR user. It may be.

As described above, in this embodiment, the actions that the MR user 6 performs on the real object and the actions that the VR user performs on the virtual object corresponding to the real object are transferred to the mixed reality space 21 and the virtual object. It is now reflected in space 11. Such reflection of each action on the mixed reality space 21 or the virtual space 11 may be as follows.

That is, for example, when the MR user 6 points to an arbitrary position in the mixed reality space 21 with a predetermined pointer such as a laser pointer, the pointed position (hereinafter referred to as "indicated position") also occurs in the virtual space 11. It may be possible to understand. Specifically, for example, in the mixed reality space 21, the detection device 460, camera 413, etc. of the MR system 400 may function as a pointed position detection means for detecting the pointed position. Information regarding the indicated position detected by the indicated position detection means may be received by the MR side input receiving unit 816 as an input from the MR user 6. Furthermore, the VR-side object control unit 920 may place a virtual object indicating light from the laser pointer at a position in the virtual space 11 that corresponds to the indicated position in the mixed reality space 21. In other words, in displaying the virtual space 11 in the VR device 210, the control unit 310 causes the position in the virtual space 11 corresponding to the indicated position in the mixed reality space 21 to shine in a predetermined color representing the light from the laser pointer. It may be controlled as follows.

Further, for example, when the VR user points to an arbitrary position in the virtual space 11 with a predetermined pointer such as a virtual laser pointer, the pointed position may also be known in the mixed reality space 21. Specifically, for example, when information regarding the operation of the VR user on the controller 280 for operating the virtual laser pointer (in other words, information regarding the movement of the VR user) is input to the VR side input reception unit 916, The VR-side object control unit 920 places a virtual object indicating light from a virtual laser pointer at a position indicated by a VR user's operation (in other words, a position indicated by a virtual laser pointer), etc. You may. In other words, in displaying the virtual space in the VR device 210, the control unit 310 causes the position in the virtual space 11 designated by the virtual VR user's operation to be set to a predetermined color representing the light from the virtual laser pointer. You can also control it so that it shines. In addition, the control unit 510 and the like are configured such that the position in the mixed reality space 21 corresponding to the designated position in the virtual space 11 (specifically, the designated position by the VR user) is a predetermined position representing the light from the virtual laser pointer. It may also be controlled so that it shines in this color. Specifically, the MR side object control unit 820 places a virtual object indicating light from a virtual laser pointer at a position in the mixed reality space 21 that corresponds to the indicated position in the virtual space 11. Good too. Further, for example, when the position indicated by the VR user is a predetermined position on the virtual blackboard 36, the blackboard display control means may control the image displayed by the display of the blackboard 35 or the projector based on the input from the VR user. However, the image may be controlled so that a position corresponding to the predetermined position on the virtual blackboard 36 lights up (that is, an image indicating that the predetermined location is pointed by the laser pointer). Further, for example, the detection device 460, the MR device 410, etc. are equipped with a device that emits laser light, and the control unit 510 controls the device in accordance with input from the VR user received by the VR side input reception unit 916. Alternatively, the laser beam may be applied to a position in the mixed reality space 21 that corresponds to the indicated position in the virtual space 11.

Further, the actions of the MR user 6 or the VR user may be reflected in the mixed reality space 21 and the virtual space 11 in the following manner. That is, for example, in the mixed reality space 21, a device (for example, an electronic piano) capable of inputting or outputting sound as a real object is arranged, and the actions of the MR user 6 or the VR user regarding the device are It may also be reflected in the real space 21 and the virtual space 11. In the following, an electronic piano as a real object is placed in the mixed reality space 21, and a virtual electronic piano as a virtual object corresponding to the electronic piano is placed in the virtual space 11 at a position corresponding to the electronic piano. It is assumed that .

The electronic piano may be able to communicate with the computer 500, for example. When the MR user 6 operates the electronic piano in the mixed reality space 21, information regarding the operation (for example, sound data corresponding to the operation) may be sent to the computer 300 via the computer 500. . Then, the sound control unit 945 may output sound from the speaker 216 based on the information. With this, for example, when the MR user 6 operates a key of the electronic piano (for example, a key for the sound of "C"), the sound corresponding to the operated key (for example, the sound of "C") is output from the speaker 216. It may be possible to do so. In other words, the MR side input reception unit 816 receives the output (for example, sound data corresponding to the operation) from the electronic piano (in other words, a sensor or the like that detects the operation on the keys) serving as a detection means for detecting the operation of the MR user 6. ) as an input from the MR user 6, and the control unit 310 (for example, the sound control unit 945) causes the speaker 216 to output a sound corresponding to the operation of the MR user 6 on the electronic piano based on the input. It may be. Further, the sound control unit 945 may control the direction in which the sound is emitted so that the sound corresponding to the operation on the electronic piano sounds as if it is being output from the virtual electronic piano in the virtual space 11.

Further, when the VR user operates the virtual electronic piano in the virtual space 11, information regarding the operation may be sent to the computer 500. Then, the sound control unit 845 may output sound from the speaker 416 based on the information. With this, for example, when a VR user operates a key of the virtual electronic piano (for example, a key for the sound of "C"), the sound corresponding to the operated key (for example, the sound of "C") is output from the speaker 416. It may be possible to do so. In other words, the VR-side input reception unit 916 receives the output from the detection means (for example, the detection device 260 or the controller 280) that detects the VR user's operation on the virtual electronic piano as an input from the VR user, and the control unit 510 ( For example, the sound control unit 845) may be configured to cause the speaker 416 to output a sound corresponding to the VR user's operation on the virtual electronic piano based on the input. Note that the control unit 510 may be configured to cause the speaker of the electronic piano to output a sound corresponding to the VR user's operation on the virtual electronic piano.

Note that when the MR user 6 operates the electronic piano, the sound corresponding to the operation may be outputted in the mixed reality space 21 by the speaker of the electronic piano or by the speaker 416. Furthermore, when the VR user operates the virtual electronic piano, the speaker 216 may output a sound corresponding to the operation in the virtual space 11.

<Processing executed by the information processing system>
Next, the flow of processing executed by the information processing system 100 will be described with reference to FIGS. 11 to 14.

An example of processing executed by the VR system 200 will be described with reference to FIG. 11.

In step S101, the processor 301 of the computer 300 identifies virtual space data and defines the virtual space 11.

In step S102, the processor 301 places the VR user's VR avatar 25 in the virtual space 11.

In step S103, the processor 301 receives information from the mixed reality space 21 from the processor 501 of the computer 500, which is connected to the computer 300 via the network 2 and controls the mixed reality space 21 corresponding to the virtual space 11. The location information indicating the location of the MR user 6 inside is acquired.

In step S104, the processor 301 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21, based on the position information acquired in step S103. do.

In step S105, the processor 301 arranges the virtual model 30 as a virtual object in the virtual space 11. The processor 301 places the virtual model 30 at a position in the virtual space 11 that corresponds to the position of the virtual model 30 in the mixed reality space 21 . Note that arranging at corresponding positions may mean that the position in the mixed reality space 21 is determined in advance, or the position in the virtual space 11 may be determined in advance.

In step S106, the processor 301 receives input from the VR user. Specifically, the processor 301 receives, for example, the output of a detection means for detecting the movement of the VR user as an input from the VR user.

In step S107, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on input from the VR user. Specifically, the processor 301 moves the VR avatar 25 or the virtual model 30 in the virtual space 11 based on input from a VR user, for example. For example, if the input is related to the VR avatar 25, the processor 301 moves the VR avatar 25. Further, for example, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30. Additionally, the processor 301 transmits information regarding input from the VR user to the processor 501.

In step S108, the processor 301 acquires information regarding the input received by the processor 501 from the MR user 6.

In step S109, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 obtained in step S108. Specifically, the processor 301 moves the MR avatar 26 or the virtual model 30 in the virtual space 11 based on input from the MR user 6, for example. For example, if the input is related to the MR avatar 26, the processor 301 moves the MR avatar 26. Furthermore, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30.

Next, an example of processing executed by the MR system 400 will be described with reference to FIG. 12.

In step S201, the processor 501 of the computer 500 defines the mixed reality space 21.

In step S202, the processor 501 detects the position of the MR user 6 within the mixed reality space 21. Further, the processor 501 uses the position information indicating the detected position of the MR user 6 to control the virtual space 11, which is a computer 300 connected to the computer 500 via the network 2 and corresponds to the mixed reality space 21. It is transmitted to the processor 301 of the computer 300.

In step S203, the processor 501 acquires position information indicating the position of the VR avatar 25 in the virtual space 11 from the processor 301.

In step S204, the processor 501 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR avatar 25 in the virtual space 11, based on the position information acquired in step S203. Note that arranging at corresponding positions may mean that the position in the virtual space 11 is determined in advance, or the position in the mixed reality space 21 may be determined in advance.

In step S205, the processor 501 places the virtual model 30 as a virtual object in the mixed reality space 21. The processor 501 places the virtual model 30 at a position in the mixed reality space 21 that corresponds to the position of the virtual model 30 in the virtual space 11 .

In step S206, processor 501 receives input from MR user 6. Specifically, the processor 501 receives, for example, the output of a detection means for detecting the movement of the MR user 6 as an input from the MR user 6.

In step S207, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the MR user 6. Specifically, the processor 501 moves the virtual model 30 in the mixed reality space 21 based on input from the MR user 6, for example. Additionally, processor 501 transmits information regarding input from MR user 6 to processor 301 .

In step S208, the processor 501 acquires information regarding the input received by the processor 301 from the VR user.

In step S209, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user obtained in step S208. Specifically, the processor 501 moves the VR avatar 25 or the virtual model 30 in the mixed reality space 21 based on, for example, input from a VR user. For example, if the input is related to the VR avatar 25, the processor 501 moves the VR avatar 25. Further, if the input is related to the virtual model 30, the processor 501 moves the virtual model 30.

Next, an example of a process for reflecting the behavior of the MR user 6 in the virtual space 11 will be described with reference to FIG. 13.

In step S301, processor 501 receives input from MR user 6. Specifically, the processor 501 receives, as an input from the MR user 6, the output of a detection means that detects the behavior of the MR user 6 with respect to a specific real object (for example, the blackboard 35, a laser pointer, or an electronic piano). .

In step S302, the processor 501 transmits information regarding the input from the MR user 6 accepted in step S301 to the processor 301 (in other words, the VR system 200).

In step S303, the processor 301 gives a predetermined change to the virtual object corresponding to the specific real object on which the MR user 6 acted, based on the information transmitted in step S302. For example, when the MR user 6 performs a writing operation to write predetermined contents on the blackboard 35, the processor 301 changes the virtual blackboard 36 corresponding to the blackboard 35 to a state where the predetermined contents are displayed.

Next, an example of a process for reflecting the VR user's actions in the mixed reality space 21 will be described with reference to FIG. 14.

In step S351, the processor 301 receives input from the VR user. Specifically, the processor 301 receives an input from the VR user and outputs the output of a detection means that detects the behavior of the VR user with respect to a virtual object (for example, the virtual blackboard 36 or the virtual electronic piano, etc.) corresponding to a specific real object. accepted as.

In step S352, the processor 301 transmits information regarding the input from the VR user accepted in step S351 to the processor 501 (in other words, the MR system 400).

In step S353, the processor 501 causes the action performed by the VR user on the virtual object corresponding to the specific real object to be reflected in the specific real object, based on the information transmitted in step S352. For example, when the VR user performs a writing operation to write predetermined contents on the virtual blackboard 36, the processor 501 changes the blackboard 35 to a state where the predetermined contents are displayed.

Note that the present invention is not limited to the embodiments described above, and can be implemented with various modifications without departing from the gist thereof. Within the scope of the present invention, each component can be freely combined, any component can be modified, or any component can be omitted. Further, the process flow described in this specification is merely an example, and the order and configuration of each process may be different.

<Additional notes>
The matters described in the above embodiments may also be described as in the following additional notes.

(Additional note 1)
a first display means (for example, VR device 210) that provides a first user with a first view that is a view of a virtual space imitating a predetermined real space;
A complex in which a second virtual object (for example, VR avatar 25 or virtual model 30) corresponding to a first virtual object (for example, VR avatar 25 or virtual model 30) in the virtual space is arranged in the predetermined real space. a second display means (for example, MR device 410) that provides a second user with a second view that is a view of real space;
a first input receiving means (for example, VR side input receiving unit 916) that receives input from the first user;
a first control means (for example, the control unit 310 or the VR-side object control unit 920) that controls the virtual object in the virtual space;
a second control means (for example, the control unit 510 or the MR side object control unit 820) for controlling the virtual object in the mixed reality space,
The first control means controls the first virtual object in the virtual space based on the input from the first user received by the first input reception means,
The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and An information processing system that controls the second virtual object in the mixed reality space based on input from one user.
According to such a configuration, the second virtual object displayed in the predetermined real space is displayed based on the input from the first user, similar to the first virtual object in the virtual space imitating the predetermined real space. It will be controlled. For example, when a first user moves a first virtual object in a virtual space imitating a predetermined real space, a second virtual object in the predetermined real space (in other words, mixed reality space) also moves in the same way. Become. Since the virtual space imitates a predetermined real space, the first user can check how the second virtual object is controlled in the real space, how it looks, etc. in the virtual space. can be imaged via a first virtual object controlled by. As described above, according to the present invention, real space and virtual space are highly integrated, and the range of uses of virtual space is expanded.

(Additional note 2)
The information processing system according to supplementary note 1, wherein the first virtual object and the second virtual object are avatars of the first user.
According to such a configuration, the first user's avatar that is controlled based on the input from the first user is displayed in the mixed reality space. Further, the second user can share an experience with the first user's avatar in the mixed reality space. Therefore, real space and virtual space are highly integrated, and the range of uses of virtual space is expanded.

(Additional note 3)
The first input receiving means receives the output of the detection means for detecting the movement of the first user as an input from the first user,
The second control means controls the second virtual object so that the movement of the first user's avatar in the mixed reality space imitates the movement of the first user detected by the detection means. The information processing system described in Appendix 2.
According to such a configuration, since the movements of the first user are directly reflected in the avatar, it is easy for the first user to convey something to the second user, express emotions, etc. Become. At this time, the first user moves the avatar in a virtual space that imitates real space, so the first user can make the avatar move more naturally. Further, from the second user's viewpoint, the first user's avatar appears in the real space, and the avatar moves naturally in the real space and highly reflects the first user's intention. Therefore, the quality of communication between the first user and the second user via the virtual space is improved, and the range of uses of the virtual space is expanded.

(Additional note 4)
The first control means includes:
arranging in the virtual space a specific virtual object (for example, a virtual blackboard 36 or a virtual electronic piano) corresponding to a specific object (for example, a blackboard 35 or an electronic piano) that actually exists in the predetermined real space;
applying a predetermined change to the specific virtual object based on the input from the first user received by the first input receiving means;
The information processing system includes a reflection unit (for example, an MR side object control unit 820, a blackboard The information processing system according to supplementary note 1, further comprising a display control unit or a sound control unit 845).
According to such a configuration, an operation performed by the first user on a virtual object in the virtual space can be reflected on an object that actually exists in the real space where the second user is present. Therefore, real space and virtual space are highly integrated, and the range of uses of virtual space is expanded.

(Appendix 5)
comprising a second input receiving means (for example, MR side input receiving unit 816) that receives input from the second user;
The information processing system according to supplementary note 1, wherein the first control means applies a predetermined change to the first virtual object based on the input from the second user received by the second input reception means.
According to such a configuration, the second user can also influence the virtual object in the virtual space. In addition, since the first user and the second user make changes to the target object in a space with a similar shape, it is difficult for the other side to make changes to the target object. This makes it possible to easily understand how things look, etc. Therefore, the range of uses of virtual space is expanded.

(Appendix 6)
comprising a second input receiving means (for example, MR side input receiving unit 816) that receives input from the second user;
The second input receiving means receives an output of a position detecting means for detecting the position of the second user in the mixed reality space as an input from the second user,
The first control means places the second user's avatar at a position in the virtual space that corresponds to the position of the second user in the mixed reality space detected by the position detection means. Supplementary Notes 1 to 5 The information processing system according to any one of.
According to such a configuration, in the virtual space, the second user's avatar is placed at a position corresponding to the actual position of the second user. Therefore, the first user can easily know where the second user is actually located. Therefore, the first user can easily grasp the relationship between the position of the second virtual object moved by the first user's input and the second user's position. Therefore, the range of uses of virtual space is expanded.

(Appendix 7)
a first display means (for example, VR device 210) that provides a first user with a first view that is a view of a virtual space imitating a predetermined real space;
A complex in which a second virtual object (for example, VR avatar 25 or virtual model 30) corresponding to a first virtual object (for example, VR avatar 25 or virtual model 30) in the virtual space is arranged in the predetermined real space. a second display means (for example, MR device 410) that provides a second user with a second view that is a view of real space;
a first input receiving means (for example, VR side input receiving unit 916) that receives input from the first user;
A first control means (for example, control unit 310 or VR side object control unit 920),
A computer of an information processing system comprising a second control means (for example, the control unit 510 or the MR side object control unit 820) that controls the virtual object in the mixed reality space functions as the second control means,
The second control means places the second virtual object at a position in the mixed reality space corresponding to the position of the first virtual object in the virtual space, and A program that controls the second virtual object in the mixed reality space based on input from one user.
According to such a configuration, the same effects as those of the program described in Supplementary Note 1 can be achieved.

11 virtual space, 21 mixed reality space, 25 VR avatar, 26 MR avatar, 30 virtual model, 35 blackboard, 36 virtual blackboard, 100 information processing system, 200 VR system, 210 VR device, 211 display, 212 gaze sensor, 213 No. 1 camera, 214 second camera, 215 microphone, 216 speaker, 217 sensor, 260 detection device, 270 display, 280 controller, 286 sensor, 300 computer, 301 processor, 302 memory, 303 storage, 304 input/output interface, 305 communication interface , 310 control unit, 311 storage unit, 400 MR system, 410 MR device, 411 display, 412 gaze sensor, 413 camera, 415 microphone, 416 speaker, 417 sensor, 460 detection device, 470 display, 480 controller, 486 sensor, 500 Computer, 501 processor, 502 memory, 503 storage, 504 input/output interface, 505 communication interface, 510 control unit, 511 storage unit, 600 server, 601 processor, 602 memory, 603 storage, 604 input/output interface, 605 communication interface, 610 control unit, 611 storage unit, 700 external device, 810 virtual space generation unit, 812 coordinate definition unit, 816 MR side input reception unit, 818 user information acquisition unit, 820 MR side object control unit, 840 display control unit, 845 sound control part, 850 communication control unit, 916 VR side input reception unit, 920 VR side object control unit, 930 virtual camera control unit, 940 display control unit, 945 sound control unit, 950 communication control unit

Claims (2)

A view of a virtual space generated based on 3D information acquired by a 3D information acquisition means that acquires 3D information about the predetermined real space, including 3D information about buildings constituting the predetermined real space. a first display means for providing a first view to a first user;
a second display means for providing a second user with a second view that is a view of a mixed reality space in which virtual objects are arranged in the predetermined real space;
a first input receiving means for receiving input from the first user;
a second input receiving means for receiving input from the second user;
a first control means for controlling a virtual object in the virtual space;
a second control means for controlling a virtual object in the mixed reality space;
reflecting means for reflecting the input from the first user received by the first input receiving means on an object actually existing in the predetermined real space;
The first control means includes:
An avatar of the second user is placed at a position in the virtual space that corresponds to a position of the second user in the mixed reality space detected by a position detection means that detects the position of the second user in the mixed reality space. and placing
arranging in the virtual space a predetermined virtual object that can be operated by the first user and the second user;
changing the predetermined virtual object in the virtual space based on the input from the first user received by the first input receiving means;
changing the predetermined virtual object in the virtual space based on the input from the second user received by the second input receiving means;
The second control means
arranging the first user's avatar at a position in the mixed reality space that corresponds to a position of the first user's avatar in the virtual space;
arranging the predetermined virtual object at a position in the mixed reality space that corresponds to a position of the predetermined virtual object in the virtual space;
changing the predetermined virtual object in the mixed reality space based on the input from the second user received by the second input receiving means;
changing the predetermined virtual object in the mixed reality space based on the input from the first user received by the first input receiving means;
The first control means illuminates a pointed position in the virtual space based on an operation by the first user of pointing a virtual laser pointer to an arbitrary position in the virtual space,
The reflecting means illuminates a position in the mixed reality space that corresponds to the pointed position in the virtual space.
Information processing system. A view of a virtual space generated based on 3D information acquired by a 3D information acquisition means that acquires 3D information about the predetermined real space, including 3D information about buildings constituting the predetermined real space. a first display means for providing a first view to a first user;
a second display means for providing a second user with a second view that is a view of a mixed reality space in which virtual objects are arranged in the predetermined real space;
a first input receiving means for receiving input from the first user;
a second input receiving means for receiving input from the second user;
a first control means for controlling a virtual object in the virtual space;
a second control means for controlling a virtual object in the mixed reality space;
a first sound control means for controlling a sound output to the first user;
a second sound control means for controlling sound output in the mixed reality space ,
The first control means includes:
An avatar of the second user is placed at a position in the virtual space that corresponds to a position of the second user in the mixed reality space detected by a position detection means that detects the position of the second user in the mixed reality space. and placing
arranging in the virtual space a predetermined virtual object that can be operated by the first user and the second user;
changing the predetermined virtual object in the virtual space based on the input from the first user received by the first input receiving means;
changing the predetermined virtual object in the virtual space based on the input from the second user received by the second input receiving means;
arranging in the virtual space a specific virtual object corresponding to a musical instrument that actually exists in the predetermined real space ;
The second control means
arranging the first user's avatar at a position in the mixed reality space that corresponds to a position of the first user's avatar in the virtual space;
arranging the predetermined virtual object at a position in the mixed reality space that corresponds to a position of the predetermined virtual object in the virtual space;
changing the predetermined virtual object in the mixed reality space based on the input from the second user received by the second input receiving means;
changing the predetermined virtual object in the mixed reality space based on the input from the first user received by the first input receiving means;
The first sound control means outputs a sound corresponding to the second user's operation on the musical instrument, and makes the sound sound as if it is coming from the direction of the specific virtual object. control the sound output,
The second sound control means outputs a sound corresponding to the first user's operation on the specific virtual object in the mixed reality space.
Information processing system.

Images