JP7354466B1 - Information processing systems and programs - Google Patents

Abstract

The present invention provides an information processing system and program that expands the range of uses of virtual space. An information processing system includes a control unit 310 that provides a first user with a first view that is a view of a virtual space, and a second view that is a view of a mixed reality space in which virtual objects are arranged in a real space. A control unit 510 that provides a view to the second user, an MR side input reception unit 816 that accepts input of position information indicating the second user's current position in the real space, and a point in the real space and a point in the virtual space. a storage unit 611 that stores information indicating a correspondence relationship between , when the location information indicates that the second user is at a location corresponding to a specific location, the control unit 510 causes the control unit 510 to reflect a predetermined event occurring at a specific location in the virtual space according to the location information. The second view is provided to the second user. [Selection diagram] Figure 9

Description

The present invention relates to an information processing system and a program.

BACKGROUND ART Conventionally, techniques for providing mixed reality (MR) spaces and virtual reality (VR) spaces to users have been known (for example, see Patent Document 1 and Patent Document 2).

Patent No. 6860488 Patent No. 6800599

Incidentally, there has been a demand for expanding the range of uses of virtual space.

The present invention has been made in view of the above circumstances, and aims to widen the range of uses of virtual space.

According to one embodiment presented in the present disclosure,
a first control means for providing a first user with a first view that is a view of a virtual space;
a second control means for providing a second user with a second view that is a view of a mixed reality space in which virtual objects are placed in the real space;
input receiving means for receiving position information indicating the current position of the second user in the real space;
Correspondence storage means for storing information indicating a correspondence between the positions of the real space and the virtual space,
The first control means provides the first user with the first view regarding a predetermined event occurring at a specific position in the virtual space;
The second control means
providing the second user with the second view according to the location information;
When the location information indicates that the second user is at a location corresponding to the specific location, the second user is configured to reflect the predetermined event taking place at the specific location in the virtual space. An information processing system is provided that provides the second user with a view of:

According to the present invention, the range of uses of virtual space can be expanded.

1 is a diagram showing a schematic configuration of an information processing system. FIG. 1 is a diagram showing a schematic configuration of a VR system. 1 is a diagram showing a schematic configuration of an MR system. FIG. 2 is a diagram conceptually showing a virtual space. FIG. 3 is a diagram showing a YZ cross section of a viewing area viewed from the X direction in virtual space. FIG. 3 is a diagram showing an XZ cross section of a viewing area viewed from the Y direction in virtual space. FIG. 2 is a diagram showing a schematic configuration of a controller. FIG. 2 is a diagram for explaining mixed reality space and virtual space. FIG. 1 is a diagram showing a functional configuration of an information processing system. FIG. 3 is a diagram for explaining a real object and a virtual object corresponding to the real object. It is a flowchart which shows an example of the process performed by a VR system. 3 is a flowchart illustrating an example of processing executed by the MR system. 12 is a flowchart illustrating an example of a process for reflecting an MR user's behavior in a virtual space. 12 is a flowchart illustrating an example of a process for reflecting a VR user's behavior in a mixed reality space. FIG. 3 is a diagram for explaining the reflection of an event in virtual space in real space. FIG. 3 is a diagram for explaining the reflection of an event in real space in virtual space.

Embodiments of the present invention will be described below with reference to the drawings.

[First embodiment]
<Configuration of information processing system>
As shown in FIG. 1, the information processing system 100 of this embodiment includes a VR (Virtual Reality) system 200, an MR (Mixed Reality) system 400, a server 600, and an external device 700. The VR system 200 is configured to be able to communicate with the server 600, the MR system 400, and the external device 700 via the network 2. Further, the MR system 400 is configured to be able to communicate with the server 600, the VR system 200, and the external device 700 via the network 2. The number of MR systems 400 that constitute the information processing system 100 is not limited to one, and a plurality of MR systems may exist. Further, the number of VR systems 200 configuring the information processing system 100 is not limited to one, and a plurality of VR systems 200 may exist. Note that in this embodiment, communication between the VR system 200 and the MR system 400 via the network 2 is performed via the server 600, but may be performed without using the server 600. Note that the network 2 includes, for example, the Internet, a mobile communication system (for example, 3G, 4G, 5G, LTE (Long Term Evolution), etc.), Wi-Fi (registered trademark), Bluetooth (registered trademark), other communication lines, Alternatively, it may be configured by any combination of these.

(VR system configuration)
As shown in FIG. 2, the VR system 200 includes a VR device 210, a computer 300, a detection device 260, a display 270, and a controller 280. The VR device 210 includes a display 211, a gaze sensor 212, a first camera 213, a second camera 214, a microphone 215, a speaker 216, and a sensor 217. Hereinafter, the user who uses the VR device 210 will be referred to as a VR user.

The computer 300 can be connected to the Internet or other network 2, and can communicate with, for example, the server 600, the computer of the MR system 400, and other computers connected to the network 2. Examples of other computers include computers of other VR systems 200, external equipment 700, and the like.

The VR device 210 can be worn on the head of a VR user and provide a virtual space to the VR user during operation. The VR device 210 may be, for example, a so-called head-mounted display including a display, or may be a head-mounted device equipped with a smartphone or other terminal having a display. For example, the VR device 210 displays an image for the right eye and an image for the left eye on the display 211. When each eye of the VR user views a respective image, the VR user can recognize the image as a three-dimensional image based on the parallax between the eyes.

The display 211 is realized, for example, as a non-transmissive display device. The display 211 is arranged, for example, on the main body of the VR device 210 so as to be located in front of both eyes of the VR user. Therefore, when the VR user views the three-dimensional image displayed on the display 211, the VR user can immerse himself in the virtual space. Note that the display 211 may be realized by a display included in a so-called smartphone or other terminal.

Detection device 260 detects the movement of a VR user using VR device 210. The detection device 260 may have a position tracking function for detecting the movement of the VR device 210, thereby detecting the movement of the VR user. Specifically, the detection device 260 has a sensor that reads light (for example, infrared light) from the VR device 210 as a sensor for detecting the movement of the VR user, and detects the movement of the VR device in the real space. It may be possible to detect the position, inclination, etc. of 210. In this case, the VR device 210 may include a plurality of light sources (not shown). Further, each light source may be realized by, for example, an LED (Light Emitting Diode) that emits infrared light.

Further, the detection device 260 may be realized by, for example, a camera. Specifically, the detection device 260 has an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor) as a sensor for detecting the movement of the VR user. It may be. In other words, the VR user's movements may be detected by the camera. The detection device 260 is, for example, a depth camera that includes a device that emits predetermined light such as infrared light or light with a predetermined pattern, and an image sensor (for example, a depth sensor), and detects the light emitted from the device. The image sensor may detect the reflected light of the image sensor, and the position, posture, etc. of the VR user may be detected based on the output from the image sensor. Further, the detection device 460 includes such a depth camera and a camera capable of acquiring RGB images, and detects the position, posture, etc. of the VR user based on the output from these cameras. There may be. Further, the detection device 260 may be, for example, a stereo camera including a plurality of image sensors, and may be one that detects the position, posture, etc. of the VR user based on outputs from the plurality of image sensors. Note that the detection of the position, posture, etc. of the VR user may be the detection of the position, posture, etc. of the VR user's body, or may be the detection of the position, tilt, etc. of the VR device 210.

Note that the VR system 200 may have one or more types of detection devices as the detection device 260, or may have a plurality of each type of detection device. Furthermore, the VR system 200 does not need to include the detection device 260. Note that a part of the detection device 260 may be configured by the computer 300 or the like. For example, analysis of the output from the image sensor (eg, image recognition, etc.) may be performed by the computer 300 or the like.

In addition to the various sensors described above, the detection device 260 may include a sensor capable of detecting the position, inclination, etc. of the detection device 260 itself. Specifically, for example, the detection device 260 includes an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. It's okay. Further, the outputs from these sensors may be sent to the computer 300 or the like, and may be used, for example, when predetermined processing is performed based on the outputs from the image sensor included in the detection device 260.

Further, the VR device 210 may include a sensor 217 instead of or in addition to the detection device 260 as a detection means for detecting the movement of the VR user. The VR device 210 can use the sensor 217 to detect the position and tilt of the VR device 210 itself. The sensor 217 may be, for example, an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. Further, the VR device 210 may have one or more types of sensors as the sensor 217, or may have a plurality of each type of sensor. As an example, when an angular velocity sensor is used as the sensor 217, the VR device 210 can detect the angular velocity around three axes of the VR device 210 in real space over time. Then, the VR device 210 calculates the temporal change in the angle of the VR device 210 around the three axes based on each angular velocity, and further calculates the tilt of the VR device 210 based on the temporal change in the angle. etc. can be done. Further, the sensor 217 may be, for example, an image sensor. Then, the position, posture, etc. of the VR user may be detected based on the output from the image sensor. In other words, the position, posture, etc. of the VR user may be detected based on information etc. from a camera included in the VR device 210 that photographs the surroundings of the VR device 210. In other words, tracking of the VR device 210 may be performed using an outside-in method or an inside-out method.

Gaze sensor 212 detects the direction in which the VR user's right and left eyes are directed. That is, the gaze sensor 212 detects the VR user's line of sight (in other words, eye movement). Detection of the direction of the line of sight is realized, for example, by a known eye tracking function. The gaze sensor 212 is realized by a sensor having the eye tracking function. Gaze sensor 212 may include a right eye sensor and a left eye sensor. The gaze sensor 212 may be, for example, a sensor that irradiates infrared light to the right and left eyes of the VR user and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. . In this case, the gaze sensor 212 can detect the VR user's line of sight based on each detected rotation angle.

The first camera 213 photographs the lower part of the VR user's face. More specifically, the first camera 213 photographs the nose, mouth, etc. of the VR user. The second camera 214 photographs the eyes, eyebrows, etc. of the VR user. Here, the VR user side of the housing of the VR device 210 is defined as the inside of the VR device 210, and the side of the housing of the VR device 210 opposite to the VR user is defined as the outside of the VR device 210. The first camera 213 may be placed outside the VR device 210, and the second camera 214 may be placed inside the VR device 210. Images captured by the first camera 213 and the second camera 214 are input to the computer 300. Note that the first camera 213 and the second camera 214 may be implemented as one camera, and the face of the VR user may be photographed with this one camera.

The microphone 215 serving as a sound input means converts the voice of the VR user into a sound signal (in other words, an electrical signal) and outputs it to the computer 300. The speaker 216 as a sound output means converts the sound signal into sound and outputs it to the VR user. Note that the VR device 210 may include earphones instead of the speaker 216 as a sound output means.

Display 270 displays an image similar to the image displayed on display 211. This allows users other than the VR user wearing the VR device 210 to view the same images (in other words, virtual space) as the VR user. The image displayed on the display 270 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the VR device 210. Examples of the display 270 include a liquid crystal display and an organic EL display.

Controller 280 is connected to computer 300 by wire or wirelessly. The controller 280 accepts input operations related to instructions from the VR user to the computer 300. Further, the controller 280 accepts input operations by the VR user for controlling the position and movement of virtual objects placed in the virtual space. The controller 280 may be configured to be graspable by a VR user, for example. Further, the controller 280 may be configured to be able to be attached to a part of the VR user's body or clothing, for example. Specifically, the controller 280 may be, for example, glove-shaped. Further, the controller 280 may be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 300.

Further, the controller 280 may include multiple light sources. Each light source may be realized by, for example, an LED that emits infrared light. Then, the detection device 260 may read the infrared light from the controller 280 and detect the position, tilt, etc. of the controller 280 in the real space. In other words, the detection device 260 may have a position tracking function that detects the movement of the controller 280, thereby detecting the movement of the VR user.

Note that the VR system 200 does not need to include the controller 280.

Furthermore, the information processing system 100 may include a sensor 286 that detects the movement of the VR user. Sensor 286 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. Further, the sensor 286 may be provided in the controller 280, for example (see FIG. 7). Note that the information processing system 100 may have one or more types of sensors as the sensor 286, or may have a plurality of each type of sensor. Furthermore, a device including the sensor 286 (for example, the controller 280, a predetermined camera, etc.) and the computer 300 may be connected to each other wirelessly, for example. The information acquired by sensor 286 may then be transmitted to computer 300 via wireless communication, for example.

Further, the position and movement of the VR user can be determined by the VR device 210 equipped with a device that performs predetermined wireless communication (for example, short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB (Ultra Wide Band) communication). , the VR device 210 may be detected by performing wireless communication between the device and surrounding devices and acquiring position information of the VR device 210.

In addition, the position and movement of the VR user can be determined by using a device worn by the VR user (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). The detection may be performed using an external device 700 that is capable of tracking the user's movement (and a device that tracks the VR user's movement in cooperation with the device through short-range wireless communication or the like). Further, these external devices 700 may be used as the controller 280. Further, the position and movement of the VR user may be detected by a GPS sensor or the like.

(Configuration of computer 300)
As shown in FIG. 2, the computer 300 includes a processor 301, a memory 302, a storage 303, an input/output interface 304, and a communication interface 305 as main components. Each component is connected to each other by a bus.

Processor 301 controls the operation of VR device 210. The processor 301 reads the program from the storage 303 and expands it into the memory 302. Processor 301 executes the developed program. The processor 301 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 302 is a main storage device. The memory 302 is configured of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory 302 provides a work area to the processor 301 by temporarily storing programs and various data that the processor 301 reads from the storage 303. The memory 302 also temporarily stores various data generated while the processor 301 is operating according to a program, various data input to the computer 300, and the like.

Storage 303 is an auxiliary storage device. The storage 303 is configured of a storage device such as a flash memory or an HDD (Hard Disk Drive), for example. The storage 303 stores programs for providing services in the information processing system 100. Further, the storage 303 stores various data for providing services in the information processing system 100. The data stored in the storage 303 includes data for defining a virtual space, data regarding virtual objects, and the like.

Note that the storage 303 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 303 built into the computer 300, programs and data stored in an external storage device may be used.

The input/output interface 304 is an interface through which the computer 300 receives data input, and an interface through which the computer 300 outputs data. The input/output interface 304 can send or receive data to/from the VR device 210, the detection device 260, and the display 270. Furthermore, data may be transmitted to or received from the display 211, gaze sensor 212, first camera 213, second camera 214, microphone 215, speaker 216, and sensor 217 included in the VR device 210.

Note that the input/output interface 304 may transmit or receive data to/from the controller 280. For example, input/output interface 304 may receive input of signals output from controller 280 and sensor 286. Further, the input/output interface 304 may transmit instructions output from the processor 301 to the controller 280. The command may instruct the controller 280 to vibrate, output sound, emit light, or the like. Upon receiving the command, the controller 280 performs vibration, sound output, light emission, etc. in accordance with the command.

Note that each of the VR device 210, the detection device 260, the display 270, the controller 280, and the sensor 286 and the computer 300 may be connected by wire or wirelessly.

The communication interface 305 controls transmission and reception of various data via the network 2 with other computers (for example, the server 600, the computer 500, or the other computer 300, etc.).

The processor 301 accesses the storage 303, loads the program stored in the storage 303 into the memory 302, and executes a series of instructions included in the program. Further, the processor 301 sends a signal for providing a virtual space to the VR device 210 via the input/output interface 304.

Note that the computer 300 may be provided outside the VR device 210, or part or all of the computer 300 may be built into the VR device 210. Further, a portable terminal (for example, a smartphone) including the display 211 may function as at least a part of the computer 300. Note that when a plurality of VR apparatuses 210 exist, the computer 300 may be provided for each VR apparatus 210 or may be used in common for the plurality of VR apparatuses 210.

(Coordinate system)
In the VR system 200, a real coordinate system that is a coordinate system in real space is set in advance. The real coordinate system has three reference directions (axes) that are parallel to a vertical direction in real space, a horizontal direction perpendicular to the vertical direction, and a front-back direction perpendicular to both the vertical direction and the horizontal direction.

The position and tilt of the VR device 210 in real space can be detected by the detection device 260 and the sensor 217. The detected tilt of the VR device 210 corresponds to each tilt of the VR device 210 around three axes in the real coordinate system, for example. The computer 300 sets the uvw visual field coordinate system to the VR device 210 based on the inclination of the VR device 210 in the real coordinate system (see FIG. 4). The uvw visual field coordinate system set in the VR device 210 corresponds to the viewpoint coordinate system when a VR user wearing the VR device 210 views an object in the virtual space.

Here, the uvw visual field coordinate system will be explained. The computer 300 sets a three-dimensional uvw visual field coordinate system centered on the head of the VR user wearing the VR device 210 (origin). More specifically, the computer 300 tilts the horizontal direction, vertical direction, and front-rear direction that define the real coordinate system by the tilt of the VR device 210 around each axis within the real coordinate system. The three newly obtained directions are set as the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the uvw visual field coordinate system in the VR device 210.

For example, when a VR user wearing the VR device 210 is standing upright and viewing the front, the processor 301 sets a visual field coordinate system parallel to the real coordinate system in the VR device 210. In this case, the horizontal direction, vertical direction, and longitudinal direction in the real coordinate system coincide with the pitch axis (u axis), yaw axis (v axis), and roll axis (w axis) of the visual field coordinate system in the VR device 210. .

After the uvw visual field coordinate system is set in the VR device 210, the detection device 260 or sensor 217 can detect the tilt of the VR device 210 in the set uvw visual field coordinate system based on the movement of the VR device 210. In this case, the detection device 260 or the sensor 217 detects the pitch angle, yaw angle, and roll angle of the VR device 210 in the uvw visual field coordinate system as the inclination of the VR device 210, respectively. The pitch angle represents the tilt angle of the VR device 210 around the pitch axis in the uvw viewing coordinate system. The yaw angle represents the tilt angle of the VR device 210 around the yaw axis in the uvw viewing coordinate system. The roll angle represents the tilt angle of the VR device 210 around the roll axis in the uvw viewing coordinate system.

Based on the detected tilt of the VR device 210, the computer 300 sets in the VR device 210 a uvw visual field coordinate system in the VR device 210 after the VR device 210 moves. The relationship between the VR device 210 and the uvw visual field coordinate system of the VR device 210 is always constant regardless of the position and tilt of the VR device 210. When the position and inclination of the VR device 210 change, the position and inclination of the uvw visual field coordinate system of the VR device 210 in the real coordinate system change in conjunction with the change in the position and inclination.

Note that the detection device 260 may specify the position of the VR device 210 in the real space as a relative position with respect to the detection device 260. Furthermore, the processor 301 may determine the origin of the uvw visual field coordinate system of the VR device 210 in the real space (real coordinate system) based on the specified relative position.

(Virtual space)
The virtual space will be further explained with reference to FIG. 4. FIG. 4 is a diagram conceptually representing one aspect of representing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure that covers the entire 360 degree direction of the center 12. In FIG. 4, the celestial sphere in the upper half of the virtual space 11 is illustrated in order not to complicate the explanation. In the virtual space 11, each mesh is defined. The position of each mesh is predefined as a coordinate value in an XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 300 associates each partial image forming the panoramic image 13 (still image, moving image, etc.) that can be developed in the virtual space 11 with each corresponding mesh in the virtual space 11 .

In the virtual space 11, an XYZ coordinate system with a predetermined point as the origin 12 is defined. For example, the XYZ coordinate system is parallel to the real coordinate system. The horizontal direction, vertical direction (vertical direction), and front-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X axis of the XYZ coordinate system is parallel to the horizontal direction of the real coordinate system, the Y axis (vertical direction) of the XYZ coordinate system is parallel to the vertical direction of the real coordinate system, and the Z axis of the direction) is parallel to the front-back direction of the real coordinate system.

When the VR device 210 is started, that is, in the initial state of the VR device 210, the virtual camera 14 is placed at a predetermined position (for example, at the center) of the virtual space 11. Furthermore, the processor 301 displays the image captured by the virtual camera 14 on the display 211 of the VR device 210. The virtual camera 14 moves within the virtual space 11 in conjunction with the movement of the VR device 210 in the real space. Thereby, changes in the tilt and position of the VR device 210 in the real space can be similarly reproduced in the virtual space 11.

As in the case of the VR device 210, the uvw visual field coordinate system is defined for the virtual camera 14. The UVW visual field coordinate system of the virtual camera 14 in the virtual space 11 is defined to be linked to the UVW visual field coordinate system of the VR device 210 in the real space (real coordinate system). Therefore, when the tilt of the VR device 210 changes, the tilt of the virtual camera 14 also changes accordingly. Note that the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in the real space, but in this embodiment, even if the VR user moves in the real space, the virtual camera 14 may move in the virtual space 11 in conjunction with the movement of the VR user in the real space. It is designed not to move.

The processor 301 of the computer 300 defines the viewing area 15 in the virtual space 11 based on the position and tilt of the virtual camera 14. The viewing area 15 corresponds to an area of the virtual space 11 that is viewed by a VR user wearing the VR device 210. In other words, the position of the virtual camera 14 can be said to be the VR user's viewpoint in the virtual space 11.

The uvw visual field coordinate system of the VR device 210 is equal to the viewpoint coordinate system when the VR user views the display 211 . The uvw visual field coordinate system of the virtual camera 14 is linked to the uvw visual field coordinate system of the VR device 210. Therefore, the line of sight of the VR user detected by the gaze sensor 212 can be regarded as the line of sight of the VR user in the uvw visual field coordinate system of the virtual camera 14.

(Visual area)
The viewing area 15 will be described with reference to FIGS. 5 and 6. FIG. 5 is a diagram showing a YZ cross section of the viewing area 15 viewed from the X direction in the virtual space 11. FIG. 6 is a diagram showing an XZ cross section of the viewing area 15 viewed from the Y direction in the virtual space 11.

As shown in FIG. 5, the viewing area 15 in the YZ cross section includes a region 18. The region 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 301 defines a range including the polar angle α centered on the reference line of sight 16 in the virtual space as a region 18 .

As shown in FIG. 6, the viewing area 15 in the XZ cross section includes a region 19. The region 19 is defined by the position of the virtual camera 14, the reference line of sight 16, and the XZ cross section of the virtual space 11. The processor 301 defines a range including the azimuth angle β centered on the reference line of sight 16 in the virtual space 11 as a region 19 . The polar angles α and β are determined depending on the position of the virtual camera 14 and the inclination (orientation) of the virtual camera 14.

The VR system 200 provides the VR user with a view in the virtual space 11 by displaying the view image 17 on the display 211 based on a signal from the computer 300 (see FIG. 4). The visibility image 17 is an image corresponding to a portion of the panoramic image 13 that corresponds to the visibility area 15. When the VR user moves the VR device 210 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the viewing area 15 in the virtual space 11 changes. Thereby, the view image 17 displayed on the display 211 is updated to an image of the panoramic image 13 that is superimposed on the view area 15 in the direction toward which the VR user faces in the virtual space 11. The VR user can visually recognize a desired direction in the virtual space 11.

While wearing the VR device 210, the VR user can view only the panoramic image 13 developed in the virtual space 11 without viewing the real world. Therefore, the information processing system 100 can provide the VR user with a highly immersive feeling in the virtual space 11.

Note that the virtual camera 14 may include two virtual cameras, that is, a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. In this case, appropriate parallax is set for the two virtual cameras so that the VR user can recognize the three-dimensional virtual space 11. Note that the virtual camera 14 may be realized by one virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by one virtual camera.

(controller)
An example of the controller 280 will be described with reference to FIG. 7.

As shown in FIG. 7, the controller 280 may include a right controller 280R and a left controller (not shown). In this case, the right controller 280R is operated with the VR user's right hand. The left controller is operated with the VR user's left hand. In one aspect, the right controller 280R and the left controller are configured as separate devices. Therefore, the VR user can freely move the right hand holding the right controller 280R and the left hand holding the left controller. In another aspect, controller 280 may be an integrated controller that accepts operation with both hands. The right controller 280R will be explained below.

The right controller 280R includes a grip 281, a frame 282, a top surface 283, buttons 284 and 285, a sensor (for example, a motion sensor) 286, an infrared LED 287, buttons 288 and 289, and an analog stick 290. Equipped with Grip 281 is configured to be held by the VR user's right hand. For example, the grip 281 may be held by the VR user's right hand palm and three fingers (middle finger, ring finger, little finger).

The button 284 is arranged on the side surface of the grip 281 and accepts operation by the middle finger of the right hand. Button 285 is arranged on the front surface of grip 281 and accepts operation by the index finger of the right hand. Further, the buttons 284 and 285 are provided with switches that detect the movement of the user who presses the buttons 284 and 285. Note that the buttons 284 and 285 may be configured as trigger-type buttons.

The sensor 286 is built into the housing of the grip 281. Additionally, the sensor 286 detects the movement of the VR user. Specifically, sensor 286 detects the movement of the VR user's hand. For example, the sensor 286 detects the rotation speed, number of rotations, etc. of the hand. Note that the controller 280 does not need to include the sensor 286.

A plurality of infrared LEDs 287 are arranged on the frame 282 along its circumferential direction. The infrared light LED 287 emits infrared light while a program using the controller 280 is being executed in accordance with the progress of the program. The infrared light emitted from the infrared light LED 287 can be used to detect the positions and postures (tilt, orientation) of the right controller 280R and the left controller.

The top surface 283 includes buttons 288 and 289 and an analog stick 290. Buttons 288 and 289 accept operations by the VR user's right thumb. Further, the buttons 288 and 289 are provided with switches that detect the movement of the user who presses the buttons 288 and 289. The analog stick 290 accepts operation in any direction within 360 degrees from the initial position (neutral position). The operation includes, for example, an operation for moving an object placed in the virtual space 11. Further, the analog stick 290 includes a sensor that detects the movement of the user who operates the analog stick 290.

Information regarding the VR user's input operation (in other words, the VR user's movement) detected by the controller 280 is sent to the computer 300.

As shown in FIG. 7, the controller 280 is defined with yaw, roll, and pitch directions, for example, with respect to the VR user's right hand. Specifically, for example, when a VR user extends his thumb and index finger, the direction in which the thumb extends is the yaw direction, the direction in which the index finger extends is the roll direction, and a plane defined by the axis of the yaw direction and the axis of the roll direction. The direction perpendicular to is defined as the pitch direction.

(Configuration of MR system)
As shown in FIG. 3, the MR system 400 includes an MR device 410, a computer 500, a detection device 460, a display 470, and a controller 480. MR device 410 includes a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417. Hereinafter, the user who uses the MR device 410 will be referred to as the MR user 6.

The computer 500 can be connected to the Internet or other networks 2, and can communicate with, for example, the server 600, the computer of the VR system 200, and other computers connected to the network 2. Examples of other computers include computers of other MR systems 400, external equipment 700, and the like.

The MR device 410 is attached to the head of the MR user 6 and can provide the MR user 6 with a mixed reality space during operation. The MR device 410 may be, for example, a glasses-type device (for example, so-called MR glasses) including a display. Further, the MR device 410 may be a contact lens type device or the like.

The display 411 is realized, for example, as a transmissive display device. Note that the transmissive display 411 may temporarily function as a non-transmissive display device by adjusting its transmittance. The display 411 is arranged on the main body of the MR apparatus 410, for example, so as to be located in front of both eyes of the MR user 6. Therefore, the MR user 6 is provided with a display in which the virtual object displayed by the display 411 is superimposed on the real space visible through the transparent display 411. That is, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which virtual objects are placed in the real space. In other words, the MR device 410 allows the MR user 6 to simultaneously view a virtual object and an object existing in real space (hereinafter referred to as a "real object").

Detection device 460 detects the movement of MR user 6 using MR device 410. The detection device 460 may have a position tracking function for detecting the movement of the MR device 410, thereby detecting the movement of the MR user 6. Specifically, the detection device 460 has a sensor that reads light (for example, infrared light) from the MR device 410 as a sensor for detecting the movement of the MR user 6, and detects the movement of the MR user 6 in the mixed reality space. It may also be a device that detects the position, inclination, etc. of the MR device 410. In this case, the MR device 410 may include a plurality of light sources (not shown). Further, each light source may be realized by, for example, an LED (Light Emitting Diode) that emits infrared light.

Further, the detection device 460 may be realized by, for example, a camera. Specifically, the detection device 460 includes an image sensor (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor) as a sensor for detecting the movement of the MR user 6. It may be something. In other words, the movement of the MR user 6 may be detected by the camera. The detection device 460 is, for example, a depth camera that includes a device that emits predetermined light such as infrared light or light with a predetermined pattern, and an image sensor (for example, a depth sensor), and detects the light emitted from the device. The image sensor may detect the reflected light of the image sensor, and the position, posture, etc. of the MR user 6 may be detected based on the output from the image sensor. Further, the detection device 460 includes such a depth camera and a camera capable of acquiring RGB images, and detects the position, posture, etc. of the MR user 6 based on the output from these cameras. It may be. Further, the detection device 460 may be, for example, a stereo camera including a plurality of image sensors, and may be one that detects the position, posture, etc. of the MR user 6 based on outputs from the plurality of image sensors. . Note that the detection of the position, posture, etc. of the MR user 6 may be the detection of the body position, posture, etc. of the MR user 6, or may be the detection of the position, tilt, etc. of the MR device 410.

Note that the MR system 400 may include one or more types of detection devices as the detection device 460, or may include a plurality of each type of detection device. For example, the MR system 400 may include multiple cameras (in other words, multiple sensors), and the multiple cameras may function as the detection device 460. In this embodiment, the MR system 400 will be described as including three detection devices 460, and each detection device 460 includes a depth camera and a camera capable of acquiring RGB images. Note that a part of the detection device 460 may be configured by the computer 500 or the like. For example, analysis of the output from the image sensor (eg, image recognition, etc.) may be performed by the computer 500 or the like. Note that the MR system 400 does not need to include the detection device 460.

In addition to the various sensors described above, the detection device 460 may include a sensor capable of detecting the position, inclination, etc. of the detection device 460 itself. Specifically, for example, the detection device 460 includes an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. It's okay. Further, the outputs from these sensors may be sent to the computer 500 or the like, and may be used, for example, when predetermined processing is performed based on the outputs from the image sensor included in the detection device 460.

Further, as understood from the configuration of the MR device 410 described later, the MR device 410 of another MR user 6 may be used as the detection device 460.

Further, the MR device 410 may include a sensor 417 instead of or in addition to the detection device 460 as a detection means for detecting the movement of the MR user 6. MR device 410 can use sensor 417 to detect the position and tilt of MR device 410 itself. The sensor 417 may be, for example, an angular velocity sensor (for example, a 3-axis angular velocity sensor), an acceleration sensor (for example, a 3-axis acceleration sensor), a geomagnetic sensor (for example, a 3-axis geomagnetic sensor), or the like. Furthermore, the MR device 410 may have one or more types of sensors as the sensor 417, or may have a plurality of each type of sensor.

Gaze sensor 412 detects the direction in which the right and left eyes of MR user 6 are directed. That is, the gaze sensor 412 detects the line of sight (in other words, the movement of the eyes) of the MR user 6. Detection of the direction of the line of sight is realized, for example, by a known eye tracking function. The gaze sensor 412 is realized by a sensor having the eye tracking function. Gaze sensor 412 may include a right eye sensor and a left eye sensor. For example, the gaze sensor 412 may be a sensor that irradiates infrared light to the right and left eyes of the MR user 6 and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris of the irradiated light. good. In this case, the gaze sensor 412 can detect the line of sight of the MR user 6 based on each detected rotation angle.

The camera 413 photographs the surroundings (for example, in front) of the MR user 6. The MR device 410 may include, for example, a depth camera and a camera capable of acquiring RGB images as the camera 413. Then, based on the output from the camera 413, the computer 500 detects the shapes of objects around the MR device 410 (in other words, the MR user 6), the relative distances between the MR device 410 and the surrounding objects, etc. It's okay. Further, the position and tilt of the MR device 410 may be detected based on the output from the camera 413. In other words, tracking of the MR device 410 may be performed using an outside-in method or an inside-out method.

The microphone 415 serving as a sound input means converts the voice of the MR user 6 into a sound signal (in other words, an electrical signal) and outputs it to the computer 500. A speaker 416 serving as a sound output means converts the sound signal into sound and outputs it to the MR user 6. Note that the MR device 410 may include earphones instead of the speaker 416 as a sound output means.

Display 470 displays an image similar to the image displayed on display 411, for example. This allows users other than the MR user 6 wearing the MR device 410 to view the same images as the MR user 6. The image displayed on the display 470 does not need to be a three-dimensional image, and may be, for example, an image for the right eye or an image for the left eye displayed on the MR device 410. Examples of the display 470 include a liquid crystal display and an organic EL display. Note that the display 470 may display an image showing a view of the mixed reality space that the MR device 410 provides to the MR user 6.

Controller 480 is connected to computer 500 by wire or wirelessly. Controller 480 accepts input operations related to instructions from MR user 6 to computer 500. Further, the controller 480 accepts input operations by the MR user 6 to control the position and movement of virtual objects placed in the mixed reality space. The controller 480 may be configured to be graspable by the MR user 6, for example. Furthermore, the controller 480 may be configured to be attachable to the body or part of clothing of the MR user 6, for example. Further, the controller 480 may be configured to be able to output at least one of vibration, sound, and light based on a signal transmitted from the computer 500.

Further, the controller 480 may include multiple light sources. Each light source may be realized by, for example, an LED that emits infrared light. The detection device 460 may read the infrared light from the controller 480 and detect the position, tilt, etc. of the controller 480 in the mixed reality space. In other words, the detection device 460 may have a position tracking function that detects the movement of the controller 480, thereby detecting the movement of the MR user 6.

Note that the MR system 400 does not need to include the controller 480. Moreover, the controller 480 may have part or all of the configuration of the controller 280, for example.

Note that the information processing system 100 may include a sensor 486 that detects the movement of the MR user 6. Sensor 486 may be, for example, an angular velocity sensor, an acceleration sensor, or a geomagnetic sensor. Further, the sensor 486 may be provided in the controller 480, for example. Note that the information processing system 100 may have one or more types of sensors as the sensor 486, or may have a plurality of each type of sensor. Further, a device including the sensor 486 (for example, the controller 480, a predetermined camera, etc.) and the computer 500 may be connected to each other wirelessly, for example. The information acquired by sensor 486 may then be transmitted to computer 300 via wireless communication, for example.

Further, the position and movement of the MR user 6 can be determined by the MR device 410, which is equipped with a device that performs predetermined wireless communication (for example, short-range wireless communication such as Wi-Fi communication, Bluetooth communication, or UWB communication), and is connected to the device. The VR device 210 may be detected by wirelessly communicating with surrounding devices and acquiring position information of the VR device 210.

In addition, the position and movement of the MR user 6 can be determined by a device worn by the MR user 6 (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). , may be detected using an external device 700 that is capable of tracking the movement of the MR user 6 (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication or the like). Further, these external devices 700 may be used as the controller 480. Furthermore, the position and movement of the MR user 6 may be detected by a GPS sensor or the like.

(Configuration of computer 500)
As shown in FIG. 3, the computer 500 includes a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505 as main components. Each component is connected to each other by a bus.

Processor 501 controls the operation of MR device 410. Processor 501 reads a program from storage 503 and expands it into memory 502. Processor 501 executes the developed program. The processor 501 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 502 is a main storage device. The memory 502 is configured of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). The memory 502 provides a work area for the processor 501 by temporarily storing programs and various data that the processor 501 reads from the storage 503. The memory 502 also temporarily stores various data generated while the processor 501 is operating according to a program, various data input to the computer 500, and the like.

Storage 503 is an auxiliary storage device. The storage 503 is configured of a storage device such as a flash memory or a hard disk drive (HDD), for example. The storage 503 stores programs for providing services in the information processing system 100. Further, the storage 503 stores various data for providing services in the information processing system 100. The data stored in the storage 503 includes data for defining a mixed reality space, data regarding virtual objects, and the like.

Note that the storage 503 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 503 built into the computer 500, programs and data stored in an external storage device may be used.

The input/output interface 504 is an interface through which the computer 500 receives data input, and an interface through which the computer 500 outputs data. Input/output interface 504 may send data to or receive data from MR device 410, detection device 460, and display 470. In addition, the input/output interface 504 may transmit or receive data with a display 411, a gaze sensor 412, a camera 413, a microphone 415, a speaker 416, and a sensor 417 included in the MR apparatus 410.

Note that the input/output interface 504 may transmit or receive data to/from the controller 480. For example, input/output interface 504 may receive input of signals output from controller 480 and sensor 486. Further, the input/output interface 504 may transmit instructions output from the processor 501 to the controller 480. The command may instruct the controller 480 to vibrate, output sound, emit light, or the like. When the controller 480 receives the command, it vibrates, outputs sound, emits light, etc. in accordance with the command.

Note that each of the MR device 410, the detection device 460, the display 470, the controller 480, and the sensor 486 and the computer 500 may be connected by wire or wirelessly.

The communication interface 505 controls transmission and reception of various data via the network 2 with other computers (for example, the server 600, the computer 300, or the other computer 500, etc.).

The processor 501 accesses the storage 503, loads the program stored in the storage 503 into the memory 502, and executes a series of instructions included in the program. Further, the processor 501 sends a signal for providing mixed reality space to the MR device 410 via the input/output interface 504.

Note that the computer 500 may be provided outside the MR apparatus 410, or part or all of the computer 500 may be built into the MR apparatus 410. Furthermore, when there are a plurality of MR apparatuses 410, the computer 500 may be provided for each MR apparatus 410, or may be used in common for the plurality of MR apparatuses 410. Further, in this case, a part of the computer 500 may be built into each of the plurality of MR apparatuses 410. In this embodiment, a case where a plurality of MR apparatuses 410 are connected to a computer 500 will be described as an example.

The method of acquiring the position and inclination of the MR device 410 in the real space (in other words, the mixed reality space) is well known and can be detected by a method similar to the method used in the VR device 210 described above, so a description thereof will be omitted.

(Server configuration)
Server 600 may send programs to computer 300. Additionally, server 600 may send programs to computer 500. Further, the server 600 enables communication between the computer 300 and the computer 500. Additionally, the server 600 enables communication between the computer 300 and other computers 300 . For example, when the information processing system 100 provides a service in which multiple VR users can participate, each computer 300 of the VR users communicates with other computers 300 via the server 600, thereby allowing multiple VR users to participate in the same virtual space. VR users may be able to share their experiences. Note that each computer 300 may communicate with other computers 300 without going through the server 600.

As shown in FIG. 1, the server 600 includes a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605. Each component is connected to each other by a bus.

Processor 601 controls the overall operation of server 600. The processor 601 reads the program from the storage 603 and expands it into the memory 602. Processor 601 executes the developed program. The processor 601 may be configured to include one or more types of, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), and an FPGA (Field-Programmable Gate Array).

Memory 602 is a main storage device. The memory 602 is configured by a storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The memory 602 provides a work area to the processor 601 by temporarily storing programs and various data that the processor 601 reads from the storage 603. The memory 602 also temporarily stores various data generated while the processor 601 is operating according to a program, various data input to the server 600, and the like.

Storage 603 is an auxiliary storage device. The storage 603 is configured of a storage device such as a flash memory or a hard disk drive (HDD), for example. The storage 603 stores programs for providing services in the information processing system 100. Further, the storage 603 stores various data for providing services in the information processing system 100. The data stored in the storage 603 includes data for defining a virtual space, data for defining a mixed reality space, data regarding virtual objects, and the like.

Note that the storage 603 may be realized as a removable storage device such as a memory card. Furthermore, instead of the storage 603 built into the server 600, programs and data stored in an external storage device may be used.

The input/output interface 604 is an interface through which the server 600 receives data input, and an interface through which the server 600 outputs data. The input/output interface 604 can send or receive data to/from input devices such as a mouse and keyboard, and output devices such as a display.

The communication interface 605 controls transmission and reception of various data via the network 2 with other computers (for example, the computer 300, the computer 500, etc.).

External device 700 may be any device that can communicate with computer 300, computer 500, or server 600. The external device 700 may be, for example, a device that can communicate with the computer 300 via the network 2, or a device that can communicate with the computer 300 through short-range wireless communication or a wired connection. Furthermore, the external device 700 may be, for example, a device that can communicate with the computer 500 via the network 2, or a device that can communicate with the computer 500 through short-range wireless communication or a wired connection. Further, the external device 700 may be a device that can communicate with the server 600 via the network 2, for example. Examples of the external device 700 include, but are not limited to, a smart device, a PC (Personal Computer), and peripheral devices of the computer 300 and the computer 500.

<Control of information processing system>
In the MR system 400 of this embodiment, the MR device 410 provides the MR user 6 with a view of a mixed reality space in which virtual objects are arranged in the real space, similar to well-known MR devices. Furthermore, in the VR system 200 of this embodiment, the VR device 210 provides a view of the virtual space to the VR user, similar to well-known VR devices. Furthermore, in the MR system 400 of this embodiment, the MR user 6 is able to operate virtual objects in the mixed reality space. Furthermore, in the VR system 200 of this embodiment, the VR user can operate virtual objects in the virtual space. In addition, in the information processing system 100 of this embodiment, the movements of the VR user using the VR system 200 are reflected in the virtual objects included in the view of the mixed reality space provided by the MR system 400. There is. Furthermore, in the information processing system 100 of this embodiment, the movements of the MR user 6 using the MR system 400 are reflected in virtual objects included in the view of the virtual space provided by the VR system 200. . Note that in this embodiment, the movement of the VR user, etc. is reflected in the virtual object in the virtual space and the mixed reality space in real time. Furthermore, in this embodiment, the movement of the MR user 6 and the like is reflected in the virtual object in the virtual space and mixed reality space in real time.

In the following, an example will be described in which the configuration according to the present embodiment is applied to a service (in other words, an application) that enables communication between the MR user 6 and the VR user. Specifically, an MR user 6 who is in a conference room as a predetermined real space and a VR user who is located away from the conference room have a meeting while simultaneously viewing an object such as a product mockup as a virtual object. An example will be described in which the configuration according to the present embodiment is applied to a service that enables the following. Note that the configuration according to this embodiment can also be applied to, for example, a system that allows students to take school classes from home. Furthermore, the configuration according to the present embodiment can also be applied to a system that allows a user to participate in a home party held at a specific house from another location. Note that the application of the configuration according to this embodiment is not limited to these services. Further, in the following description, it is assumed that a plurality of detection devices 460 are installed in advance at predetermined positions in the conference room, as shown in FIG. 8 . Further, the description will be made assuming that three MR users 6, MR user 6A, MR user 6B, and MR user 6C, exist in the conference room. Note that, hereinafter, the avatar object of the VR user will be referred to as the VR avatar 25. Further, the avatar object of the MR user 6 is called an MR avatar 26. Specifically, the avatar object of MR user 6A is called MR avatar 26A, the avatar object of MR user 6B is called MR avatar 26B, and the avatar object of MR user 6C is called MR avatar 26C.

Referring to FIG. 8, the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 (in other words, the mixed reality space 21) and the view provided to the MR users 6A, 6B, and 6C in the real conference room 21 The field of view provided to the VR user entering the virtual space 11 will be explained. FIG. 8 is a schematic diagram showing the state of the actual conference room 21 and the state of the virtual space 11 when the MR users 6A, 6B, 6C and the VR user are using the service. The upper part of FIG. 8 shows the state of the actual conference room 21, and the lower part of FIG. 8 shows the state of the virtual space 11. In addition, in FIG. 8, the VR avatar 25 is wearing the VR device 210, but this is to make the explanation easier to understand, and in reality, the VR avatar 25 is wearing the VR device 210 (virtual VR device 210) may not be attached. Further, in FIG. 8, the MR avatar 26 is not wearing the MR device 410 (virtual MR device 410), but the MR avatar 26 may be wearing the MR device 410.

As shown in FIG. 8, a desk 8 and the like as real objects are arranged in the conference room 21. Furthermore, three detection devices 460 are arranged in the conference room 21. Furthermore, three MR users 6A, 6B, and 6C are present in the conference room 21. On the other hand, there are no VR users in the conference room 21. Here, on the display 411 of the MR apparatus 410 used by each of the MR users 6A, 6B, and 6C, a VR avatar 25 and a mock-up virtual object 30 (hereinafter also referred to as "virtual model 30") are displayed as virtual objects. ) is displayed. That is, in the MR device 410, the VR avatar 25 and the virtual model 30 are displayed in the real conference room 21 that can be seen through the transparent display 411. Therefore, the MR users 6A, 6B, and 6C are provided with a view as if the VR avatar 25 and virtual model 30, which do not actually exist in the conference room 21, were present in the conference room 21.

Further, the virtual space 11 that the VR user views via the VR device 210 imitates the conference room 21 and includes virtual objects 31 such as the desk 8 that exist in the actual conference room 21 . Further, in the virtual space 11, MR avatars 26A, 26B, 26C of the MR users 6A, 6B, 6C, a virtual model 30, etc. are arranged. Therefore, the VR user is provided with a view as if he or she were in the conference room 21 where the MR users 6A, 6B, and 6C are present.

In this way, in the information processing system 100 of the present embodiment, the MR user 6 can visually recognize other MR users 6 and real objects actually existing at the same location through the transparent display 411. Further, from the viewpoint of the MR user 6, the VR avatar 25 of the VR user appearing as a virtual object can be visually recognized. Therefore, the MR user 6 can communicate with the VR user, other MR users 6, etc. without losing the feeling of being in the real world. Further, the VR user is in a different location from the MR user 6 in the real world, and in such a case, the shape of the room etc. is often different between the VR user's location and the MR user 6's location. For this reason, even on the VR user side, if the MR device 410 or the like is used to display the MR avatar 26 in real space, there is a risk that the display will look unnatural. In this embodiment, the VR user can enter the virtual space 11 and communicate with the MR user 6 and the like, so it is possible to prevent such discomfort from occurring. In addition, since the virtual space 11 provided to the VR user is a simulation of the location where the MR user 6 is, when the VR user and the MR user 6 move around in the virtual space 11 or the real space, respectively, or move virtual objects, In addition, the movements of the avatar, virtual model 30, etc. displayed to the other party become natural. Furthermore, when operating a virtual object such as the virtual model 30, the VR user or MR user 6 can easily imagine how it will appear to the other party, thereby facilitating smooth communication.

The functional configuration of the information processing system 100 will be described below with reference to FIG. 9. Note that the functional configuration shown in FIG. 9 is only an example. Each of the VR system 200, MR system 400, and server 600 may have at least some of the functions provided by other devices. In other words, the computer 300, the computer 500, the server 600, or another device may include some or all of the functional blocks included in each of the computer 300, the computer 500, and the server 600 in this embodiment. Further, each device such as the computer 300, the computer 500, and the server 600 does not need to be realized by an integrated device, and may be realized by, for example, a plurality of devices connected via a network or the like. .

Further, in this embodiment, the processor 301, the processor 501, or the processor 601 will be described as performing each process described below by executing a program stored in the information processing system 100. However, at least a portion of the processing to be described later and performed by the processor 301 may be executed by a processor other than the processor 301. Further, at least a portion of the processing to be described later which is performed by the processor 501 may be executed by a processor other than the processor 501. Further, at least a part of the processing to be described later which is performed by the processor 601 may be executed by a processor other than the processor 601. In other words, the computer that executes the program in this embodiment may be any computer including the computer 300, the computer 500, and the server 600, or may be realized by a combination of multiple devices.

FIG. 9 is a block diagram showing the functional configuration of the information processing system 100. As shown in FIG. 9, the computer 300 (in other words, the VR system 200) has a control unit 310, a storage unit 311, functions as Further, the computer 500 (in other words, the MR system 400) functions as a control unit 510 and a storage unit 511 through the cooperation of a processor 501, a memory 502, a storage 503, an input/output interface 504, and a communication interface 505. Further, the server 600 functions as a control unit 610 and a storage unit 611 through cooperation of a processor 601, a memory 602, a storage 603, an input/output interface 604, and a communication interface 605.

The control unit 510 of the MR system 400 includes a virtual space generation unit 810, a coordinate definition unit 812, an MR side input reception unit 816, a user information acquisition unit 818, an MR side object control unit 820, and a display control unit 840. , a sound control section 845, and a communication control section 850.

The virtual space generation unit 810 generates a virtual space (in other words, virtual space data representing the virtual space) based on a predetermined real space. In other words, the virtual space generation unit 810 generates a virtual space imitating a predetermined real space. Here, the predetermined real space is a predetermined place that exists in the real world, and may be, for example, a conference room or a specific room such as a school classroom. Further, the predetermined real space may be, for example, a school or a specific building such as a house. Further, the predetermined real space may be, for example, a specific town. In other words, the predetermined real space does not have to be a space partitioned off by a wall or the like. Moreover, imitating a predetermined real space may mean that the generated virtual space has a structure similar to the basic structure of the real space. Specifically, for example, in a virtual space that imitates a conference room in the real world, the shape of the virtual room as a virtual space (for example, the shape of walls, floors, etc.) approximately matches the shape of the conference room in the real world. It may be a virtual object in which virtual objects corresponding to real objects such as desks and chairs existing in a conference room in the real world are arranged. At this time, virtual objects such as desks, chairs, walls, floors, etc. do not need to be exact copies of the form of real objects; for example, they may have simplified shapes, patterns, colors, etc. Good too. Furthermore, for example, real objects may include objects that do not appear in the virtual world (real objects for which there is no corresponding virtual object), such as posters pasted on walls or small items placed on desks, floors, etc. .

In this embodiment, as described above, the predetermined real space is a specific conference room 21, and the virtual space 11 imitating the specific conference room 21 is generated.

As for the method of generating a virtual space imitating a real space, a known method can be used, and the method is not particularly limited, but for example, the following method may be used. That is, the virtual space generation unit 810 generates the virtual space 11 based on information from a sensor that can detect the shapes and positions of real objects such as walls, floors, desks, and chairs that make up the real space. Specifically, the virtual space generation unit 810 generates data from an image sensor included in the detection device 460 or the MR device 410 (for example, an image sensor that acquires an RGB image, an image sensor that acquires a monochrome image, or a depth sensor). A virtual space may be generated by acquiring three-dimensional information of the real space based on the information. Further, the virtual space generation unit 810 may use information from an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, or the like to generate the virtual space 11, for example.

The virtual space generation unit 810 stores data indicating the generated virtual space 11 (hereinafter referred to as “virtual space data”) in the storage unit 611 as a virtual space data storage unit, for example.

Note that the virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in advance before the VR user or MR user 6 starts using the service. Furthermore, the virtual space generation unit 810 may generate virtual space data indicating the virtual space 11 in real time while the VR user or MR user 6 is using the service, for example.

The coordinate definition unit 812 defines coordinates in the real space (in other words, coordinates in the mixed reality space 21; hereinafter referred to as "mixed reality coordinates"). Further, the coordinate definition unit 812 defines coordinates of the virtual space 11 (hereinafter referred to as "virtual space coordinates").

Mixed reality coordinates have, for example, three mutually orthogonal axes: x-axis, y-axis, and z-axis. The virtual space coordinates have, for example, three mutually orthogonal axes: an X-axis, a Y-axis, and a Z-axis. For each point in the virtual space 11 (in other words, the virtual conference room) that corresponds to (in other words, indicates the same point) each point in the real space (in other words, the real conference room 21), the coordinate definition unit 812 Each coordinate is defined (in other words, they are associated) so that the mixed reality coordinates indicating each point in the real space and the virtual space coordinates indicating each point in the virtual space have a one-to-one correspondence. The coordinate definition unit 812 defines virtual space coordinates associated with mixed reality coordinates, for example, when the virtual space generation unit 810 generates a virtual space imitating a real space. At this time, the coordinate definition unit 812 may associate the mixed reality coordinates and the virtual space coordinates based on the feature points of the conference room detected by the detection device 460, the MR device 410, or the like, for example. Further, the coordinate definition unit 812 may associate the mixed reality coordinates and the virtual space coordinates based on, for example, the position of the detection device 460 or the position of a marker installed in advance in the conference room.

Further, the coordinate definition unit 812 stores information regarding the correspondence between mixed reality coordinates and virtual space coordinates (in other words, the correspondence between each point in the real space and each point in the virtual space) as a correspondence storage unit. The information is stored in the storage unit 611.

Note that a known method can be used as a method for defining mixed reality coordinates and virtual space coordinates, and there are no particular limitations.

Note that the timing of defining mixed reality coordinates and the timing of defining virtual space coordinates (in other words, the timing of associating mixed reality coordinates and virtual space coordinates) may be, for example, when the virtual space generation unit 810 generates a virtual The timing may be when a space is generated, or the timing when the MR device 410 starts displaying a virtual object.

Note that the generation of the virtual space, the definition of each coordinate, etc. are performed in advance by a dedicated device as the external device 700 that can scan the real space with high precision, and the generated data is stored in the storage unit 611. You can also leave it there.

The MR side input receiving unit 816 receives input from the MR user 6. In other words, the MR side input reception unit 816 acquires input information from the MR user 6. Specifically, the MR side input reception unit 816 includes the detection device 460, the controller 480, the camera 413, the device worn by the MR user, the gaze sensor 412, the sensor 417, the sensor 486, and the detection device 460 and the MR device 410. The output of the provided image sensor or the like is accepted as input from the MR user 6. In other words, the MR-side input receiving unit 816 detects the motion of the MR user 6 (for example, the detection device 460, the controller 480, the camera 413, the gaze sensor 412, the sensor 417, the sensor 486, and the detection device 460 or the MR The output of an image sensor (such as an image sensor included in the device 410) is received as an input from the MR user 6. These devices and sensors are for acquiring information regarding the movement of the MR user 6, which is used to control virtual objects in the virtual space 11 and the mixed reality space 21, as will be described later. The user 6 can input information for moving the virtual object etc. to the MR system 400 by his/her own movements.

Specifically, the MR-side input reception unit 816 may receive, for example, a camera serving as the detection device 460 or an output from the camera 413 as an input from the MR user 6. More specifically, the MR side input reception unit 816 indicates the position of the MR user 6 in the mixed reality space 21, which is detected by image recognition from images taken by image sensors included in each of these cameras, for example. The position information may be received as input from the MR user 6. In other words, the MR side input receiving unit 816 may receive an output from a position detection means for detecting the position of the MR user 6 in the mixed reality space 21 as an input from the MR user 6. Note that the position detecting means, for example, performs wireless communication (for example, Wi-Fi communication) between the MR device 410 and a predetermined device (for example, the detection device 460, a beacon (not shown), etc.) of the MR system 400. , Bluetooth communication, or UWB communication), or positioning using a GPS (Global Positioning System) sensor included in the MR device 410, to detect the position of the MR user 6 and obtain the position information. etc. may be used. Note that the position information indicating the position of the MR user 6 can also be said to be information regarding the movement of the MR user 6.

Further, the MR side input reception unit 816 receives, from the MR user 6, information regarding the movement of the MR user 6, which is detected by image recognition from an image taken by a camera as the detection device 460 or an image sensor included in the camera 413, for example. may be accepted as input. For example, the MR side input reception unit 816 receives information regarding the hand movements of the MR user 6 detected by image recognition, information regarding the inclination of the MR user 6, and the direction in which the MR user 6 is facing, as input from the MR user 6. Good too.

Further, the MR side input reception unit 816 may be configured to include, for example, a device worn by the MR user 6 (for example, a watch-type, wristband-type, ring-type, or clothing-type wearable device, or a device implanted in the body). Information regarding the movement of the MR user 6 obtained using a device that is capable of tracking the movement of the MR user 6 (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication, etc.) may be accepted as an input from the MR user 6.

Further, the MR side input reception unit 816 may accept, for example, information indicating the operation of the MR user 6 detected by the controller 480 (in other words, information regarding the movement of the MR user 6) as an input from the MR user 6. good.

The MR side input receiving unit 816 also receives outputs related to the inclination and facing direction of the MR device 410 from the detection device 460 or the sensors (for example, acceleration sensor, angular velocity sensor, geomagnetic sensor, etc.) included in the MR device 410. Data (in other words, information regarding the movements of the MR user 6) may be accepted as input from the MR user 6.

The user information acquisition unit 818 acquires information regarding the MR user 6 existing in the mixed reality space 21. Specifically, for example, the user information acquisition unit 818 acquires identification information that allows each MR user 6A, 6B, and 6C in the mixed reality space 21 to be identified (in other words, identification information that allows each MR device 410 to be identified). ) to obtain. For example, the user information acquisition unit 818 communicates with the MR device 410 worn by each user, and receives information from the MR device 410 about the user using the MR device 410 (for example, who is logged in to the MR device 410? account information, etc.) may be acquired as the identification information. Further, for example, the user information acquisition unit 818 communicates with the MR device 410 worn by each user, and acquires information unique to each MR device 410 that allows each MR device 410 to be identified from the MR device 410. It may be obtained as information. Further, for example, the user information acquisition unit 818 identifies the MR user 6 in the mixed reality space 21 by image recognition from an image captured by an image sensor included in the detection device 460 or the MR device 410, and obtains the identification information. You may obtain it. Further, the user information acquisition unit 818 associates the information acquired by the MR side input reception unit 816 with the identification information. In other words, the user information acquisition unit 818 performs control so that when information regarding the movement of the MR user 6 is used in controlling a virtual object, which will be described later, it is possible to determine which MR user 6's information belongs.

The MR side object control unit 820 controls virtual objects in the mixed reality space 21.

The MR side object control unit 820 arranges a virtual object in the mixed reality space 21. Specifically, the MR side object control unit 820 arranges, for example, the VR user's VR avatar 25, an object to which a predetermined change is given by at least one of the VR user or the MR user 6, and the like. In this embodiment, as shown in FIG. ) are placed in the mixed reality space 21.

For example, the MR-side object control unit 820 places a virtual object corresponding to the virtual object at a position in the mixed reality space 21 that corresponds to a position in the virtual space 11 where the virtual object is placed. Here, the virtual object corresponding to the virtual object, that is, the virtual object in the mixed reality space 21 corresponding to the virtual object in the virtual space 11, is, for example, a virtual object having the same form as the virtual object in the virtual space 11 ( In other words, they may represent the same object) or may have different forms. In this embodiment, a virtual model 30 having the same form as the virtual model 30 in the virtual space 11 is arranged in the mixed reality space 21 . Further, in the mixed reality space 21, a VR avatar 25 having the same form as the VR avatar 25 in the virtual space 11 is arranged.

Note that the arrangement of the VR avatar 25 in the mixed reality space 21 can be said as follows. That is, it can be said that the MR-side object control unit 820 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR user in the virtual space 11. Here, the position of the VR user in the virtual space 11 is, for example, the position of the VR avatar 25 in the virtual space 11. In addition, in content where the VR user enters the virtual space 11 from a first-person perspective, if the VR user cannot see himself (VR avatar 25) (specifically, if only a part of the hand, etc. However, even in such a case, the VR user in the virtual space 11 recognized by the computer 300 can be said to be the VR avatar 25, and the position of the VR user in the virtual space 11 recognized by the computer 300 is This can be said to be the position of the VR avatar 25 in the virtual space.

Note that the appearance (in other words, form) of the VR avatar 25 may or may not imitate the appearance of the VR user. For example, the appearance of the VR avatar 25 may be one that imitates the appearance of a predetermined animal. Further, the VR avatar 25 placed in the virtual space 11 and the VR avatar 25 placed in the mixed reality space 21 may have different appearances. According to such a configuration, the VR user can display his/her own avatar in the mixed reality space 21 while keeping the form of his or her avatar in the virtual space 11 hidden. Note that the appearance of the VR avatar 25 placed in the virtual space 11 may be the appearance visually recognized by the VR user who operates the VR avatar 25, and the appearance of the VR avatar 25 placed in the virtual space 11 may be the appearance visually recognized by the VR user who operates the VR avatar 25. It may also be an appearance that is visible to other VR users operating other VR avatars 25 that exist in the VR avatar 25 .

Further, the MR-side object control unit 820 may arrange the virtual object in the mixed reality space 21 as follows, for example.

That is, for example, when placing the VR avatar 25, the MR-side object control unit 820 arranges the mixed reality space 21 based on position information indicating the position of the VR user (in other words, the VR avatar 25) in the virtual space 11. Decide where to place it inside. Specifically, for example, when the VR avatar 25 is placed in the virtual space 11, the VR-side object control unit 920, which will be described later, determines the position of the VR avatar 25 in the virtual space 11 (for example, coordinate values of virtual space coordinates). The information is transmitted to the MR side object control unit 820 via the server 600. Based on the information, the MR side object control unit 820 determines the position in the mixed reality space 21 corresponding to the position of the VR user in the virtual space 11 (for example, the mixed reality corresponding to the coordinate values of the virtual space coordinates). The VR avatar 25 is placed at the coordinate value position). As a result, the VR avatar 25 is placed at the same position in the conference room in the virtual space 11 and the mixed reality space 21.

In other words, when placing a virtual object (for example, the VR avatar 25 or the virtual model 30, etc.) in the mixed reality space 21, the MR side object control unit 820 controls the virtual object ( For example, the position in the mixed reality space 21 to place the VR avatar 25 or the virtual model 30, etc.) is determined based on the position information indicating the position (for example, the position information transmitted by the VR-side object control unit 920). You may.

Further, for example, when placing a virtual object in the mixed reality space 21, the MR side object control unit 820 determines which position in the mixed reality space 21 it should be placed based on information detected by a predetermined sensor of the MR system 400. You may decide where to place it. Specifically, for example, a predetermined marker is installed in advance in the real space, and the MR side object control unit 820 places the marker at the position photographed by the image sensor included in the detection device 460 or the MR device 410. A virtual model 30 as a virtual object may be arranged. Further, the MR-side object control unit 820 may place the virtual model 30 as a virtual object at a position specified by the MR user 6. The instruction may be given by a predetermined gesture or the like (for example, a gesture pointing to a point where the virtual model 30 is to be placed), or by placing the marker. Further, the MR side object control unit 820 detects a flat surface in the mixed reality space 21 from an image captured by an image sensor included in the detection device 460 or the MR device 410, and arranges the virtual model 30 on the flat surface. You may also do this.

Note that when placing a virtual object in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the virtual object, and places the virtual object based on the object data. Here, the MR-side object control unit 820 may receive object data from, for example, the control unit 610 of the server 600 or the VR-side object control unit 920 of the VR system 200. Specifically, when arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the VR avatar 25 from the control unit 610, and receives the object data indicating the form of the object data. The VR avatar 25 may be placed in the mixed reality space 21. Furthermore, when arranging the VR avatar 25 in the mixed reality space 21, the MR side object control unit 820 receives object data indicating the form of the VR avatar 25 from the VR side object control unit 920, and receives the form indicated by the object data. The VR avatar 25 may be placed in the mixed reality space 21.

Display control unit 840 controls image display on display 411 of MR apparatus 410. The display control unit 840 generates an image for displaying the virtual object in the mixed reality space 21 placed by the MR side object control unit 820 at the position placed by the MR side object control unit 820. Furthermore, the display control unit 840 causes the display 411 to display the image. As a result, the MR user 6 is provided with a view of the mixed reality space in which the virtual object is placed at a desired position in the real space.

Further, the display control unit 840 controls image display on the display 470. For example, the display control unit 840 causes the display 470 to display an image similar to the image displayed on the display 411. Note that the display control unit 840 may cause the display 470 to display an image showing a view of the mixed reality space that the MR device 410 provides to the MR user 6.

When the utterance of the MR user 6 is detected by the microphone 415 of the MR device 410, the sound control unit 845 acquires sound data corresponding to the utterance. Further, the sound control unit 845 transmits the acquired sound data to the computer 300 etc. via the network 2. Furthermore, upon receiving sound data from the computer 300 via the network 2, the sound control unit 845 outputs sound (speech) corresponding to the sound data from the speaker 416. This allows the MR user 6 to communicate with the VR user by telephone, for example. Note that the MR-side input receiving unit 816 may receive sound data related to the utterance of the MR user 6 detected by the microphone 415 as an input from the MR user 6.

The communication control unit 850 can communicate with the server 600, the computer 300, and other information communication devices via the network 2. Communication control unit 850 transmits information used by server 600 or computer 300 to server 600 or computer 300, for example. Further, the communication control unit 850 receives information used by the computer 500 from the server 600 or the computer 300, for example.

The control unit 310 of the VR system 200 includes a VR side input reception unit 916, a VR side object control unit 920, a virtual camera control unit 930, a display control unit 940, a sound control unit 945, a communication control unit 950, Equipped with

The VR-side input reception unit 916 receives input from a VR user. In other words, the VR side input reception unit 916 acquires input information from the VR user. Specifically, the VR side input reception unit 916 includes the detection device 260, the controller 280, the first camera 213, the second camera 214, the wearable device worn by the VR user, the gaze sensor 212, the sensor 217, the sensor 286, and The output of the detection device 260, the image sensor, etc. included in the VR device 210 is accepted as input from the VR user. In other words, the VR side input reception unit 916 uses detection means for detecting the movement of the VR user (for example, the detection device 260, the controller 280, the first camera 213, the second camera 214, the wearable device worn by the VR user, the gaze The outputs of the sensors 212, 217, 286, image sensors included in the detection device 260 and the VR device 210, etc.) are accepted as inputs from the VR user. As described later, these devices and sensors are for acquiring information regarding the movements of the VR user, which is used to control virtual objects, etc. in the virtual space 11 and the mixed reality space 21. It is possible to input information for moving the virtual object etc. to the VR system 200 by the user's own movements.

Specifically, the VR side input reception unit 916 may accept, for example, an output from a camera as the detection device 260 or a camera included in the VR device 210 that photographs the surroundings of the VR device 210 as an input from a VR user. good. More specifically, the VR side input reception unit 916 inputs position information indicating the position of the VR user in real space, which is detected by image recognition from images taken by image sensors included in each of these cameras, to the VR user. It may also be accepted as input from In other words, the VR-side input reception unit 916 may receive an output from a position detection means that detects the position of the VR user in real space as an input from the VR user. Note that the position detection means, for example, performs wireless communication (for example, Wi-Fi communication) between the VR device 210 and a predetermined device of the VR system 200 (for example, the detection device 260, a beacon (not shown), etc.). , Bluetooth communication, or UWB communication), or one that detects the position of the VR user based on positioning using a GPS (Global Positioning System) sensor included in the VR device 210, and acquires the position information. It may be. Note that the position information indicating the position of the VR user can also be said to be information regarding the movement of the VR user.

Further, the VR-side input reception unit 916 may be configured to receive a VR image detected by image recognition from an image captured by a camera as the detection device 260 or an image sensor included in a camera that captures the surroundings of the VR device 210 included in the VR device 210. Information regarding the user's movements may be accepted as input from the VR user. For example, the VR-side input reception unit 916 may receive information regarding the movement of the VR user's hand detected by image recognition, or information regarding the tilt or direction of the VR user as input from the VR user.

Further, the VR-side input reception unit 916 is configured to input, for example, a device worn by a VR user (for example, a wearable device such as a watch, wristband, ring, or clothing type, or a device implanted in the body). , information regarding the user's movement obtained using a device that is capable of tracking the movement of the VR user (and a device that tracks the movement of the MR user 6 in cooperation with the device through short-range wireless communication, etc.) is transmitted to the VR user. It may also be accepted as input from

Further, the VR-side input reception unit 916 may receive, for example, information indicating the VR user's operation detected by the controller 280 (in other words, information regarding the VR user's movement) as an input from the VR user.

Further, the VR side input reception unit 916 may output, for example, an output from the detection device 260 or a sensor (for example, an acceleration sensor, an angular velocity sensor, a geomagnetic sensor, etc.) included in the VR device 210 regarding the inclination or facing direction of the VR device 210. Data (in other words, information regarding the movements of the VR user) may be accepted as input from the VR user.

The control unit 310 controls the virtual space 11. The control unit 310 acquires virtual space data indicating the virtual space 11 stored in the storage unit 611, and defines the virtual space 11 to be provided to the VR user based on the virtual space data.

The VR-side object control unit 920 controls virtual objects in the virtual space 11.

The VR-side object control unit 920 places a virtual object in the virtual space 11 indicated by the virtual space data. Object data indicating a virtual object is stored in, for example, the storage unit 611 or the storage unit 311 as an object data storage unit. Then, the VR-side object control unit 920 places the virtual object in the virtual space 11 using the object data.

Specifically, the VR-side object control unit 920 controls the MR avatars 26A, 26B, and 26C of the MR users 6A, 6B, and 6C, and at least one of the VR users and the MR users 6 to create a predetermined object in the virtual conference room. A virtual object etc. that can be given a change is placed. In the present embodiment, the VR-side object control unit 920 controls a mock-up virtual object 30 (hereinafter also referred to as "virtual model 30") that is moved by the VR avatar 25, the MR avatar 26, the VR user, and the MR user 6. ) are placed in the virtual space 11. Note that virtual objects 31 and the like corresponding to the desk 8 as a real object are also arranged in the virtual space 11.

Note that, for example, when the VR system 200 (in other words, computer 300) used by a VR user is connected to the VR system 200 (in other words, computer 300) used by another VR user via the network 2, the VR The side object control unit 920 may arrange the VR avatar 25 of the other VR user in the virtual space 11. The other VR user's VR avatar 25 can be operated by the other VR user.

The VR-side object control unit 920 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21, for example. That is, the VR-side object control unit 920 determines the positions of the MR users 6A, 6B, and 6C in the real conference room 21, and the positions of the MR avatars 26A, 26B, and 26C in the virtual conference room. MR avatars 26A, 26B, and 26C are arranged so that they are at the same position.

Note that the appearance (in other words, form) of the MR avatar 26 may or may not be an imitation of the appearance of the MR user 6. For example, the appearance of the MR avatar 26 may be one that imitates the appearance of a predetermined animal. When placing a virtual object in the mixed reality space 21, the VR-side object control unit 920 receives object data indicating the form of the virtual object, and places the virtual object based on the object data. Here, the VR-side object control unit 920 may receive object data from the control unit 610 of the server 600 or from the MR-side object control unit 820 of the MR system 400. Specifically, for example, when placing the MR avatar 26 in the virtual space 11, the VR side object control unit 920 receives object data indicating the form of the MR avatar 26 from the control unit 610 or the MR side object control unit 820. After receiving the object data, the MR avatar 26 having the form indicated by the object data may be placed in the virtual space 11. The appearance of the MR avatar 26 can be set in advance by the MR user 6, and when the MR avatar 26 is displayed on the VR device 210, it is displayed with the appearance set by the MR user 6. It may be possible to do so. Further, when the appearance of the MR avatar 26 is one that imitates the appearance of the MR user 6, the VR side object control unit 920 detects the MR user's The MR avatar 26 generated based on the image No. 6 (for example, generated by the control unit 510) may be placed in the virtual space 11.

The virtual camera control unit 930 arranges the virtual camera 14 in the virtual space 11. Further, the virtual camera control unit 930 controls the position of the virtual camera 14 in the virtual space 11 and the inclination (orientation) of the virtual camera 14. Note that in this embodiment, the virtual camera control unit 930 arranges the virtual camera 14 at the eye position of the VR avatar 25 in the virtual space 11. In other words, the virtual camera control unit 930 links the position of the virtual camera 14 with the position of the VR avatar 25, and moves the virtual camera 14 within the virtual space 11 when the VR avatar 25 moves within the virtual space 11. Note that the position of the virtual camera 14 does not need to be linked to the position of the VR avatar 25. For example, it may be possible to move the VR avatar 25 within the virtual space 11 while keeping the virtual camera 14 fixed at a predetermined position.

The display control unit 940 controls image display on the display 211 of the VR device 210. The display control unit 940 defines the viewing area 15 according to the position and tilt of the virtual camera 14 (in other words, the tilt of the head of the VR user wearing the VR device 210). Further, the display control unit 940 generates the visual field image 17 displayed on the display 211 based on the defined visual field area 15. The view image 17 generated by the display control unit 940 is output to the VR device 210.

Further, the display control unit 940 controls image display on the display 270. For example, the display control unit 940 causes the display 270 to display an image similar to the image displayed on the display 211.

When the sound control unit 945 detects an utterance from the VR user using the microphone 215 from the VR device 210, it acquires sound data corresponding to the utterance. Further, the sound control unit 945 transmits the acquired sound data to the computer 500, the computer 300 of another VR user, etc. via the network 2. Furthermore, upon receiving sound data from the computer 500 or another user's computer 300 via the network 2, the sound control unit 945 outputs sound (speech) corresponding to the sound data from the speaker 216. This allows the VR user to communicate with the MR user 6 by telephone, for example. Note that the VR-side input reception unit 916 may accept sound data related to the VR user's utterances detected by the microphone 215 as input from the VR user.

The communication control unit 950 can communicate with the server 600, the computer 500, the computer 300 of another VR user, and other information communication devices via the network 2. The communication control unit 950 transmits information used by the server 600, the computer 500, or the other VR user's computer 300 to the server 600, the computer 500, or the other VR user's computer 300, for example. Further, the communication control unit 850 receives information used by the computer 300, for example, from the server 600, the computer 500, or another VR user's computer 300.

(Fusion of mixed reality space and virtual space)
In this embodiment, the virtual object displayed by the MR device 410 and the virtual object displayed by the VR device 210 are controlled based on the input from the MR user 6 acquired by the MR side input reception unit 816. ing.

Specifically, the MR side object control unit 820 controls the virtual object in the mixed reality space 21, specifically, the virtual model 30, based on the input from the MR user 6 received by the MR side input reception unit 816. give a change of Here, predetermined changes include complex changes such as moving the virtual object, changing the inclination or direction of the virtual object, and changing the relative positional relationship between the parts of the virtual object. It may also be something that moves a virtual object in the real space 21. Furthermore, the predetermined change may include changing the color of the virtual object, changing information displayed by the virtual object, or changing the form of the virtual object.

More specifically, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on, for example, information regarding the movement of the MR user 6 received by the MR-side input reception unit 816. Specifically, for example, when a gesture of the MR user 6 carrying the virtual model 30 is detected by the detection means (for example, the detection device 460 or the camera 413, etc.), and the detection means outputs an output related to the gesture, the MR side The input receiving unit 816 receives the output as an input from the MR user 6. Furthermore, the MR-side object control unit 820 moves the virtual model 30 in the mixed reality space 21 based on the input received by the MR-side input reception unit 816 and in accordance with the instruction from the MR user 6 using the gesture. Note that the gesture of the MR user 6 that causes a predetermined change to the virtual model 30 as a virtual object does not have to be a gesture that touches the virtual model 30 in the mixed reality space 21 . For example, it may be possible to rotate the virtual object by holding up a hand toward the virtual model 30 and waving the hand sideways.

Further, the MR-side object control unit 820 controls the virtual object based on the information regarding the operation of the MR user 6 on the controller 480 (in other words, the information regarding the movement of the MR user 6) received by the MR-side input reception unit 816, for example. It is also possible to give a predetermined change to. Further, the MR-side object control unit 820 may, for example, give a predetermined change to the virtual object based on the sound data regarding the utterance of the MR user 6, which is received by the MR-side input reception unit 816.

Further, the VR-side object control unit 920 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 that the MR-side input reception unit 816 receives.

Specifically, the VR-side object control unit 920 controls, for example, changes similar to the changes given to the virtual model 30 in the mixed reality space 21 based on the input from the MR user 6 received by the MR-side input reception unit 816. is given to the virtual model 30 in the virtual space 11. That is, for example, as described above, when the virtual model 30 in the mixed reality space 21 moves due to the gesture of the MR user 6 carrying the virtual model 30, the VR side object control unit 920 moves the virtual model 30 in the virtual space 11. The model 30 is moved in the same manner.

Note that the method of applying a change similar to the change applied to the virtual model 30 in the mixed reality space 21 to the virtual model 30 in the virtual space is not particularly limited, but may be, for example, as follows. . That is, for example, when the MR side object control unit 820 moves the virtual model 30 (in other words, the virtual object) in the mixed reality space 21 based on the input from the MR user 6 (in other words, at a predetermined timing), the VR Information indicating the position and inclination of the virtual model 30 after movement (in other words, information regarding the state of the virtual model 30) may be sent to the side object control unit 920. Then, the VR-side object control unit 920 may arrange the virtual model 30 at a position indicated by the information in the virtual space 11, or may arrange the virtual model 30 at an inclination indicated by the information. For example, the communication control unit 850 may also transmit information based on input from the MR user 6 (for example, the amount of movement, movement trajectory, and inclination of the virtual model 30 calculated (in other words, acquired) by the control unit 510 based on the input). etc.) to the communication control unit 950, and the VR-side object control unit 920 may move a virtual object such as the virtual model 30 based on the information. That is, in the present embodiment, the VR side object control unit 920 gives a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 accepted by the MR side input reception unit 816. "Based on the input from the MR user 6" does not mean that the VR side object control unit 920 itself receives the input and controls the virtual object, but the virtual object in the virtual space 11 responds to the input as a result. Any material that makes a predetermined change may be used.

Further, the VR-side object control unit 920 moves the MR avatar 26 in the virtual space 11 based on the input from the MR user 6 received by the MR-side input reception unit 816, for example. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information regarding the movement of the MR user 6 received by the MR-side input reception unit 816. For example, the VR-side object control unit 920 controls the MR avatar 26 based on position information indicating the position of the MR user 6 in the mixed reality space 21, which the MR-side input reception unit 816 receives as input from the MR user 6. move. That is, the VR-side object control unit 920 places the MR avatar 26 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21 based on the position information, and the MR user 6 When moving in the real space 21, the MR avatar 26 is placed at a position in the virtual space 11 corresponding to the position after movement.

Further, the VR side object control unit 920 also controls the movement of each part of the body of the MR user 6 (for example, hand movement, head movement) that the MR side input reception unit 816 receives as input from the MR user 6. The MR avatar 26 is moved based on information regarding the MR user 6's inclination and the direction in which the MR user 6 is facing. That is, the VR-side object control unit 920 controls the movement of each part of the body of the MR user 6 (in other words, the posture), the inclination and direction of the MR user 6 in the virtual space, based on these pieces of information. Reflect it on MR avatar 26 in 11. That is, in the present embodiment, the VR-side object control unit 920 determines that the movement of the MR avatar 26 in the virtual space 11 imitates the movement of the MR user 6 detected by the detection means that detects the movement of the MR user 6. The movement of the MR avatar 26 is controlled so that the movement of the MR avatar 26 is controlled. In other words, the VR-side object control unit 920 causes the movement of the MR avatar 26 to be linked to the movement of the MR user 6 for at least part of the body (for example, hands, feet, head, eyes, mouth, etc.). The movement of the MR avatar 26 is controlled as follows.

Note that at least part of the movement of the MR avatar 26 may be realized by the MR user 6's operation on the controller 480 (for example, operation on an analog stick, button, etc.). That is, for example, the VR side object control unit 920 may move a part of the body of the MR avatar 26 based on information indicating the operation of the MR user 6 on the controller 480 as an input from the MR user 6. good.

Note that at least one of the position or movement of the MR avatar 26 in the virtual space 11 may not be linked to the position or movement of the MR user 6 in the mixed reality space 21. That is, for example, even if the MR user 6 moves in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not move accordingly. Further, for example, even if the MR user 6 moves his hand in the mixed reality space 21, the MR avatar 26 in the virtual space 11 may not make the same movement.

Furthermore, in this embodiment, the virtual object displayed by the VR device 210 and the virtual object displayed by the MR device 410 are controlled based on the input from the VR user acquired by the VR side input reception unit 916. It has become.

Specifically, the VR-side object control unit 920 causes the virtual object in the virtual space 11, specifically, the virtual model 30, to undergo a predetermined change based on the input from the VR user that the VR-side input reception unit 916 has received. give.

More specifically, the VR-side object control unit 920, for example, based on information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movements) received by the VR-side input receiving unit 916. , gives a predetermined change to the virtual model 30 in the virtual space 11. Specifically, for example, the VR-side object control unit 920 moves the virtual model 30 in the virtual space 11 in accordance with an instruction from the VR user by operating the controller 280.

Further, the VR-side object control unit 920 may, for example, give a predetermined change to the virtual object based on the gesture of the VR user detected by the detection means. Further, the VR-side object control unit 920 may, for example, give a predetermined change to the virtual object based on the sound data related to the VR user's utterances, which is received by the VR-side input reception unit 916.

Further, the MR side object control unit 820 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user that the VR side input reception unit 916 has received.

Specifically, the MR side object control unit 820, for example, causes a change similar to the change given to the virtual model 30 in the virtual space 11 based on the input from the VR user accepted by the VR side input reception unit 916. It is given to the virtual model 30 in the mixed reality space 21. That is, for example, as described above, when the virtual model 30 in the virtual space 11 is moved by the VR user's operation on the controller 280, the MR side object control unit 820 moves the virtual model 30 in the mixed reality space 21 to the same position. Move it like this.

Note that there is no particular limitation on the method of applying the same change to the virtual model 30 in the virtual space 11 to the virtual model 30 in the mixed reality space 21, but for example, the following method may be used. good. That is, for example, when the VR side object control unit 920 moves the virtual model 30 (in other words, a virtual object) in the virtual space 11 based on an input from a VR user (in other words, at a predetermined timing), the MR side object Information indicating the position and inclination of the virtual model 30 after movement (in other words, information regarding the state of the virtual model 30) may be sent to the control unit 820. Then, the MR side object control unit 820 may arrange the virtual model 30 at a position indicated by the information in the mixed reality space 21, or may arrange the virtual model 30 at an inclination indicated by the information. For example, the communication control unit 950 may also transmit information based on input from the VR user (for example, the amount of movement, movement trajectory, inclination, etc. of the virtual model 30 calculated (in other words, acquired) by the control unit 310 based on the input). The MR-side object control unit 820 may move a virtual object such as the virtual model 30 based on the information. That is, in the present embodiment, the MR side object control unit 820 gives a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user accepted by the VR side input reception unit 916. "Based on the input from the VR user" does not mean that the MR side object control unit 820 itself receives the input and controls the virtual object, but the virtual object in the mixed reality space 21 responds to the input as a result. Any material that makes a predetermined change may be used.

Further, the VR-side object control unit 920 gives a predetermined change to the VR avatar 25 in the virtual space 11 based on the input from the VR user that the VR-side input reception unit 916 has received. Specifically, the VR-side object control unit 920, for example, based on information indicating the VR user's operation on the controller 280 (in other words, information regarding the VR user's movement), received by the VR-side input receiving unit 916, A predetermined change is given to the VR avatar 25 in the virtual space 11. Specifically, for example, the VR-side object control unit 920 moves the VR avatar 25 in the virtual space 11 in accordance with an instruction from the VR user by operating the controller 280.

In addition, the VR-side object control unit 920 also controls the movement of each part of the VR user's body (for example, hand movement, head movement, The VR avatar 25 is moved based on information regarding eye movements, changes in facial expressions, etc., and information regarding the tilt and direction of the VR user. That is, the VR-side object control unit 920 controls the movement of each part of the VR user's body (in other words, the posture), the inclination of the VR user, and the direction in which the VR user is facing in the virtual space 11 based on each of these pieces of information. reflected on the VR avatar 25. That is, in the present embodiment, the VR-side object control unit 920 makes the movement of the VR avatar 25 in the virtual space 11 imitate the movement of the VR user detected by the detection means that detects the movement of the VR user. Then, the movement of the VR avatar 25 is controlled. In other words, the VR-side object control unit 920 controls the movement of the VR avatar 25 to be linked to the movement of the VR user with respect to at least a part of the body (for example, hands, feet, head, eyes, mouth, etc.). Then, the movement of the VR avatar 25 is controlled.

Further, the MR-side object control unit 820 may, for example, make changes similar to changes given to the VR avatar 25 in the virtual space 11 based on the input from the VR user received by the VR-side input reception unit 916 in the mixed reality space. Give to VR avatar 25 out of 21. That is, for example, as described above, when the VR avatar 25 in the virtual space 11 moves due to the VR user's operation on the controller 280, the MR side object control unit 820 moves the VR avatar 25 in the mixed reality space 21 to the same position. Move it like this. Note that there is no particular limitation on the method of applying the same change to the VR avatar 25 in the virtual space 11 to the VR avatar 25 in the mixed reality space 21, but for example, the above-mentioned virtual model 30 may be used. You can do it as well.

Note that the position of the VR avatar 25 in the virtual space 11 and the position in the mixed reality space 21 may move as the VR user moves in the real space. That is, for example, the VR-side object control unit 920 may move the VR avatar 25 based on position information indicating the position of the VR user in real space, which is input from the VR user.

Further, movement of a part of the VR avatar 25 in the virtual space 11 and the VR avatar 25 in the mixed reality space 21 may be realized by operating the controller 280. That is, for example, the VR side object control unit 920 and the MR side object control unit 820 control a part of the body of the VR avatar 25 based on information indicating the VR user's operation on the controller 280, which is input from the VR user. You may also move it.

In this embodiment, the results of actions taken on real objects actually existing in the conference room 21 as a predetermined real space can also be shared between the VR user and the MR user 6. This point will be explained below by taking as an example the control of an object (for example, a blackboard, a whiteboard, etc.) on which a predetermined entry can be made as a real object. As shown in FIG. 8, a blackboard 35 as a real object is arranged in the mixed reality space 21, and a virtual object corresponding to the blackboard 35 is placed in the virtual space 11 at a position corresponding to the blackboard 35. It is assumed that a virtual blackboard 36 is placed.

In this embodiment, the action (in other words, the result of the action) performed by the MR user 6 on the blackboard 35 as a real object in the mixed reality space 21 is reflected on the blackboard 35 in the mixed reality space 21. ing. Further, in this embodiment, the action is reflected on the virtual blackboard 36 in the virtual space 11.

In the mixed reality space 21, the MR user 6 can make predetermined entries on the blackboard 35. Further, the writing operation by the MR user 6 on the blackboard 35 is detected by a detection means for detecting the writing operation (hereinafter referred to as "MR side writing detection means"), and the output from the MR side writing detection means (in other words, information regarding the entry operation) is received by the MR side input reception unit 816 as input from the MR user 6. The MR side entry detection means may be, for example, a detection means capable of detecting the hand movement of the MR user 6. Specifically, the MR side entry detection means may be, for example, a pen-shaped controller as the controller 480. Further, the MR side entry detection means may be a camera as the detection device 460, the camera 413, or the like. Furthermore, the MR side entry detection means may be a device worn by the MR user 6 or the like. Further, the MR side entry detection means may be a sensor provided on the blackboard 35 (for example, a touch sensor, a sensor that reads reflected light of infrared light from a device provided on the blackboard 35 that emits infrared light, etc.). In other words, the blackboard 35 is capable of communicating with the computer 500, and information regarding the writing operation of the MR user 6 detected by the sensor included in the blackboard 35 is sent to the MR side input receiving unit 816 as an input from the MR user 6. It may be accepted.

The writing actions of the MR user 6 are reflected on the blackboard 35 in the mixed reality space 21 . In other words, when the MR user 6 performs a writing operation, the blackboard 35 displays the contents written by the MR user 6 (for example, the picture of the moon in FIG. 10(a), The state changes to displaying a picture of a heart and the words "Sample". For example, the blackboard 35 is an electronic blackboard that electronically displays (in other words, reflects) the contents filled in by the MR user 6, and when the MR user 6 performs a writing operation, the contents are displayed. It may be changed to . Specifically, for example, the writing surface of the blackboard 35 itself serves as a display, and the display may display the written contents. Further, the blackboard 35 may include a projector that projects an image onto the writing surface, and the projector may project the written contents onto the writing surface. In other words, the reflection of the user's actions on the mixed reality space 21 (in other words, the changes in the mixed reality space 21 seen through the MR device 410 caused by the actions of the MR user 6) is caused by some change ( For example, this may be realized by a change in display content). In addition, when displaying the entry electronically, the computer of the blackboard 35 may control the display based on the detection result of the sensor provided on the blackboard, and the computer 500 may control the display based on the detection result of the MR side entry detection means. The display may be controlled based on. That is, for example, the control unit 510 or the like of the computer 500 may control the display of the blackboard 35 based on the input from the MR user 6 received by the MR side input receiving unit 816 (hereinafter referred to as "blackboard display control means"). ). Specifically, the blackboard display control means may control, based on input from the MR user 6, so that the image displayed by the display or projector of the blackboard 35 becomes an image showing the written content. Note that the writing operation of the MR user 6 may be reflected on the blackboard 35 in the mixed reality space 21 by the MR user 6 actually writing letters or pictures on the blackboard 35 using a writing instrument such as chalk. In other words, the action of actually writing characters or pictures on the blackboard 35 with a writing instrument may be detected by the MR side writing detection means as the writing action of the MR user 6.

Further, for example, the MR-side object control unit 820 generates a virtual object (hereinafter referred to as "input object 37") indicating the entry content entered by the MR user 6, and places it at the position of the blackboard 35 in the mixed reality space 21. (See FIG. 10(a)). Specifically, the MR-side object control unit 820 may generate the entry object 37 based on the input from the MR user 6 that the MR-side input reception unit 816 receives. When viewing the blackboard 35 through the MR device 410, the state in which the blackboard 35 displays the entry contents (in other words, the entry object 37) may be visually recognized. In other words, the reflection of the user's actions on the mixed reality space 21 (in other words, the changes in the mixed reality space 21 seen via the MR device 410 caused by the actions of the MR user 6) is based on a predetermined change in the real object. It may be realized by adding a virtual object.

Furthermore, the VR-side object control unit 920 applies a predetermined change to the virtual object corresponding to the real object based on the input from the MR user 6 that the MR-side input reception unit 816 has received. Specifically, as shown in FIG. 10(b), the VR side object control unit 920 causes the input from the MR user 6 to be reflected on the virtual blackboard 36, so that the virtual blackboard 36 reflects the input contents written by the MR user 6. control so that it is displayed. In other words, the VR-side object control unit 920 changes changes regarding the real object (for example, the blackboard 35) in the mixed reality space 21 based on the behavior of the MR user 6 into the virtual object (for example, the blackboard 35) in the virtual space 11 corresponding to the real object. For example, it is reflected on the virtual blackboard 36). Specifically, for example, when information regarding the entry operation of the MR user 6 is input to the MR side input receiving unit 816, the VR side object control unit 920 displays the entry object 38 indicating the entry content entered by the MR user 6. may be placed at the position of the virtual blackboard 36 in the virtual space 11 (see FIG. 10(b)). Further, the VR-side object control unit 920 may control the virtual blackboard 36 to be in a state in which written contents are displayed, for example, by changing the form of the virtual blackboard 36 itself. Further, the virtual blackboard 36 is a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, and the VR-side object control unit 920 causes the virtual blackboard 36 to display the image. You can leave it there. Further, the MR system 400 includes a camera that photographs the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 is a virtual object that displays an image photographed by the camera, and the VR-side object control unit 920 The image may be displayed on the virtual blackboard 36.

Further, in the present embodiment, an action (in other words, a result of the action) performed by the VR user on the virtual blackboard 36 as a virtual object in the virtual space 11 is reflected on the virtual blackboard 36 in the virtual space 11. It has become. Furthermore, in this embodiment, the action is reflected on the blackboard 35 in the mixed reality space 21.

In the virtual space 11, the VR user can make predetermined entries on the virtual blackboard 36. Further, the writing operation on the virtual blackboard 36 by the VR user is detected by a detection means (hereinafter referred to as "VR side writing detection means") that detects the writing operation, and the output from the VR side writing detection means is detected by the VR user. The VR-side input receiving unit 916 receives the input as the input. The VR side entry detection means may be, for example, a detection means capable of detecting the movement of a VR user's hand. Specifically, the VR side entry detection means may be, for example, the controller 280 or the like. Further, the VR side entry detection means may be a camera or the like as the detection device 260. Furthermore, the VR side entry detection means may be a device worn by the VR user.

The VR user's writing actions are reflected on the blackboard 35 in the mixed reality space 21 . In other words, when the VR user performs a writing operation, the blackboard 35 changes to a state where the content written by the VR user is displayed. In other words, the input from the VR user accepted by the VR-side input reception unit 916 is reflected on the blackboard 35. For example, the input from the VR user may be reflected on the blackboard 35 as follows. That is, for example, the MR-side object control unit 820 may place the entry object 37 indicating the content entered by the VR user at the position of the blackboard 35 in the mixed reality space 21. Further, the blackboard display control means may control, based on input from the VR user, so that the image displayed by the display or projector of the blackboard 35 becomes an image showing the content written by the VR user. That is, the MR-side object control unit 820 or the blackboard display control unit can function as a reflection unit that reflects the input from the VR user received by the VR-side input reception unit 916 on the real object in the mixed reality space 21.

Further, the VR user's writing operation is also reflected on the virtual blackboard 36 in the virtual space 11. In other words, when the VR user performs a writing operation, the virtual blackboard 36 changes to a state where the contents written by the VR user are displayed. In other words, the virtual blackboard 36 reflects the input from the VR user accepted by the VR-side input reception unit 916. Specifically, the VR-side object control unit 920 applies a predetermined change to the virtual blackboard 36 based on the input from the VR user that the VR-side input reception unit 916 receives. More specifically, the VR-side object control unit 920 controls the input from the VR user to be reflected on the virtual blackboard 36 so that the virtual blackboard 36 is in a state of displaying the content written by the VR user. Specifically, for example, when information regarding the entry operation of the VR user is input to the VR-side input reception unit 916, the VR-side object control unit 920 inputs the entry object 38 indicating the entry content entered by the VR user, It may be placed at the position of the virtual blackboard 36 in the virtual space 11. Further, the VR-side object control unit 920 may control the virtual blackboard 36 to be in a state in which written contents are displayed, for example, by changing the form of the virtual blackboard 36 itself. Further, the virtual blackboard 36 is a virtual object that displays an image displayed by the display of the blackboard 35 or a projector, and the VR-side object control unit 920 causes the virtual blackboard 36 to display the image. You can leave it there. Further, the MR system 400 includes a camera that photographs the blackboard 35 in the mixed reality space 21, the virtual blackboard 36 is a virtual object that displays an image photographed by the camera, and the VR-side object control unit 920 The image may be displayed on the virtual blackboard 36. In other words, the input from the VR user received by the VR side input reception unit 916 is reflected on the real object in the mixed reality space 21, so that the virtual blackboard 36 changes based on the input from the VR user. It may be.

As described above, in this embodiment, the actions that the MR user 6 performs on a real object and the actions that the VR user performs on a virtual object corresponding to the real object are transferred to the mixed reality space 21 and the virtual object. It is now reflected in space 11. Such reflection of each action on the mixed reality space 21 or the virtual space 11 may be as follows.

That is, for example, when the MR user 6 points to an arbitrary position in the mixed reality space 21 with a predetermined pointer such as a laser pointer, the pointed position (hereinafter referred to as "indicated position") also occurs in the virtual space 11. It may be possible to understand. Specifically, for example, in the mixed reality space 21, the detection device 460, camera 413, etc. of the MR system 400 may function as a pointed position detection means for detecting the pointed position. Information regarding the indicated position detected by the indicated position detection means may be received by the MR side input receiving unit 816 as an input from the MR user 6. Furthermore, the VR-side object control unit 920 may place a virtual object indicating light from the laser pointer at a position in the virtual space 11 that corresponds to the indicated position in the mixed reality space 21. In other words, in displaying the virtual space 11 in the VR device 210, the control unit 310 causes the position in the virtual space 11 corresponding to the indicated position in the mixed reality space 21 to shine in a predetermined color representing the light from the laser pointer. It may be controlled as follows.

Further, for example, when the VR user points to an arbitrary position in the virtual space 11 with a predetermined pointer such as a virtual laser pointer, the pointed position may also be known in the mixed reality space 21. Specifically, for example, when information regarding the operation of the VR user on the controller 280 for operating the virtual laser pointer (in other words, information regarding the movement of the VR user) is input to the VR side input reception unit 916, The VR-side object control unit 920 places a virtual object indicating light from a virtual laser pointer at a position indicated by a VR user's operation (in other words, a position indicated by a virtual laser pointer), etc. You may. In other words, in displaying the virtual space in the VR device 210, the control unit 310 causes the position in the virtual space 11 designated by the virtual VR user's operation to be set to a predetermined color representing the light from the virtual laser pointer. You can also control it so that it shines. In addition, the control unit 510 and the like are configured such that the position in the mixed reality space 21 corresponding to the designated position in the virtual space 11 (specifically, the designated position by the VR user) is a predetermined position representing the light from the virtual laser pointer. It may also be controlled so that it shines in this color. Specifically, the MR side object control unit 820 places a virtual object indicating light from a virtual laser pointer at a position in the mixed reality space 21 that corresponds to the indicated position in the virtual space 11. Good too. Further, for example, when the position indicated by the VR user is a predetermined position on the virtual blackboard 36, the blackboard display control means may control the image displayed by the display of the blackboard 35 or the projector based on the input from the VR user. However, the image may be controlled so that a position corresponding to the predetermined position on the virtual blackboard 36 lights up (that is, an image indicating that the predetermined location is pointed by the laser pointer). Further, for example, the detection device 460, the MR device 410, etc. are equipped with a device that emits laser light, and the control unit 510 controls the device in accordance with input from the VR user received by the VR side input reception unit 916. Alternatively, the laser beam may be applied to a position in the mixed reality space 21 that corresponds to the indicated position in the virtual space 11.

Further, the actions of the MR user 6 or the VR user may be reflected in the mixed reality space 21 and the virtual space 11 as follows. That is, for example, in the mixed reality space 21, a device (for example, an electronic piano) capable of inputting or outputting sound as a real object is arranged, and the actions of the MR user 6 or the VR user regarding the device are It may also be reflected in the real space 21 and the virtual space 11. In the following, an electronic piano as a real object is placed in the mixed reality space 21, and a virtual electronic piano as a virtual object corresponding to the electronic piano is placed in the virtual space 11 at a position corresponding to the electronic piano. It is assumed that .

The electronic piano may be able to communicate with the computer 500, for example. When the MR user 6 operates the electronic piano in the mixed reality space 21, information regarding the operation (for example, sound data corresponding to the operation) may be sent to the computer 300 via the computer 500. . Then, the sound control unit 945 may output sound from the speaker 216 based on the information. With this, for example, when the MR user 6 operates a key of the electronic piano (for example, a key for the sound of "C"), the sound corresponding to the operated key (for example, the sound of "C") is output from the speaker 216. It may be possible to do so. In other words, the MR side input reception unit 816 receives the output (for example, sound data corresponding to the operation) from the electronic piano (in other words, a sensor or the like that detects the operation on the keys) serving as a detection means for detecting the operation of the MR user 6. ) as an input from the MR user 6, and the control unit 310 (for example, the sound control unit 945) causes the speaker 216 to output a sound corresponding to the operation of the MR user 6 on the electronic piano based on the input. It may be. Further, the sound control unit 945 may control the direction in which the sound is emitted so that the sound corresponding to the operation on the electronic piano sounds as if it is being output from the virtual electronic piano in the virtual space 11.

Further, when the VR user operates the virtual electronic piano in the virtual space 11, information regarding the operation may be sent to the computer 500. Then, the sound control unit 845 may output sound from the speaker 416 based on the information. With this, for example, when a VR user operates a key of the virtual electronic piano (for example, a key for the sound of "C"), the sound corresponding to the operated key (for example, the sound of "C") is output from the speaker 416. It may be possible to do so. In other words, the VR-side input reception unit 916 receives the output from the detection means (for example, the detection device 260 or the controller 280) that detects the VR user's operation on the virtual electronic piano as an input from the VR user, and the control unit 510 ( For example, the sound control unit 845) may be configured to cause the speaker 416 to output a sound corresponding to the VR user's operation on the virtual electronic piano based on the input. Note that the control unit 510 may be configured to cause the speaker of the electronic piano to output a sound corresponding to the VR user's operation on the virtual electronic piano.

Note that when the MR user 6 operates the electronic piano, the sound corresponding to the operation may be outputted in the mixed reality space 21 by the speaker of the electronic piano or by the speaker 416. Further, when the VR user operates the virtual electronic piano, the speaker 216 may output the sound corresponding to the operation in the virtual space 11.

<Processing executed by the information processing system>
Next, the flow of processing executed by the information processing system 100 will be described with reference to FIGS. 11 to 14.

An example of processing executed by the VR system 200 will be described with reference to FIG. 11.

In step S101, the processor 301 of the computer 300 identifies virtual space data and defines the virtual space 11.

In step S102, the processor 301 places the VR user's VR avatar 25 in the virtual space 11.

In step S103, the processor 301 receives information from the mixed reality space 21 from the processor 501 of the computer 500, which is connected to the computer 300 via the network 2 and controls the mixed reality space 21 corresponding to the virtual space 11. The location information indicating the location of the MR user 6 inside is acquired.

In step S104, the processor 301 places the MR avatar 26 of the MR user 6 at a position in the virtual space 11 that corresponds to the position of the MR user 6 in the mixed reality space 21, based on the position information acquired in step S103. do.

In step S105, the processor 301 arranges the virtual model 30 as a virtual object in the virtual space 11. The processor 301 places the virtual model 30 at a position in the virtual space 11 that corresponds to the position of the virtual model 30 in the mixed reality space 21 . Note that arranging at corresponding positions may mean that the position in the mixed reality space 21 is determined in advance, or the position in the virtual space 11 may be determined in advance.

In step S106, the processor 301 receives input from the VR user. Specifically, the processor 301 receives, for example, the output of a detection means for detecting the movement of the VR user as an input from the VR user.

In step S107, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on input from the VR user. Specifically, the processor 301 moves the VR avatar 25 or the virtual model 30 in the virtual space 11 based on input from a VR user, for example. For example, if the input is related to the VR avatar 25, the processor 301 moves the VR avatar 25. Further, for example, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30. Additionally, the processor 301 transmits information regarding input from the VR user to the processor 501.

In step S108, the processor 301 acquires information regarding the input received by the processor 501 from the MR user 6.

In step S109, the processor 301 applies a predetermined change to the virtual object in the virtual space 11 based on the input from the MR user 6 obtained in step S108. Specifically, the processor 301 moves the MR avatar 26 or the virtual model 30 in the virtual space 11 based on input from the MR user 6, for example. For example, if the input is related to the MR avatar 26, the processor 301 moves the MR avatar 26. Further, if the input is related to the virtual model 30, the processor 301 moves the virtual model 30.

Next, an example of processing executed by the MR system 400 will be described with reference to FIG. 12.

In step S201, the processor 501 of the computer 500 defines the mixed reality space 21.

In step S202, the processor 501 detects the position of the MR user 6 within the mixed reality space 21. Further, the processor 501 uses position information indicating the detected position of the MR user 6 to control the virtual space 11, which is a computer 300 connected to the computer 500 via the network 2 and corresponds to the mixed reality space 21. It is transmitted to the processor 301 of the computer 300.

In step S203, the processor 501 acquires position information indicating the position of the VR avatar 25 in the virtual space 11 from the processor 301.

In step S204, the processor 501 places the VR avatar 25 at a position in the mixed reality space 21 that corresponds to the position of the VR avatar 25 in the virtual space 11, based on the position information acquired in step S203. Note that arranging at corresponding positions may mean that the position in the virtual space 11 is determined in advance, or the position in the mixed reality space 21 may be determined in advance.

In step S205, the processor 501 places the virtual model 30 as a virtual object in the mixed reality space 21. The processor 501 places the virtual model 30 at a position in the mixed reality space 21 that corresponds to the position of the virtual model 30 in the virtual space 11 .

In step S206, processor 501 receives input from MR user 6. Specifically, the processor 501 receives, for example, the output of a detection means for detecting the movement of the MR user 6 as an input from the MR user 6.

In step S207, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the MR user 6. Specifically, the processor 501 moves the virtual model 30 in the mixed reality space 21 based on input from the MR user 6, for example. Further, the processor 501 transmits information regarding input from the MR user 6 to the processor 301.

In step S208, the processor 501 acquires information regarding the input received by the processor 301 from the VR user.

In step S209, the processor 501 applies a predetermined change to the virtual object in the mixed reality space 21 based on the input from the VR user obtained in step S208. Specifically, the processor 501 moves the VR avatar 25 or the virtual model 30 in the mixed reality space 21 based on, for example, input from a VR user. For example, if the input is related to the VR avatar 25, the processor 501 moves the VR avatar 25. Further, if the input is related to the virtual model 30, the processor 501 moves the virtual model 30.

Next, an example of a process for reflecting the behavior of the MR user 6 in the virtual space 11 will be described with reference to FIG. 13.

In step S301, processor 501 receives input from MR user 6. Specifically, the processor 501 receives, as an input from the MR user 6, the output of a detection means that detects the behavior of the MR user 6 with respect to a specific real object (for example, the blackboard 35, a laser pointer, or an electronic piano). .

In step S302, the processor 501 transmits information regarding the input from the MR user 6 accepted in step S301 to the processor 301 (in other words, the VR system 200).

In step S303, the processor 301 gives a predetermined change to the virtual object corresponding to the specific real object on which the MR user 6 acted, based on the information transmitted in step S302. For example, when the MR user 6 performs a writing operation to write predetermined contents on the blackboard 35, the processor 301 changes the virtual blackboard 36 corresponding to the blackboard 35 to a state where the predetermined contents are displayed.

Next, an example of a process for reflecting the VR user's actions in the mixed reality space 21 will be described with reference to FIG. 14.

In step S351, the processor 301 receives input from the VR user. Specifically, the processor 301 inputs the output of a detection means for detecting the VR user's behavior toward a virtual object (for example, the virtual blackboard 36 or the virtual electronic piano, etc.) corresponding to a specific real object from the input from the VR user. accepted as.

In step S352, the processor 301 transmits information regarding the input from the VR user accepted in step S351 to the processor 501 (in other words, the MR system 400).

In step S353, the processor 501 causes the action performed by the VR user on the virtual object corresponding to the specific real object to be reflected in the specific real object, based on the information transmitted in step S352. For example, when the VR user performs a writing operation to write predetermined contents on the virtual blackboard 36, the processor 501 changes the blackboard 35 to a state where the predetermined contents are displayed.

[Second embodiment]
Next, a second embodiment of the present invention will be described. The information processing system 100 of the second embodiment can employ the contents described in the first embodiment to the extent that no contradiction occurs. For example, the information processing system 100 of the second embodiment may include part or all of the configuration of the information processing system of the first embodiment. In the following, description of the matters described in the first embodiment will be omitted or simplified.

In the following, a case where the information processing system 100 is applied to a service in which an event occurring in the virtual space 11 is reflected in the mixed reality space 21 will be described as an example. Specifically, in the following, a specific position in the virtual space 11 imitating a place that actually exists (as an example, hereinafter referred to as "town A") (as an example, hereinafter referred to as "town A") In front of store B (in other words, in front of a virtual object imitating store B). An information processing system 100 that can be experienced by an MR user 6 at a corresponding position in the real space (here, in front of store B) via the MR device 410 (here, live music can be viewed) will be described.

The coordinate definition unit 812 defines the correspondence between coordinates measured by a GPS sensor (in other words, position detection means) and virtual space coordinates (in other words, the correspondence between points in the real space and points in the virtual space 11). ) is stored in the storage unit 611 as a correspondence relationship storage unit. In other words, in this embodiment, information indicating the correspondence between the positions of the real space (in other words, the mixed reality space) and the virtual space 11 is stored in advance in the correspondence storage section.

The MR side input receiving section 816 receives input from the position detecting means. The position detection means acquires position information indicating the position of the MR user 6 using a GPS sensor included in the MR device 410. That is, the MR side input receiving unit 816 receives an input of positional information (for example, positional information acquired using a GPS sensor) indicating the current position of the MR user 6 in real space.

The control unit 310 of the VR system 200 controls the execution of a predetermined event at a specific position in the virtual space 11. In this embodiment, the predetermined event is a predetermined event (specifically, a live music event). Specifically, as shown in FIG. 15, the VR-side object control unit 920 places an event-related object 50 as a virtual object related to a predetermined event at a specific position in the virtual space 11. More specifically, the VR-side object control unit 920, for example, creates an object of a character appearing in a music live (hereinafter referred to as "appearing character") as an event-related object 50 in front of store B in the virtual space 11. .), stage objects for music live performances, etc. Note that the VR-side object control unit 920 arranges the event-related objects 50 based on the VR user's instructions (in other words, the input from the VR user that the VR-side input reception unit 916 receives (for example, an operation on the controller 280)). You can. Further, the VR-side object control unit 920 may place the event-related object 50 based on instructions from the operator of this service.

Further, the VR-side object control unit 920 moves the event-related object 50 (for example, an appearing character) to control the progress of the event (in other words, progress the event). Note that the VR-side object control unit 920 moves the appearing character based on the VR user's instructions (in other words, the input from the VR user that the VR-side input receiving unit 916 receives (for example, an operation on the controller 280)). Good too. Further, the movements of the appearing characters are prepared in advance, and the VR-side object control unit 920 may move the appearing characters by reproducing the data prepared in advance. In other words, the appearing character may be the VR avatar 25, a non-player character, or the like.

That is, in the information processing system 100 of the present embodiment, the VR system 200 (in other words, the control unit 310) operates the virtual space 11 (in other words, the control unit 310) in which you can move around by operating your own character (for example, the VR avatar 25). , as a predetermined event for a VR user (in other words, a VR user who operates a character in a specific position) who is at a specific location (in other words, in front of virtual store B) in the virtual town A). Provides views related to live music. Here, the VR user may be a user different from the user who operates the character appearing in the music live (in other words, for example, an audience member of the music live), or the same user (in other words, for example, live performers, etc.).

The event-related object 50 is attached with information (in other words, a tag) indicating that the object is related to a predetermined phenomenon (in other words, a predetermined event). Furthermore, position information indicating the position (in other words, coordinates) of the event-related object 50 in the virtual space is attached to the event-related object 50. Note that the position information of the event-related objects 50 may be provided to each event-related object 50 individually, or may be provided to events all at once.

As shown in FIG. 15, the MR side object control unit 820 places a virtual object corresponding to the event-related object 50 in the mixed reality space 21 at a position corresponding to the position in the virtual space 11 where the event-related object 50 is placed. Place objects. Here, the virtual object corresponding to the event-related object 50, that is, the virtual object in the mixed reality space 21 corresponding to the event-related object 50 in the virtual space 11, is, for example, the event-related object 50 in the virtual space 11. They may have the same form (in other words, they represent the same object) or may have different forms. In this embodiment, an event-related object 50 (specifically, a character appearing in a music live performance, a stage of a music live performance, etc.) having the same form as the event-related object 50 in the virtual space 11 is arranged in the mixed reality space 21. Ru.

Further, the MR-side object control unit 820 moves the event-related object 50 (for example, the appearing character) to advance the event to be executed in the mixed reality space 21. Here, the progress of the event by the MR side object control unit 820 (specifically, control to move the appearing character, etc.) and the progress of the event by the VR side object control unit 920 (specifically, control to move the appearing character, etc.) ) are linked. That is, the MR side object control unit 820 reflects the event taking place in the virtual space 11 on the mixed reality space 21. In other words, in this embodiment, events occurring in the virtual space 11 are also reflected in the mixed reality space 21 in real time. Specifically, the MR-side object control unit 820 controls the event-related object 50 in the mixed reality space 21 to move in the same manner as the event-related object 50 in the virtual space 11. Note that "reflected in real time" includes cases where there is a slight lag in the occurrence of an event (in other words, the progress of an event), and also includes cases where there is a slight lag in the occurrence of an event (in other words, the progress of an event), and also when there is a slight lag between the event in the virtual space 11 and the event in the mixed reality space 21. Either progression may take precedence. Furthermore, here, "reflection" does not mean that the event-related object 50 in the mixed reality space 21 moves exactly the same as the event-related object 50 in the virtual space 11. For example, among the movements of the appearing characters in the virtual space 11, some movements may not be reflected in the appearing characters displayed in the mixed reality space 21. In other words, the control unit 510 of the MR system 400 may provide the MR user 6 with a view that can be determined to be related to the same event as the view provided to the VR user.

In this embodiment, when a live music event is held in front of store B in town A in the virtual space 11, the live music event is performed in front of store B in town A in the mixed reality space 21. That is, the MR-side object control unit 820 places the event-related object 50 related to a predetermined event occurring at a specific position in the virtual space 11 at a position in the mixed reality space 21 corresponding to the specific position. At the same time, the event-related object 50 is moved so as to make a movement corresponding to the movement in the virtual space 11. Then, the display control unit 840 displays the event-related object 50 placed at the position in the mixed reality space 21 indicated by the position information on the display of the MR apparatus 410 according to the position information input to the MR side input reception unit 816. 411. Specifically, when the MR user 6 is in real space, the MR user 6 inputs the position information obtained using the GPS sensor (specifically, the position information indicating the position of the MR user 6) to the MR side input receiving unit 816 that receives the input. When position information indicating that the user is in front of store B is input, the control unit 510 displays an event-related object 50 related to a live music event being held in front of store B in the virtual space 11 on the display 411. display. As a result, a view of the mixed reality space 21 in which the event-related object 50 is placed in front of the store B in the real space (in other words, a live music event is performed in the virtual space 11 in front of the store B in the real space) is provided to the MR user 6. That is, when the position information indicates that the MR user 6 is at a position in the real space that corresponds to a specific position in the virtual space 11, the control unit 510 performs an operation at the specific position in the virtual space 11. The MR user 6 is provided with a view of the mixed reality space 21 in which the event being viewed is reflected.

Note that the control unit 510 may include a benefit granting unit that grants a benefit to the MR user 6 based on the fact that the MR user 6 has visited a position corresponding to the specific position in real space. The benefit may be, for example, a virtual object (for example, an item that a character can equip) that the user who has acquired the benefit can use within the game (in other words, within the service), or it may be a virtual object that can be used by a user who has acquired the benefit (for example, an item that can be equipped by a character), or a virtual object that can be used in a real space. It may be an electronic ticket or the like that can be exchanged with the user, or it may be a predetermined point owned by the user within the service. Furthermore, "based on having visited" may mean directly determining whether or not the person has visited and granting benefits, and specific actions that can be performed by visiting (for example, attending a live music concert). It may also be possible to provide a benefit when viewing (starting, finishing, or continuing for a predetermined period of time, etc.) is performed. According to such a configuration, it is possible to motivate the MR user 6 to visit a position corresponding to a specific position in real space.

As described above, in this embodiment, the MR user 6 is provided with a view of the mixed reality space 21 that reflects a predetermined event occurring at a specific position in the virtual space 11, but the predetermined event is It's not limited to live performances. For example, the predetermined event may be an event in which a character operated by a specific user (in other words, an event-related object 50 related to the predetermined event) moves. Specifically, for example, assume that the VR system 200 provides a game in which a character can be moved in a virtual space 11 imitating a city A. In this case, the VR-side object control unit 920 operates based on an instruction from the VR user as the specific user (in other words, an input from the VR user (for example, an operation on the controller 280) received by the VR-side input reception unit 916). Then, the character is caused to perform a predetermined action (for example, an action such as jumping or running) in a virtual space 11 imitating the city A. Further, the MR-side object control unit 820 places the character at a position in the mixed reality space 21 that corresponds to the position where the character is placed in the virtual space 11, and An action similar to the action is executed in a mixed reality space 21. That is, in this case, for example, when the VR user as the specific user is playing a game in which the character is moved in the virtual space 11 imitating the city A, if the MR user 6 visits the city A in the real space. , the control unit 510 displays a view of the mixed reality space 21 provided via the display 411 of the MR device 410 of the movement of the character in the game (in other words, on the virtual space 11) that is being moved by the specific user. be reflected in. That is, in this example, the view of the virtual space 11 provided to the specific user in the VR device 210 of the VR user as the specific user, and the view of the MR device 410 of the MR user 6 who is in town A in the real space. The virtual object operated by the specific user moves in the same way as the view of the mixed reality space 21 provided to the MR user 6. According to such a configuration, it is possible to reproduce the movement of a character on a game screen in a real city, and it is possible to improve the interest of the service.

Note that the MR users 6 who can receive the view of the mixed reality space 21 reflecting a predetermined event occurring at a specific position in the virtual space 11 may be limited. Specifically, the storage unit 511 functions as a rights information storage unit that stores information regarding the right to receive the view (hereinafter referred to as “rights information”), and the control unit 510 The view may be provided to the MR user 6 if the user has the right, and the view may not be provided to the MR user 6 if the user does not have the right. In addition, the rights may be granted to the MR user 6 by the MR user 6 paying a consideration (e.g., cash payment, consumption of assets held by the MR user 6 (e.g. virtual currency, etc.) within the service, etc.). It may be provided. Specifically, for example, the right to view live music may be available for purchase within the service. More specifically, for example, information regarding the presence or absence of a live music ticket that can be purchased within the service is stored as the rights information, and the control unit 510 controls the ability of the MR user 6 who has the ticket to view the live music event. may be controlled so that the MR user 6 who does not have the ticket cannot view the live music event. In other words, information related to ownership of an item (for example, a ticket, etc.) related to the right to receive the view is stored as the right information, and the control unit 510 controls, when the MR user 6 owns the item, The view may be provided to the MR user 6. Note that the right (in other words, the item) may be obtained by completing a predetermined mission within the service.

Further, the control unit 510 is configured to reflect a predetermined event occurring at a specific position in the virtual space 11 and related to a VR user selected by the MR user 6 from among a plurality of users. A view of the mixed reality space 21 may be provided to the MR user 6. In other words, the MR side input reception unit 816 receives an input by the MR user 6 related to an operation (for example, an operation on the controller 480) for selecting which of a plurality of users a predetermined event related to a user is to be reflected. Upon reception, the control unit 510 may reflect a predetermined event related to the VR user selected by the operation on the view provided to the MR user 6. The operation may be, for example, an operation in which the MR user 6 selects a user from among his/her friends to whom the event will be reflected. Further, predetermined events related to all friends are reflected, and the operation may be an operation of registering another user as a friend. In other words, in the service of this embodiment, the MR user 6 may be provided with a view of the mixed reality space 21 that reflects a predetermined event related to his or her friend. Further, for example, the operation may be an operation in which the MR user 6, such as a professional player, selects a user from among a plurality of players to whom a predetermined event is to be reflected. That is, for example, when the player selected by the MR user 6 (in other words, a specific VR user) is playing the above-mentioned game in which a character is moved in the virtual space 11 imitating the city A, the control unit 510 The movement of the character being moved by the player in the game (in other words, on the virtual space 11) may be reflected in the view of the mixed reality space 21 provided to the MR user 6. In other words, the MR user 6 may be able to receive a view of the mixed reality space 21 that reflects a predetermined event related to the user associated with the user among the plurality of users. . Further, for example, when a celebrity selected by the MR user 6 is performing a live performance in the virtual space 11, the control unit 510 causes the live performance to be reflected in the view of the mixed reality space 21 provided to the MR user 6. etc. Note that when there are multiple events related to a specific user, it may be possible to select the specific event related to the specific user as the event to be reflected.

In addition, in this way, in the service of this embodiment, each user (for example, MR user 6 and VR user) can register other users (for example, VR user and MR user 6) as friends. It may be. Each user can register other users as friends, for example, based on a predetermined operation on the controllers 280, 480. Moreover, registering as a friend can also be said to mean that a certain user associates another user with himself/herself. Moreover, registering as a friend can also be said to bookmark a specific user other than yourself. Registering as a friend means making information about a certain user (specifically, a friend) (for example, information about login status, etc.) easier to recall than information about other users (specifically, non-friends). It can also be said to be kept. Note that there may be a situation in which a certain user has registered another user as a friend, but the other user has not registered the certain user as a friend. That is, there may be a state in which the certain user unilaterally follows the other user. Further, there may be a state in which the certain user and the other user are mutually following each other. Note that it may also be possible that there is no unilateral following state. In other words, in this embodiment, a "friend" may refer only to another user with whom a mutual follow relationship has been established from the perspective of a certain user, and a unilateral follow relationship may not be established. It may also refer to other users.

The control unit 510 also controls a mixed reality in which a predetermined event occurring at a specific position in the virtual space 11 and having an attribute corresponding to the attribute set for the MR user 6 is reflected. A view of space 21 may be provided to the MR 6 user. Specifically, for example, each MR user 6 has attributes related to the user's hobbies and preferences, more specifically, attributes such as "I like music" and "I like comedy" (or "I like a specific celebrity"). may be attached. Then, for example, the control unit 510 reflects a predetermined event with the attribute "music" on the MR user 6 with the attribute "music lover", and the control unit 510 reflects the predetermined event with the attribute "music" on the MR user 6 with the attribute "music lover". A predetermined event to which the attribute "music" is attached may not be reflected to the MR users 6 to whom the attribute is not attached. Further, for example, the control unit 510 reflects a predetermined event with the attribute "comedy" on the MR user 6 who has the attribute "comedy lover", so that the attribute "comedy lover" is reflected on the MR user 6. A predetermined event to which the attribute "comedy" is attached may not be reflected to the MR user 6 who is not attached. Note that the attributes set for the MR users 6 may be set by each MR user 6 himself (for example, by inputting his own preferences via the controller 480), and may be set using AI (Artificial Intelligence) may be automatically set (for example, according to the user's behavior within the service).

As described above, the control unit 510 controls a specific event selected by the MR user 6 from among a plurality of events that occur in the virtual space 11 (for example, an event related to a specific user selected by the MR user 6, An event having an attribute corresponding to the own attribute set by the MR user 6, an event related to a ticket purchased by the MR user 6, etc.) may be reflected in the mixed reality space 21. In other words, the control unit 510 does not have to reflect in the mixed reality space 21 an event that is not selected by the MR user 6 among the multiple events that occur in the virtual space 11 . In this way, when unselected events are not reflected in the mixed reality space 21, there may be events that are reflected regardless of the user's selection. In other words, "not to reflect an unselected event in the mixed reality space 21" means an event that is reflected in the mixed reality space 21 when a selection is made, and an event that is reflected in the mixed reality space 21 when a selection is not made. It is sufficient that at least one event that is not reflected in the mixed reality space 21 exists. Note that the event selection may be made before the specific event selected by the MR user 6 occurs (in other words, starts), or after the specific event occurs (in other words, It may also be possible to perform the process while the process is being performed.

Note that the virtual space 11 in which an event takes place does not have to imitate the real space in which the event is reflected. For example, an area corresponding to a plaza in front of store B in town A that actually exists is prepared in the virtual space 11, and a predetermined event that takes place in the area is transmitted to the MR user 6 who is in the plaza in the real space. It may also be reflected on the device 410. In this case, the storage unit 611 serving as the correspondence storage unit can store, for example, information indicating the correspondence relationship between the area and the plaza. Note that the virtual space 11 in which events in the real space are reflected, which will be described in the third embodiment, does not have to imitate the real space.

Further, the virtual space 11 in which the event takes place may be one that imitates the topography of the real space in which the event is reflected, and may have a different landscape. For example, in this embodiment, the virtual space 11 may be one that imitates the topography of the city A that actually exists, and may have a different cityscape. Specifically, the virtual space 11 is a reproduction of an area corresponding to town A in the past (for example, the Edo period), and each point in the recreated area is compared with each point in current town A in the real space. Points in the real space and points in the virtual space 11 may be associated so that they point to the same location on the map (in other words, locations at the same latitude and longitude). Then, the reproduced predetermined event that takes place at a specific location in the area may be reflected on the MR device 410 of the MR user 6 who is located at a position corresponding to the specific location in real space. Note that the virtual space 11, which will be described in the third embodiment and in which events in the real space are reflected, may also be one that imitates the topography of the real space and may have a different landscape. .

Note that the view of the mixed reality space 21 regarding a predetermined event may not be provided by the MR device 410 that superimposes a virtual object on the real space visible through the transparent display 411, but may be provided by, for example, a view taken by a camera. The virtual object may be provided by a smartphone or the like that displays on the display 411 an image in which a virtual object is superimposed on an image of a real space. In other words, the view of the mixed reality space 21 may be provided by, for example, displaying the event-related object 50 on a transparent display, or by superimposing the event-related object 50 on an image taken of the real space. This may also be done by displaying an image on a display. In other words, a predetermined system including a smartphone or the like that displays on a display an image in which a virtual object is superimposed on an image taken of real space has at least a part of the configuration of the MR system 400. Good too. In other words, the MR system 400 in this embodiment or other embodiments can be read as the predetermined system (for example, a smartphone, etc.). Further, in this specification, the user of the predetermined system also corresponds to the MR user 6.

Furthermore, the view of the virtual space 11 regarding a predetermined event does not have to be provided by the head-mounted display as the VR device 210; for example, it is possible to display an image of the virtual space 11 on a PC monitor, television, etc. may be provided by. In other words, a PC, a game device, or a predetermined system including these devices and a predetermined display may have at least a part of the configuration of the VR system 200. In other words, the VR system 200 in this embodiment or other embodiments can be read as the predetermined system. Furthermore, in this specification, users of the predetermined system also correspond to VR users.

[Third embodiment]
Next, a third embodiment of the present invention will be described. The information processing system 100 of the third embodiment can adopt the contents described in the first embodiment or the second embodiment to the extent that no contradiction occurs. For example, the information processing system 100 of the third embodiment may include part or all of the configuration of the information processing system of the first embodiment. Further, the information processing system 100 of the third embodiment may include part or all of the configuration of the information processing system of the second embodiment. In the following, description of the matters described in the first embodiment or the second embodiment will be omitted or simplified.

In the following, a case where the information processing system 100 is applied to a service in which an event occurring in the real space is reflected in the virtual space 11 will be described as an example. Specifically, in the following, the process is performed at a specific location (as an example, "in front of store B in town A" in the following) in a predetermined place in real space (as an example, in the following "town A"). An event being held (hereinafter referred to as "live music" as an example) is located at a position corresponding to the specific position in the virtual space 11 imitating the predetermined place (here, in front of store B (in other words, Then, a case will be described in which this is performed in front of a virtual object imitating store B.). That is, in the second embodiment, events taking place in the virtual space 11 are reflected in the real space, whereas in the third embodiment, events taking place in the real space are reflected in the virtual space 11. reflected in Note that the information processing system 100 includes both the configuration according to the second embodiment and the configuration according to the third embodiment, so that events occurring in the virtual space 11 are reflected in the real space, and events occurring in the virtual space 11 are reflected in the real space. It may be configured such that events occurring in the space are reflected in the virtual space 11.

The coordinate definition unit 812 defines the correspondence between coordinates measured by a GPS sensor (in other words, position detection means) and virtual space coordinates (in other words, the correspondence between points in the real space and points in the virtual space 11). ) is stored in the storage unit 611 as a correspondence relationship storage unit. In other words, in this embodiment, information indicating the correspondence between the positions of the real space (in other words, the mixed reality space) and the virtual space 11 is stored in advance in the correspondence storage section.

In this embodiment, as shown in FIG. 16, a detection device 460 is placed in front of store B as a specific location. Furthermore, the detection device 460 is a camera, and is configured to photograph the front of store B as a specific position in real space. Note that a plurality of detection devices 460 (in other words, cameras) may be arranged in front of store B. Further, the detection device 460 (in other words, a camera) in this embodiment is not for detecting the movement of the MR user 6, but may be for simply taking a picture of a predetermined event.

Note that in this embodiment, the MR system 400 may or may not include the MR device 410. For example, MR system 400 may include detection device 460 and computer 500. Further, the computer 500 may be provided outside the detection device 460, or part or all of the computer 500 may be built into the detection device 460. In other words, the MR system 400 in this embodiment or other embodiments can be read as a camera serving as the detection device 460 installed in front of store B in this embodiment.

Furthermore, the detection device 460 is equipped with a GPS sensor, and is capable of acquiring position information indicating the position of the detection device 460. Further, the MR side input reception unit 816 receives input of position information acquired using the GPS sensor. That is, the MR side input receiving unit 816 receives input of position information indicating a specific position. Note that the position information acquired using the GPS sensor can also be said to be position information indicating the position where a predetermined event photographed by the detection device 460 is taking place. Further, when the detection device 460 photographs a predetermined person such as the MR user 6 (in other words, when detecting the movement of the predetermined person), the position information acquired using the GPS sensor is It can also be said to be position information indicating the current position of a person.

The control unit 310 causes the virtual space 11 to reflect a predetermined event that occurs at a specific position in the real space. In this embodiment, the predetermined event is a predetermined event (specifically, a live music event). That is, the control unit 310 controls so that the live music concert that is being held in front of the store B in the real city A is also performed in front of the store B in the virtual space 11.

Specifically, the MR-side input receiving unit 816 (in other words, the control unit 510) receives, as an input, an image captured by an image sensor included in a camera serving as the detection device 460, for example. Here, the image is an image taken of a live music event as a predetermined event. Further, the communication control unit 850 sends information indicating the image and a specific position where the image was acquired to the communication control unit 950, and the control unit 310 transmits these information as shown in FIG. Based on the information, the image is displayed at a position corresponding to the specific position in the virtual space 11 (that is, in front of store B). In other words, when the VR user (in other words, the VR avatar 25) is located in the virtual space 11 at a position corresponding to a specific position in the real space, the control unit 310 displays the image on the display 211 (i.e., the specific image of the event taking place at the location). As a result, in the virtual space 11 (in other words, virtual town A) where you can operate your character and move around, you can move to a position corresponding to the specific position (in other words, in front of virtual store B). It is possible to provide a VR user (in other words, a VR user who operates a character located at a specific location) with a view of an event occurring at the specific location in real space. That is, when the VR user (in other words, the VR avatar 25) is located at a position in the virtual space 11 that corresponds to a specific position in the real space, the control unit 310 controls the Based on the image, the VR user is provided with a view of the virtual space 11 that reflects the event taking place at the specific location in the real space. This allows the VR user to share the experience with a person at the specific location in real space. In this embodiment, information regarding the correspondence between the coordinates measured by the GPS sensor and the virtual space coordinates is stored as information indicating the correspondence between a point in the real space and a point in the virtual space 11. For example, it relates to the correspondence between identification information that enables identification of the detection device 460 installed at a specific position and coordinates indicating a position in the virtual space 11 corresponding to the specific position. The information is stored as information indicating a correspondence relationship between a point in the real space and a point in the virtual space 11, and based on the information, a predetermined event at a position in the virtual space 11 corresponding to a specific position is stored. may be reflected. That is, the identification information that allows the detection device 460 to be identified may function as information indicating a specific position.

Note that the image related to a predetermined event that is displayed on the display 211 by the control unit 310 may be captured by the camera 413 of the MR device 410 worn by the predetermined MR user 6. In other words, the MR device 410 may function as the detection device 460 of this embodiment. In other words, even if the MR device 410 is equipped with a GPS sensor for acquiring positional information regarding a specific position where a predetermined event occurs or a camera for acquiring an image regarding the predetermined event, good. Further, in this case, the MR user 6 who wears the MR device 410 may be a performer or the like of a predetermined event as a predetermined event, or may be an audience member or the like. Furthermore, the acquisition of position information and the acquisition of images related to a predetermined event may be performed by different MR apparatuses 410 (in other words, MR apparatuses 410 used by different users).

Note that a predetermined event that occurs at a specific position in the real space may be reflected in the virtual space 11 as follows. That is, rather than displaying an image taken of a predetermined event as described above, the control unit 310 causes an avatar of a person (here, a performer of a music live performance) related to the predetermined event to perform an action related to the predetermined event. You can let it. Specifically, the MR user 6 in the first embodiment may be replaced with the person, and the avatar of the person as the MR avatar 26 may be moved on the virtual space 11. That is, for example, the MR side input receiving unit 816 receives the output of a detection means for detecting the movement of the person as an input from the person. Specifically, for example, the MR side input reception unit 816 receives information regarding the movement of the MR user 6 from the person, which is detected by image recognition from an image taken by an image sensor included in a camera as the detection device 460. May be accepted as input. Further, the user information acquisition unit 818 acquires identification information that allows the person to be identified. Specifically, the user information acquisition unit 818 acquires the identification information by identifying the MR user 6 in the mixed reality space 21 through image recognition from an image captured by an image sensor included in the detection device 460. Good too. Then, the VR-side object control unit 920 places the avatar of the person (in other words, the MR avatar 26) at a position in the virtual space 11 that corresponds to a specific position in the real space. Further, the VR-side object control unit 920 moves the avatar of the person in the virtual space 11 based on the input from the person received by the MR-side input reception unit 816, for example. Specifically, the VR-side object control unit 920 moves the MR avatar 26 based on, for example, information regarding the movement of the MR user 6 received by the MR-side input reception unit 816. That is, the VR-side object control unit 920 makes the movement of the avatar of the person in the virtual space 11 imitate the movement of the person detected by the detection means that detects the movement of the person in the real space. Then, the movement of the person's avatar is controlled. Thereby, a predetermined event that takes place at a specific position in the real space may be reflected in the virtual space 11. Even with such a configuration, when a VR user (in other words, VR avatar 25) is located at a position in the virtual space 11 corresponding to a specific position in the real space, information regarding a predetermined event photographed by a predetermined camera may be displayed. Based on the image, it is possible to provide the VR user with a view of the virtual space 11 that reflects the event taking place at the particular location in the real space. Specifically, it is possible to provide the VR user with a view of the virtual space 11 in which a predetermined event is reproduced by an avatar of a person photographed by a predetermined camera.

Note that the detection of the movement of the person may be performed by the camera 413 of the MR device 410 worn by a predetermined MR user 6, and the controller 480, the device worn by the MR user 6, the gaze sensor 412, the sensor 417, Alternatively, it may be performed using a sensor 486 or the like. Further, the MR user 6 who wears the MR device 410 may be a performer or the like of a predetermined event as a predetermined event, or may be an audience member or the like. In addition, when detecting a person's movement by image recognition from an image taken by an image sensor provided with a predetermined camera and reflecting it on the avatar in the virtual space 11, the performer (in other words, Only the avatar corresponding to a specific person (identified based on image recognition or presence or absence of a specific device that can track the user's movements) appears in the virtual space 11 (persons other than the specific person). (in other words, 3D models) may be created in real time for all the people seen in the camera so that the avatars of all the people appear in the virtual space 11. .

Note that the virtual space 11 is displayed on the display 411 of the MR device 410 of the MR user 6 (for example, the performer or audience member of the predetermined event) who is in front of the store B, which is the specific location where the predetermined event is taking place. A VR avatar 25 of a VR user viewing the predetermined event may be displayed. For example, when position information indicating that the MR user 6 is at a specific position is input to the MR side input receiving unit 816 of the MR apparatus 410, the control unit 510 of the MR apparatus 410 controls the Information about the VR avatar 25 located at a position in the virtual space 11 may be acquired via the communication control unit 850 and the VR avatar 25 may be displayed on the display 411. That is, when the position information input to the MR side input receiving unit 816 indicates that the MR user 6 is at the specific position, the control unit 510 determines that the MR user 6 is at a position in the virtual space corresponding to the specific position. An image of the VR avatar 25 may be displayed on the display. According to such a configuration, a person located at a location where a predetermined event is occurring in the real space can see the reaction of the VR user in the virtual space 11. Specifically, for example, when a predetermined live performance (for example, a music live performance, a comedy live performance, etc.) is being performed in a real space, performers or audience members of the live performance as the MR user 6 may watch the display 411 of the MR device 410. It becomes possible to visually recognize the VR avatar 25 (in other words, the audience in the virtual space 11) through the VR avatar 25 (in other words, the audience in the virtual space 11). For this reason, performers or spectators at a live performance as MR users 6 can visually grasp how many spectators are gathered, including the virtual space 11, and how each audience member is reacting. becomes possible. Further, for example, it becomes possible for the VR user and the MR user 6 who are friends to view a predetermined event together.

In this case, the MR device 410 on which the VR avatar 25 is displayed and the device that photographs the predetermined event (for example, the camera or MR device 410 installed in front of store B) may be different devices. Often, they may be the same device (specifically, the MR device 410 worn by one MR user 6).

Note that the view of the mixed reality space 21 provided to the MR user 6 who is located at a specific position where a predetermined event is occurring is a view of the mixed reality space 21 that shows virtual objects (specifically, VR For example, an image in which a virtual object (specifically, the VR avatar 25) is superimposed on an image of real space taken by a camera may be provided. It may also be provided by a smartphone or the like displayed on a display. That is, the information processing system 100 according to the present embodiment, for example, when a performer performs a predetermined live performance while taking selfies using his or her smartphone at a specific position in real space, the information processing system 100 displays information on the display of the smartphone. It may also be configured to display a VR avatar 25 (for example, an avatar of an audience member) located at a position in the virtual space 11 corresponding to the specific position. In this case, the display of the smartphone or the like may display an image in which only the VR avatar 25 is displayed, instead of an image in the mixed reality space 21 in which the VR avatar 25 is superimposed on an image in the real space, or an image in which the background is displayed in the virtual space 11. (For example, an image of the VR avatar 25 at a position in the virtual space 11 corresponding to the specific position) may be displayed.

Note that the control unit 310 provides instructions to the VR user (in other words, the VR avatar 25) based on the fact that the VR user (in other words, the VR avatar 25) has visited a position corresponding to a specific position where a predetermined event is occurring in the virtual space 11. It may also include a benefit granting section that grants benefits. The benefit may be, for example, a virtual object (for example, an item that a character can equip) that the user who has acquired the benefit can use in the game (in other words, within the service), or it may be a virtual object that can be used by a user who has acquired the benefit (for example, an item that can be equipped by a character), or a virtual object that can be used in a real space It may be an electronic ticket or the like that can be exchanged with the user, or it may be a predetermined point owned by the user within the service. Furthermore, "based on having visited" may mean directly determining whether or not the person has visited and granting benefits, and specific actions that can be performed by visiting (for example, attending a live music concert). It may also be possible to provide a benefit when viewing (starting, finishing, or continuing for a predetermined period of time) is performed. According to such a configuration, it is possible to motivate the VR user to visit a position corresponding to a specific position in the virtual space 11.

Note that the VR users who can receive a view of the virtual space 11 that reflects a predetermined event occurring at a specific position in the real space may be limited. Specifically, the storage unit 311 functions as a rights information storage unit that stores information regarding the right to receive the view (hereinafter referred to as “rights information”), and the control unit 310 functions as a rights information storage unit that stores information regarding the right to receive the view, and the control unit 310 If the user has the right, the view may be provided to the VR user, and if the user does not have the right, the view may not be provided to the VR user. In addition, the rights are granted to the VR user by, for example, paying a consideration (e.g., payment of cash, consumption of assets held by the VR user (e.g., virtual currency, etc.) within the service, etc.). It may be something. Specifically, for example, the right to view live music may be available for purchase within the service. More specifically, for example, information regarding the presence or absence of a live music ticket that can be purchased within the service is stored as rights information, and the control unit 310 controls whether a VR user who has the ticket can view the live music event. However, the control may be such that VR users who do not have the ticket cannot view the live music event. In other words, information regarding possession of an item (for example, a ticket, etc.) related to the right to receive the view is stored as the right information, and the control unit 310 controls the ownership of the item (for example, ticket, etc.) when the VR user owns the item. Views may be provided to VR users. Note that the right (in other words, the item) may be obtained by completing a predetermined mission within the service.

The control unit 310 also controls a virtual virtual machine in which a predetermined event occurring at a specific position in the real space and related to the MR user 6 selected from a plurality of users by the VR user is reflected. A view of the space 11 may be provided to the VR user. In other words, the VR-side input reception unit 916 receives an input from a VR user regarding an operation (for example, an operation on the controller 280) for selecting which of a plurality of users a predetermined event related to a user is to be reflected. , the control unit 310 may reflect a predetermined event related to the MR user 6 selected by the operation on the view provided to the VR user. The operation may be, for example, an operation in which the VR user selects a user from among his or her own friends to whom the event will be reflected. Further, predetermined events related to all friends are reflected, and the operation may be an operation of registering another user as a friend. In other words, in the service of this embodiment, the VR user may be provided with a view of the virtual space 11 that reflects a predetermined event related to his or her friend. In other words, the VR user may be provided with a view of the virtual space 11 that reflects a predetermined event related to the user associated with the VR user among the plurality of users. For example, when a celebrity selected by a VR user is performing a live performance in the real space, the control unit 310 may reflect the live performance in the view of the virtual space 11 provided to the VR user. good. Note that when there are multiple events related to a specific user, it may be possible to select the specific event related to the specific user as the event to be reflected.

The control unit 310 also controls the virtual space 11 to reflect a predetermined event that is occurring at a specific position in the real space and has an attribute that corresponds to the attribute set for the VR user. The view may be provided to the VR user. Specifically, for example, each VR user has attributes related to the user's hobbies and preferences, more specifically, attributes such as "I like music" and "I like comedy" (or "I like a specific celebrity"). It may be possible to do so. Then, for example, the control unit 310 causes a predetermined event with the attribute "music" to be reflected on the VR user with the attribute "music lover", and the attribute "music lover" is added to the VR user. A predetermined event with the attribute "music" may not be reflected to VR users who have not. Further, for example, the control unit 310 may reflect a predetermined event with the attribute "comedy" on a VR user with the attribute "comedy lover", and add the attribute "comedy lover" to the VR user. A predetermined event with the attribute "comedy" may not be reflected to VR users who have not done so. Note that the attributes set for VR users may be set by each VR user themselves (for example, by inputting their own preferences via the controller 280), and AI (Artificial Intelligence) may be automatically set (for example, according to the user's actions within the service).

As described above, the control unit 310 controls a specific event selected by the VR user (for example, an event related to a specific user selected by the VR user, an event related to a specific user selected by the VR user, An event having an attribute corresponding to the set own attribute, an event related to a ticket purchased by the VR user, etc.) may be reflected in the virtual space 11. In other words, the control unit 310 does not need to reflect in the virtual space 11 an event that is not selected by the VR user among a plurality of events that occur in the real space. In this way, when unselected events are not reflected in the virtual space 11, there may be events that are reflected regardless of the user's selection. In other words, "not reflecting an unselected event in the virtual space 11" means an event that is reflected in the virtual space 11 when a selection is made, and an event that is reflected in the virtual space 11 when a selection is not made. It is sufficient if there is at least one event that is not reflected in 11. Note that the event selection may be possible before the specific event selected by the VR user occurs (in other words, starts), or after the specific event occurs (in other words, the event starts). It may also be possible to do this while the process is being performed.

Note that the information processing system 100 of the second embodiment and the information processing system 100 of the third embodiment can also be applied to games played using both real space and virtual space. A specific example is shown below.

(Exploration game)
For example, the information processing system 100 may be applied to a game in which a character moving within the virtual space 11 is searched for in the real world (hereinafter referred to as a "search game"). Specifically, for example, first, a virtual space 11 imitating a theme park existing in the real world (in other words, a virtual space 11 corresponding to a theme park) is prepared. Further, the VR-side object control unit 920 moves a predetermined character (hereinafter referred to as a "search target character"; in other words, a search target object) so as to move around within the virtual space 11. The search target character may be one that is operated by the VR user (in other words, it moves based on the VR user's operation), or it may be one that moves without the user's operation. In other words, the search target character may be the VR avatar 25 or may be a so-called non-player character. Further, the appearance of the search target character may be, for example, the appearance of a character belonging to a theme park (for example, a character symbolizing the theme park).

The MR system 400 (in other words, the control unit 510) displays the search target character on the display 411 when the MR user 6 is at a position in the real space that corresponds to the position where the search target character exists in the virtual space 11. let In other words, the MR system 400 does not display the search target character on the display 411 when the MR user 6 is not at the position in the real space that corresponds to the position where the search target character exists in the virtual space 11 . In other words, this search game is a game in which a character to be searched who moves around in the virtual space 11 is searched for in the real space.

In addition, the MR side input reception unit 816 receives information about the situation in which the MR user 6 is at a position in the real space corresponding to the position where the search target character is present in the virtual space 11 (in other words, the search target character is displayed on the display 411). (in a situation where the MR user 6 is in a situation where the MR user 6 is in a certain situation), the input related to a specific operation by the MR user 6 is accepted. The specific operation may be, for example, an operation of taking a photo of the search target character displayed on the display 411 (in other words, an operation of storing the image). In other words, the specific operation here can be said to be an operation related to storing information indicating that the search target character has been discovered. Here, the information indicating that the search target character has been found is a photograph of the search target character, but it may also be a flag that is set when the search target character is found. Further, the information may be automatically stored without any specific operation. In other words, the information indicating that the search target character has been found is stored based on the fact that the MR user 6 has visited a position in the real space that corresponds to the position where the search target character exists in the virtual space 11. That's fine. That is, the control unit 510 causes the storage unit 511 to store information indicating that the operation target character has been discovered, based on the fact that the MR user 6 has visited the relevant position in the real space.

Furthermore, the control unit 510 provides a benefit to the MR user based on the fact that the MR user 6 has visited the relevant position in the real space (in other words, when information indicating that the search target character has been discovered is stored). Give. The benefit may be an item that can be used in the virtual space 11. Further, the benefit may be an electronic certificate or trophy indicating that the character to be searched for has been discovered. Further, the benefit may be an electronic ticket or the like that can be exchanged for a real object (for example, goods of a character to be searched for) in a real space (for example, at a real theme park).

Furthermore, in this search game, the VR user may be able to send information to the MR user 6 to assist in the search. Note that the VR user may be a user who operates the search target character, or may be a user different from the user. Specifically, for example, the VR user operates his or her VR avatar 25 in the virtual space 11 to search for the search target character in the virtual space 11 imitating a theme park. Further, the VR-side input reception unit 916 performs an operation (by the VR user) of sending sighting information (for example, information about the place where the character was sighted; information that helps the search) of the character to be searched to the MR user 6 ( For example, input related to an operation on the controller 280 is accepted. Further, the control unit 510 receives the sighting information via the communication control unit 850, and causes a predetermined output means to perform an output that helps in the search based on the sighting information. Specifically, for example, the control unit 510 causes the display 411 to display a display notifying the location where the search target character has been sighted in the virtual space 11, or causes the speaker 416 to display a sound indicating the location where the search target character has been sighted. or output it. According to such a configuration, it is possible for the MR user 6 who is actually going to the theme park and the VR user who is not actually at the theme park (for example, at home, etc.) to cooperate and enjoy the game. becomes.

(survival game)
Further, for example, the information processing system 100 may be applied to a survival game in which a user playing in the virtual space 11 and a user playing in the real space can compete against each other. Specifically, for example, first, a venue where a user in the real space plays is prepared, and a virtual space 11 imitating the venue (in other words, a virtual space 11 imitating the venue). Note that the venue may be a so-called survival game field or a predetermined area such as a town. Further, the user on the real space side wears the MR device 410, and the user on the virtual space 11 side wears the VR device 210.

The control unit 310 of the VR system 200 controls the VR avatar 25 of the VR user based on the input from the VR user accepted by the VR side input reception unit 916. Specifically, the control unit 310 moves the VR avatar 25 in the virtual space 11 based on input from the VR user.

Further, the control unit 310 of the VR system 200 receives the input from the MR user 6 accepted by the MR side input reception unit 816 via the communication control unit 950, and controls the MR avatar 26. Specifically, the control unit 310 uses the position information (that is, the position information indicating the position of the MR user 6 in real space) acquired by the GPS sensor (in other words, position detection means) included in the MR apparatus 410 to , moves the MR avatar 26 in the virtual space 11. More specifically, the control unit 310 moves the MR avatar 26 so that the MR avatar 26 is located at a position in the virtual space 11 that corresponds to the position where the MR user 6 is located in the venue on the real space side. That is, on the display 211 of the VR device 210, it appears that the MR user 6 is present at a position in the virtual space 11 (in other words, inside the venue) that corresponds to the position where the MR user 6 is in the mixed reality space 21. A visual display is provided. Note that in order to more accurately obtain the position of the MR user 6, height information of the MR device 410 and the like may be obtained via the sensor 417 of the MR device 410 and the like.

Further, the control unit 510 of the MR system 400 moves the VR avatar 25 so that the VR avatar 25 is located at a position in the mixed reality space 21 that corresponds to the position where the VR avatar 25 is placed in the virtual space 11. . That is, the display 411 of the MR device 410 displays a display that makes it appear that the VR user is present at a position in the mixed reality space 21 (in other words, inside the venue) that corresponds to the position where the VR user is in the virtual space. is provided.

Further, the VR-side input reception unit 916 receives input related to an attack on the MR user 6 (in other words, the MR avatar 26) by the VR user. Further, the MR side input reception unit 816 receives input related to an attack on the VR user (in other words, the VR avatar 25) by the MR user 6. Note that the attack on the other party by these VR users or MR users 6 may be, for example, shooting a gun at the other party, and the input operation related to the attack is performed using the gun-shaped controllers 280, 480. It may also be done via a

Further, the control unit 310 determines whether the attack on the MR user 6 by the VR user hits the MR user 6 or not. Specifically, the control unit 310 controls, for example, when an input operation is performed by the VR user while a gun held by the VR avatar 25 is pointed at the MR avatar 26, and a bullet fired from the gun directly hits the MR avatar 26. In this case, it is determined that the attack by the VR user has hit the MR user 6. Furthermore, the control unit 310 determines whether the attack on the VR user by the MR user 6 hits the VR user. Specifically, the control unit 310 controls, for example, when an input operation is performed by the MR user 6 while a gun held by the MR avatar 26 is pointed at the VR avatar 25, and a bullet fired from the gun directly hits the VR avatar 25. In this case, it is determined that the attack by the MR user 6 has hit the VR user. Here, the direction in which the gun held by the MR avatar 26 faces is linked to, for example, the direction of the controller 480 held by the MR user 6, and the aiming direction of the gun held by the MR avatar 26 is The controller 480 can be aimed at a position in the virtual space 11 that corresponds to the position the controller 480 is aiming at. That is, when the MR user 6 performs an input operation while aiming the gun-shaped controller 480 at the VR avatar 25 in the mixed reality space 21, the control unit 310 determines whether an attack by the MR user 6 occurs. It is determined that it has hit a VR user. Note that the attack on the MR user 6 by the VR user can be the same as in a normal shooting game. Note that part or all of the determination regarding the attack on the VR user by the MR user 6 or the attack on the MR user 6 by the VR user may be performed by the control unit 510, the control unit 610, or the like. In other words, the determination unit that determines whether the MR user 6 attacks the VR user 6 or the determination unit that determines whether the VR user attacks the MR user 6 are the control unit 310, the control unit 510, and the control unit 610. It may be realized by any of the following.

Although the case where the VR user and the MR user 6 fight as enemies has been explained here as an example, in the survival game (in other words, the battle game), the VR user and the MR user 6 play cooperatively. may be possible. That is, for example, in the case of a survival game in which multiple teams (for example, two teams) made up of multiple users can compete against each other, one of the multiple teams is a VR user (in other words, Then, the other team may be composed only of the MR users 6 (in other words, the users of the real space). Furthermore, one team may include both VR users and MR users 6.

(Escape Game)
Further, for example, the information processing system 100 may be applied to an escape game (in other words, a puzzle solving game) played cooperatively by a user in the real space and a user in the virtual space 11. Specifically, for example, first, a venue where a user in the real space plays is prepared, and a virtual space 11 imitating the venue (in other words, a virtual space 11 imitating the venue). Note that the venue may be a predetermined area such as a town, or may be a specific facility. Further, the user on the real space side wears the MR device 410, and the user on the virtual space 11 side wears the VR device 210.

The user wearing the MR device 410 and the user wearing the VR device 210 work together to accomplish a set mission. Specifically, the MR device 410 and the VR device 210 display virtual objects related to the mission at corresponding positions. The virtual object may be, for example, an object related to the reception of an action performed by the VR user and the MR user 6 in cooperation. Specifically, the VR user performs a predetermined action (for example, an operation of waving his hand) on a predetermined device as the virtual object displayed on the VR device 210, and the MR user 6 performs a predetermined action on the predetermined device as the virtual object displayed on the VR device 210. perform a predetermined action (for example, an operation of waving one's hand over the predetermined device as the virtual object; the action may be the same as or different from the action performed by the VR user). Then, the predetermined device starts up (in other words, the game (event) progresses). Further, the virtual object may be related to an item, such as a treasure chest, that can be obtained by at least one of the VR user and the MR user 6, for example. Further, the virtual object may be an object that includes a description related to solving a mystery (for example, a text about the mystery or a hint for solving the mystery is written).

Furthermore, in a game that is executed using such an information processing system 100 and is played cooperatively between the MR user 6 and the VR user, one of the MR user 6 and the VR user controls the progress of the game. The second action related to the progress of the game may be made possible by the other player on the condition that the first action related to the first action has been performed. In this case, for example, the first action is an action that can only be performed by the one user and cannot be performed by the other user. Further, for example, the second action is an action that can only be performed by the other user and cannot be performed by the one user. Specifically, for example, when the MR system 400 (in other words, the control unit 510) detects that the MR user 6 has visited a specific place in real space (for example, the MR system 400 (in other words, the control unit 510) detects that the MR user 6 has visited a specific place, When the GPS information is input to the MR side input reception unit 816), the VR system 200 (in other words, the control unit 310) may accept an input related to the second action by the VR user. In other words, the second action by the VR user may be an action that becomes executable based on the MR user 6 visiting a specific place in real space. Further, for example, when the VR system 200 (in other words, the control unit 310) detects that the VR user has visited a specific place in the virtual space 11, the MR system 400 (in other words, the control unit 510) detects that the MR user For example, an input related to the second action in step 6 may be accepted. In other words, the second action by the MR user 6 may be an action that becomes executable based on the VR user visiting a specific place in the virtual space 11. Further, for example, only while the MR user 6 is pointing the camera of the smartphone serving as the MR device 410 at a specific location in the real space, the VR device 210 displays a predetermined display related to the progress of the game (for example, when solving puzzles). Such a hint may be displayed on the display 211. Further, the virtual space 11 represents the past of the mixed reality space 21, and the VR system 200 performs a predetermined action (for example, It is also possible to accept an input related to an action of placing an item at a specific position inside. Then, when the VR user performs the first action in the virtual space 11 (in other words, in the past), the MR system 400 performs a predetermined action as the second action by the MR user 6 in the real space (in other words, in the present). (For example, an action of going to a location in the mixed reality space 21 corresponding to the specific location and discovering the item) and proceeding with the game (in other words, an event) based on the input. The configuration may be such that this is possible.

Note that the present invention is not limited to the embodiments described above, and can be implemented with various modifications without departing from the gist thereof. Within the scope of the present invention, each component can be freely combined, any component can be modified, or any component can be omitted. Further, the process flow described in this specification is merely an example, and the order and configuration of each process may be different.

<Additional notes>
The matters described in the above embodiments may also be described as in the following additional notes.

(Additional note 1)
a first control means (e.g., control unit 310) that provides a first user with a first view that is a view of a virtual space;
a second control means (for example, control unit 510) that provides a second user with a second view that is a view of a mixed reality space in which virtual objects are placed in the real space;
input receiving means (for example, MR side input receiving unit 816) that receives position information indicating the current position of the second user in the real space;
Correspondence storage means (for example, storage unit 611) for storing information indicating the correspondence between the positions of the real space and the virtual space,
The first control means provides the first user with the first view regarding a predetermined event occurring at a specific position in the virtual space;
The second control means
providing the second user with the second view according to the location information;
When the location information indicates that the second user is at a location corresponding to the specific location, the second user is configured to reflect the predetermined event taking place at the specific location in the virtual space. An information processing system that provides the second user with a view of.
According to such a configuration, when the second user is at a predetermined position in the real space, a composite image that reflects a predetermined event occurring at a specific position in the virtual space corresponding to the predetermined position is displayed. It becomes possible to provide a view of the real space to the second user. Therefore, an event occurring at a specific location in virtual space can be viewed at a location corresponding to the specific location in real space. Therefore, the range of uses of virtual space is expanded.

(Additional note 2)
The first control means moves a virtual object related to the predetermined event in the virtual space based on the first user's operation;
The second control means causes the second view to reflect the movement of the virtual object being moved by the first user's operation, and causes the second view in which the predetermined event is reflected to be displayed by the second user. The information processing system described in Appendix 1.
According to such a configuration, the second user can see in the real space how the virtual object that the first user moves in the virtual space is moving. Therefore, it becomes possible to provide the user with an unprecedented experience, and the range of uses of virtual space is expanded.

(Additional note 3)
Supplementary Note 1: The apparatus further includes a benefit granting means (for example, a control unit 510) that grants a benefit to the second user based on the fact that the second user has visited a position corresponding to the specific position in the real space. The information processing system described in .
According to such a configuration, a benefit can be given to the second user who visits a position in the real space where a predetermined event occurring in the virtual space is reflected. Therefore, the second user can be motivated to visit a position corresponding to a specific position in the real space, and the range of uses of the virtual space is expanded.

(Additional note 4)
The predetermined event reflected in the second view is an event selected by the second user from among a plurality of events that take place in the virtual space. Information processing system.
According to such a configuration, it becomes possible for the second user to select the event to be reflected from among a plurality of events. Therefore, in the mixed reality space provided to the second user, it is possible to prevent the processing from becoming too heavy due to an increase in the number of reflected events, or from cluttering the second user's field of view. can.

(Appendix 5)
comprising a rights information storage means (for example, a storage unit 511) that stores information regarding the right to receive the provision of the second view in which the predetermined event is reflected;
The right is granted to the second user by paying a consideration,
In any one of Supplementary Notes 1 to 3, the second control means provides the second user with the second view in which the predetermined event is reflected when the second user has the right. The information processing system described.
According to such a configuration, it is possible to demand compensation from the second user who wishes to view a predetermined event in real space. Therefore, the range of uses of virtual space is expanded.

(Appendix 6)
The second control means provides the second user with the second view in which the predetermined event related to the first user selected by the second user from a plurality of users is reflected. The information processing system according to any one of ~3.
According to such a configuration, the second user can select which user from among the plurality of users who use the information processing system will receive the reflection of an event. Therefore, the range of uses of virtual space is expanded.

(Appendix 7)
The second control means provides the second user with the second view that reflects the predetermined event and has an attribute that corresponds to the attribute set for the second user. The information processing system described in item 1.
According to such a configuration, it is possible to provide a view that reflects events suitable for the second user, depending on the attributes of the second user. Therefore, the range of uses of virtual space is expanded.

(Appendix 8)
a first control means (e.g., control unit 310) that provides a first user with a first view that is a view of a virtual space;
a second control means (for example, control unit 510) that provides a second user with a second view that is a view of a mixed reality space in which virtual objects are placed in the real space;
input receiving means (for example, MR side input receiving unit 816) that receives position information indicating the current position of the second user in the real space;
A program for causing a computer of an information processing system to function as the second control means, including a correspondence storage unit (for example, a storage unit 611) that stores information indicating a correspondence relationship between positions of the real space and the virtual space. And,
The first control means provides the first user with the first view regarding a predetermined event occurring at a specific position in the virtual space;
The second control means
providing the second user with the second view according to the location information;
When the location information indicates that the second user is at a location corresponding to the specific location, the second user is configured to reflect the predetermined event taking place at the specific location in the virtual space. A program that provides the second user with a view of.
According to such a configuration, the same effects as the information processing system described in Supplementary Note 1 can be achieved.

11 virtual space, 21 mixed reality space, 25 VR avatar, 26 MR avatar, 30 virtual model, 35 blackboard, 36 virtual blackboard, 50 event-related object, 100 information processing system, 200 VR system, 210 VR device, 211 display, 212 gaze sensor, 213 first camera, 214 second camera, 215 microphone, 216 speaker, 217 sensor, 260 detection device, 270 display, 280 controller, 286 sensor, 300 computer, 301 processor, 302 memory, 303 storage, 304 input/output interface, 305 communication interface, 310 control unit, 311 storage unit, 400 MR system, 410 MR device, 411 display, 412 gaze sensor, 413 camera, 415 microphone, 416 speaker, 417 sensor, 460 detection device, 470 display, 480 controller , 486 sensor, 500 computer, 501 processor, 502 memory, 503 storage, 504 input/output interface, 505 communication interface, 510 control unit, 511 storage unit, 600 server, 601 processor, 602 memory, 603 storage, 604 input/output interface, 605 communication interface, 610 control unit, 611 storage unit, 700 external device, 810 virtual space generation unit, 812 coordinate definition unit, 816 MR side input reception unit, 818 user information acquisition unit, 820 MR side object control unit, 840 display control part, 845 sound control unit, 850 communication control unit, 916 VR side input reception unit, 920 VR side object control unit, 930 virtual camera control unit, 940 display control unit, 945 sound control unit, 950 communication control unit

Claims (7)

a first control means for providing a first user with a first view that is a view of a virtual space;
a second control means for providing a second user with a second view that is a view of a mixed reality space in which virtual objects are placed in the real space;
input receiving means for receiving position information indicating the current position of the second user in the real space;
Correspondence storage means for storing information indicating a correspondence between the positions of the real space and the virtual space,
The first control means provides the first user with the first view regarding a predetermined event occurring at a specific position in the virtual space;
The second control means
providing the second user with the second view according to the location information;
the predetermined event occurring at the specific location in the virtual space when the location information indicates that the second user is at a location corresponding to the specific location; An information processing system that provides the second user with the second view in which the predetermined event related to the first user selected by the user from among a plurality of users is reflected. The first control means moves a virtual object related to the predetermined event in the virtual space based on the first user's operation;
The second control means causes the second view to reflect the movement of the virtual object being moved by the first user's operation, and causes the second view in which the predetermined event is reflected to be displayed by the second user. The information processing system according to claim 1 . The information processing system according to claim 1, further comprising a privilege granting unit that grants a privilege to the second user based on the second user visiting a position corresponding to the specific position in the real space. . The predetermined event reflected in the second view is an event selected by the second user from among a plurality of events that take place in the virtual space. information processing system. comprising a rights information storage means for storing information regarding the right to receive the provision of the second view in which the predetermined event is reflected;
The right is granted to the second user by paying a consideration,
4. The second control means, when the second user has the right, provides the second view in which the predetermined event is reflected to the second user. The information processing system described in . The second control means provides the second user with the second view in which the predetermined event is reflected and has an attribute corresponding to an attribute set for the second user. The information processing system according to item 1. a first control means for providing a first user with a first view that is a view of a virtual space;
a second control means for providing a second user with a second view that is a view of a mixed reality space in which virtual objects are placed in the real space;
input receiving means for receiving position information indicating the current position of the second user in the real space;
A program for causing a computer of an information processing system to function as the second control means, including a correspondence storage means for storing information indicating a correspondence between positions of the real space and the virtual space, the program comprising:
The first control means provides the first user with the first view regarding a predetermined event occurring at a specific position in the virtual space;
The second control means
providing the second user with the second view according to the location information;
the predetermined event occurring at the specific location in the virtual space when the location information indicates that the second user is at a location corresponding to the specific location; The program provides the second user with the second view in which the predetermined event related to the first user selected by the user from among a plurality of users is reflected.

Images