JP7488867B2 - Programs and Systems - Google Patents

Description

The present invention relates to a program.

Patent document 1 discloses an example of a technology that allows a user to view content in a virtual space.

JP 2017-176728 A

Conventional technology leaves room for better control of avatar objects.

One aspect of the present disclosure aims to more appropriately control avatar objects.

According to one aspect of the present invention, a program executed by a computer having a processor and a memory is provided. The program causes the processor to execute a definition step of defining a virtual space including an avatar associated with a first user, a detection step of detecting a movement of the first user, and a control step of determining information on a new movement that the avatar can perform based on a first movement of the avatar, and controlling the avatar based on the detected movement of the first user and the information on the determined movement.

According to one aspect of the present disclosure, avatar objects can be controlled more appropriately.

FIG. 1 is a diagram illustrating an outline of the configuration of an HMD system according to an embodiment. FIG. 1 is a block diagram illustrating an example of a hardware configuration of a computer according to an embodiment. FIG. 2 is a diagram conceptually illustrating a uvw field of view coordinate system set in an HMD according to an embodiment. FIG. 1 is a diagram conceptually illustrating one mode of expressing a virtual space according to an embodiment. 1 is a top view of a user's head wearing an HMD according to an embodiment. 1 is a diagram showing a YZ cross section of a field of view in a virtual space as viewed from an X direction. 13 is a diagram showing an XZ cross section of a field of view in a virtual space as viewed from a Y direction. FIG. 2 is a diagram illustrating a schematic configuration of a controller according to an embodiment. FIG. 2 illustrates an example of yaw, roll, and pitch directions defined relative to a user's right hand according to one embodiment. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a server according to an embodiment. FIG. 1 is a block diagram illustrating a modular configuration of a computer according to an embodiment. 11 is a sequence chart showing a part of a process executed in an HMD set according to an embodiment. 1 is a schematic diagram showing a situation in which each HMD provides a virtual space to a user in a network. FIG. 13 is a diagram showing a field of view image of user 5A in FIG. FIG. 11 is a sequence diagram showing a process executed in an HMD system according to an embodiment. FIG. 2 is a block diagram showing a detailed configuration of modules of a computer according to an embodiment. 1 is a diagram illustrating an outline of a configuration of a distribution system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a hardware configuration of a user terminal according to an embodiment. FIG. 2 is a block diagram showing a detailed configuration of modules of a user terminal according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 13 is a diagram for explaining a method for acquiring dimensional data. FIG. 2 is a diagram illustrating an example of a data structure of location information according to an embodiment. FIG. 4 illustrates an example of a data structure of dimension data according to an embodiment. 1 is a flow chart illustrating a process for obtaining dimensional data according to one embodiment. FIG. 13 illustrates an example of a data structure for a rotation direction according to an embodiment. 1 is a sequence chart illustrating a part of a process executed in a distribution system according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 1 illustrates an example of a user's posture according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display screen of a user terminal in one embodiment. FIG. 2 illustrates a number of facial expressions that an avatar object can have, according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 illustrates a display screen of a user terminal according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 1 illustrates a trained model according to an embodiment. 1 is a sequence chart illustrating a part of a process executed in a distribution system according to an embodiment. FIG. 1 illustrates an example of a user's posture according to an embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 is a diagram showing a display screen of a user terminal according to an embodiment. FIG. 2 is a diagram showing a display screen of a user terminal according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. FIG. 1 illustrates a trained model according to an embodiment. FIG. 2 is a diagram depicting a user and an avatar object according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. 11 is a sequence chart showing a part of a process executed in an HMD set according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. 11 is a sequence chart showing a part of a process executed in an HMD set according to an embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 is a diagram showing a display screen of a user terminal in one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. FIG. 2 illustrates a virtual space and view image according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment. A diagram showing a virtual space and a display surface of a user terminal according to one embodiment.

[Embodiment 1]
Hereinafter, the embodiment of this technical idea will be described in detail with reference to the drawings. In the following description, the same parts are given the same reference numerals. Their names and functions are also the same. Therefore, detailed descriptions thereof will not be repeated. In one or more embodiments shown in this disclosure, the elements included in each embodiment can be combined with each other, and the combined result also forms part of the embodiment shown in this disclosure.

[Configuration of HMD system]
The configuration of an HMD (Head-Mounted Device) system 100 will be described with reference to Fig. 1. Fig. 1 is a diagram showing an outline of the configuration of the HMD system 100 according to the present embodiment. The HMD system 100 is provided as a system for home use or a system for commercial use.

The HMD system 100 includes a server 600, HMD sets 110A, 110B, 110C, and 110D, an external device 700, and a network 2. Each of the HMD sets 110A, 110B, 110C, and 110D is configured to be able to communicate with the server 600 and the external device 700 via the network 2. Hereinafter, the HMD sets 110A, 110B, 110C, and 110D are collectively referred to as the HMD set 110. The number of HMD sets 110 constituting the HMD system 100 is not limited to four, and may be three or less, or five or more. The HMD set 110 includes an HMD 120, a computer 200, an HMD sensor 410, a display 430, and a controller 300. The HMD 120 includes a monitor 130, a gaze sensor 140, a first camera 150, a second camera 160, a microphone 170, and a speaker 180. The controller 300 may include a motion sensor 420.

In one aspect, the computer 200 can be connected to the Internet or other network 2, and can communicate with a server 600 or other computers connected to the network 2. Examples of other computers include computers of other HMD sets 110 and external devices 700. In another aspect, the HMD 120 can include a sensor 190 instead of the HMD sensor 410.

The HMD 120 is worn on the head of the user 5 and can provide a virtual space to the user 5 during operation. More specifically, the HMD 120 displays an image for the right eye and an image for the left eye on the monitor 130. When each eye of the user 5 views the respective image, the user 5 can recognize the image as a three-dimensional image based on the parallax between the two eyes. The HMD 120 can include both a so-called head-mounted display equipped with a monitor and a head-mounted device to which a smartphone or other terminal equipped with a monitor can be attached.

The monitor 130 is realized, for example, as a non-transparent display device. In one aspect, the monitor 130 is disposed on the main body of the HMD 120 so as to be positioned in front of both eyes of the user 5. Thus, when the user 5 visually recognizes the three-dimensional image displayed on the monitor 130, the user 5 can be immersed in the virtual space. In one aspect, the virtual space includes, for example, images of a background, objects that the user 5 can operate, and menus that the user 5 can select. In one aspect, the monitor 130 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor provided in a so-called smartphone or other information display terminal.

In another aspect, the monitor 130 may be realized as a transmissive display device. In this case, the HMD 120 may be an open type such as a pair of glasses, rather than a closed type that covers the eyes of the user 5 as shown in FIG. 1. The transmissive monitor 130 may be temporarily configurable as a non-transmissive display device by adjusting its transmittance. The monitor 130 may include a configuration that simultaneously displays a portion of an image that constitutes a virtual space and the real space. For example, the monitor 130 may display an image of the real space captured by a camera mounted on the HMD 120, or may make the real space visible by setting the transmittance of a portion of the image high.

In one aspect, the monitor 130 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 130 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the monitor 130 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is perceived by only one of the eyes.

In one aspect, the HMD 120 includes multiple light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 410 has a position tracking function for detecting the movement of the HMD 120. More specifically, the HMD sensor 410 reads multiple infrared rays emitted by the HMD 120 and detects the position and tilt of the HMD 120 in real space.

In another aspect, the HMD sensor 410 may be realized by a camera. In this case, the HMD sensor 410 can detect the position and inclination of the HMD 120 by performing image analysis processing using image information of the HMD 120 output from the camera.

In another aspect, the HMD 120 may include a sensor 190 as a position detector instead of or in addition to the HMD sensor 410. The HMD 120 may use the sensor 190 to detect the position and inclination of the HMD 120 itself. For example, if the sensor 190 is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 120 may use any of these sensors instead of the HMD sensor 410 to detect the position and inclination of the HMD 120 itself. As an example, if the sensor 190 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD 120 around three axes in real space over time. The HMD 120 calculates the change in angle of the HMD 120 around the three axes over time based on each angular velocity, and further calculates the inclination of the HMD 120 based on the change in angle over time.

The gaze sensor 140 detects the direction in which the gaze of the right eye and left eye of the user 5 is directed. In other words, the gaze sensor 140 detects the gaze of the user 5. The detection of the gaze direction is realized, for example, by a known eye tracking function. The gaze sensor 140 is realized by a sensor having the eye tracking function. In one aspect, the gaze sensor 140 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 140 may be, for example, a sensor that irradiates the right eye and left eye of the user 5 with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. The gaze sensor 140 can detect the gaze of the user 5 based on each detected rotation angle.

The first camera 150 photographs the lower part of the face of the user 5. More specifically, the first camera 150 photographs the nose and mouth of the user 5. The second camera 160 photographs the eyes and eyebrows of the user 5. The housing of the HMD 120 on the user 5 side is defined as the inside of the HMD 120, and the housing of the HMD 120 on the opposite side to the user 5 is defined as the outside of the HMD 120. In one aspect, the first camera 150 may be disposed outside the HMD 120, and the second camera 160 may be disposed inside the HMD 120. The images generated by the first camera 150 and the second camera 160 are input to the computer 200. In another aspect, the first camera 150 and the second camera 160 may be realized as a single camera, and the face of the user 5 may be photographed by this single camera.

The microphone 170 converts the speech of the user 5 into an audio signal (electrical signal) and outputs it to the computer 200. The speaker 180 converts the audio signal into sound and outputs it to the user 5. In another aspect, the HMD 120 may include earphones instead of the speaker 180.

The controller 300 is connected to the computer 200 by wire or wirelessly. The controller 300 accepts input of commands from the user 5 to the computer 200. In one aspect, the controller 300 is configured to be held by the user 5. In another aspect, the controller 300 is configured to be attached to a part of the body or clothing of the user 5. In yet another aspect, the controller 300 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the computer 200. In yet another aspect, the controller 300 accepts operations from the user 5 to control the position and movement of an object placed in a virtual space.

In one aspect, the controller 300 includes multiple light sources. Each light source is realized, for example, by an LED that emits infrared light. The HMD sensor 410 has a position tracking function. In this case, the HMD sensor 410 reads multiple infrared rays emitted by the controller 300 and detects the position and tilt of the controller 300 in real space. In another aspect, the HMD sensor 410 may be realized by a camera. In this case, the HMD sensor 410 can detect the position and tilt of the controller 300 by performing image analysis processing using image information of the controller 300 output from the camera.

In one aspect, the motion sensor 420 is attached to the hand of the user 5 and detects the movement of the hand of the user 5. For example, the motion sensor 420 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the computer 200. The motion sensor 420 is provided, for example, in the controller 300. In one aspect, the motion sensor 420 is provided, for example, in the controller 300 configured to be held by the user 5. In another aspect, for safety in the real space, the controller 300 is attached to something that is not easily blown away by being worn on the hand of the user 5, such as a glove type. In yet another aspect, a sensor that is not attached to the user 5 may detect the movement of the hand of the user 5. For example, a signal from a camera that captures the user 5 may be input to the computer 200 as a signal representing the movement of the user 5. The motion sensor 420 and the computer 200 are connected to each other wirelessly, as an example. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth (registered trademark) or other known communication methods are used.

The display 430 displays an image similar to the image displayed on the monitor 130. This allows users other than the user 5 wearing the HMD 120 to view an image similar to that of the user 5. The image displayed on the display 430 does not need to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 430 include a liquid crystal display and an organic EL monitor.

The server 600 may transmit a program to the computer 200. In another aspect, the server 600 may communicate with other computers 200 to provide virtual reality to the HMD 120 used by other users. For example, in an amusement facility, when multiple users play a participatory game, each computer 200 communicates signals based on the actions of each user with the other computers 200 via the server 600, allowing multiple users to enjoy a common game in the same virtual space. Each computer 200 may also communicate signals based on the actions of each user with the other computers 200 without going through the server 600.

The external device 700 may be any device capable of communicating with the computer 200. The external device 700 may be, for example, a device capable of communicating with the computer 200 via the network 2, or a device capable of directly communicating with the computer 200 via short-range wireless communication or a wired connection. Examples of the external device 700 include, but are not limited to, smart devices, PCs (Personal Computers), and peripheral devices of the computer 200.

[Computer hardware configuration]
A computer 200 according to the present embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of a hardware configuration of the computer 200 according to the present embodiment. The computer 200 includes, as main components, a processor 210, a memory 220, a storage 230, an input/output interface 240, and a communication interface 250. Each component is connected to a bus 260.

The processor 210 executes a series of instructions contained in a program stored in the memory 220 or the storage 230 based on a signal provided to the computer 200 or based on the satisfaction of a predetermined condition. In one aspect, the processor 210 is realized as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an MPU (Micro Processor Unit), an FPGA (Field-Programmable Gate Array), or other device.

Memory 220 temporarily stores programs and data. Programs are loaded, for example, from storage 230. Data includes data input to computer 200 and data generated by processor 210. In one aspect, memory 220 is realized as RAM (Random Access Memory) or other volatile memory.

The storage 230 permanently holds programs and data. The storage 230 is realized, for example, as a ROM (Read-Only Memory), a hard disk drive, a flash memory, or other non-volatile storage device. The programs stored in the storage 230 include a program for providing a virtual space in the HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with other computers 200. The data stored in the storage 230 includes data and objects for defining the virtual space, etc.

In another aspect, the storage 230 may be realized as a removable storage device such as a memory card. In yet another aspect, a configuration may be used in which programs and data stored in an external storage device are used instead of the storage 230 built into the computer 200. With such a configuration, for example, in a situation where multiple HMD systems 100 are used, such as an amusement facility, it becomes possible to update programs and data collectively.

The input/output interface 240 communicates signals between the HMD 120, the HMD sensor 410, the motion sensor 420, and the display 430. The monitor 130, the gaze sensor 140, the first camera 150, the second camera 160, the microphone 170, and the speaker 180 included in the HMD 120 can communicate with the computer 200 via the input/output interface 240 of the HMD 120. In some aspects, the input/output interface 240 is realized using a terminal such as a Universal Serial Bus (USB), a Digital Visual Interface (DVI), a High-Definition Multimedia Interface (HDMI (registered trademark)), or other terminal. The input/output interface 240 is not limited to those mentioned above.

In one aspect, the input/output interface 240 may further communicate with the controller 300. For example, the input/output interface 240 receives input of signals output from the controller 300 and the motion sensor 420. In another aspect, the input/output interface 240 sends instructions output from the processor 210 to the controller 300. The instructions instruct the controller 300 to vibrate, output sound, emit light, etc. When the controller 300 receives the instructions, it executes either vibration, sound output, or light emission according to the instructions.

The communication interface 250 is connected to the network 2 and communicates with other computers (e.g., the server 600) connected to the network 2. In one aspect, the communication interface 250 is realized as, for example, a LAN (Local Area Network) or other wired communication interface, or a WiFi (Wireless Fidelity), Bluetooth (registered trademark), NFC (Near Field Communication), or other wireless communication interface. The communication interface 250 is not limited to the above.

In one aspect, the processor 210 accesses the storage 230, loads one or more programs stored in the storage 230 into the memory 220, and executes a series of instructions contained in the programs. The one or more programs may include an operating system for the computer 200, an application program for providing a virtual space, game software executable in the virtual space, and the like. The processor 210 sends a signal for providing the virtual space to the HMD 120 via the input/output interface 240. The HMD 120 displays an image on the monitor 130 based on the signal.

2, the computer 200 is shown as being provided outside the HMD 120, but in another aspect, the computer 200 may be built into the HMD 120. As an example, a portable information and communication terminal (e.g., a smartphone) including the monitor 130 may function as the computer 200.

The computer 200 may be configured to be shared by multiple HMDs 120. With such a configuration, for example, the same virtual space can be provided to multiple users, allowing each user to enjoy the same application as other users in the same virtual space.

In one embodiment, a real coordinate system, which is a coordinate system in real space, is set in advance in the HMD system 100. The real coordinate system has three reference directions (axes) that are parallel to the vertical direction in real space, the horizontal direction perpendicular to the vertical direction, and the front-to-back direction perpendicular to both the vertical and horizontal directions. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the real coordinate system are defined as the x-axis, y-axis, and z-axis, respectively. More specifically, in the real coordinate system, the x-axis is parallel to the horizontal direction in real space. The y-axis is parallel to the vertical direction in real space. The z-axis is parallel to the front-to-back direction in real space.

In one aspect, the HMD sensor 410 includes an infrared sensor. When the infrared sensor detects infrared rays emitted from each light source of the HMD 120, it detects the presence of the HMD 120. The HMD sensor 410 further detects the position and tilt (orientation) of the HMD 120 in real space according to the movement of the user 5 wearing the HMD 120, based on the values of each point (each coordinate value in the real coordinate system). More specifically, the HMD sensor 410 can detect changes over time in the position and tilt of the HMD 120, using each value detected over time.

Each tilt of the HMD 120 detected by the HMD sensor 410 corresponds to each tilt around the three axes of the HMD 120 in the real coordinate system. The HMD sensor 410 sets a uvw field of view coordinate system in the HMD 120 based on the tilt of the HMD 120 in the real coordinate system. The uvw field of view coordinate system set in the HMD 120 corresponds to the viewpoint coordinate system when the user 5 wearing the HMD 120 views an object in the virtual space.

[uvw visual coordinate system]
The uvw field of view coordinate system will be described with reference to Fig. 3. Fig. 3 is a diagram conceptually showing the uvw field of view coordinate system set in the HMD 120 according to an embodiment. The HMD sensor 410 detects the position and inclination of the HMD 120 in the real coordinate system when the HMD 120 is started. The processor 210 sets the uvw field of view coordinate system in the HMD 120 based on the detected values.

As shown in FIG. 3, the HMD 120 sets a three-dimensional uvw field of view coordinate system with the head of the user 5 wearing the HMD 120 as its center (origin). More specifically, the HMD 120 tilts the horizontal, vertical, and front-back directions (x-axis, y-axis, z-axis) that define the real coordinate system around each axis by the tilt of the HMD 120 around each axis in the real coordinate system, and sets the three newly obtained directions as the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the HMD 120.

In a certain situation, when the user 5 wearing the HMD 120 stands upright and looks straight ahead, the processor 210 sets a uvw field of view coordinate system parallel to the real coordinate system in the HMD 120. In this case, the horizontal direction (x-axis), vertical direction (y-axis), and front-to-back direction (z-axis) in the real coordinate system coincide with the pitch axis (u-axis), yaw axis (v-axis), and roll axis (w-axis) of the uvw field of view coordinate system in the HMD 120.

After the uvw field of view coordinate system is set in the HMD 120, the HMD sensor 410 can detect the tilt of the HMD 120 in the set uvw field of view coordinate system based on the movement of the HMD 120. In this case, the HMD sensor 410 detects the pitch angle (θu), yaw angle (θv), and roll angle (θw) of the HMD 120 in the uvw field of view coordinate system as the tilt of the HMD 120. The pitch angle (θu) represents the tilt angle of the HMD 120 around the pitch axis in the uvw field of view coordinate system. The yaw angle (θv) represents the tilt angle of the HMD 120 around the yaw axis in the uvw field of view coordinate system. The roll angle (θw) represents the tilt angle of the HMD 120 around the roll axis in the uvw field of view coordinate system.

The HMD sensor 410 sets the uvw field of view coordinate system of the HMD 120 after the HMD 120 moves on the HMD 120 based on the detected tilt of the HMD 120. The relationship between the HMD 120 and the uvw field of view coordinate system of the HMD 120 is always constant regardless of the position and tilt of the HMD 120. When the position and tilt of the HMD 120 change, the position and tilt of the uvw field of view coordinate system of the HMD 120 in the real coordinate system change in conjunction with the change in the position and tilt.

In one aspect, the HMD sensor 410 may identify the position of the HMD 120 in real space as a relative position with respect to the HMD sensor 410 based on the infrared light intensity and the relative positional relationship between multiple points (e.g., the distance between each point) acquired based on the output from the infrared sensor. The processor 210 may determine the origin of the uvw field of view coordinate system of the HMD 120 in real space (actual coordinate system) based on the identified relative position.

[Virtual space]
The virtual space will be further described with reference to FIG. 4. FIG. 4 is a diagram conceptually showing one mode of expressing the virtual space 11 according to an embodiment. The virtual space 11 has a spherical structure covering the entire 360-degree direction of the center 12. In FIG. 4, in order to avoid complicating the description, the upper half of the celestial sphere in the virtual space 11 is illustrated. Each mesh is defined in the virtual space 11. The position of each mesh is defined in advance as a coordinate value in the XYZ coordinate system, which is a global coordinate system defined in the virtual space 11. The computer 200 associates each partial image constituting the panoramic image 13 (still image, video, etc.) that can be deployed in the virtual space 11 with each corresponding mesh in the virtual space 11.

In a certain aspect, an XYZ coordinate system is defined in the virtual space 11 with the center 12 as the origin. The XYZ coordinate system is, for example, parallel to the real coordinate system. The horizontal direction, vertical direction (up-down direction), and front-to-back direction in the XYZ coordinate system are defined as the X-axis, Y-axis, and Z-axis, respectively. Therefore, the X-axis (horizontal direction) of the XYZ coordinate system is parallel to the x-axis of the real coordinate system, the Y-axis (vertical direction) of the XYZ coordinate system is parallel to the y-axis of the real coordinate system, and the Z-axis (front-to-back direction) of the XYZ coordinate system is parallel to the z-axis of the real coordinate system.

When the HMD 120 is started up, i.e., in the initial state of the HMD 120, the virtual camera 14 is placed at the center 12 of the virtual space 11. At a certain stage, the processor 210 displays an image captured by the virtual camera 14 on the monitor 130 of the HMD 120. The virtual camera 14 moves in the virtual space 11 in a similar manner in conjunction with the movement of the HMD 120 in the real space. This allows changes in the position and tilt of the HMD 120 in the real space to be reproduced in the virtual space 11 in a similar manner.

A uvw field of view coordinate system is defined for the virtual camera 14, as in the case of the HMD 120. The uvw field of view coordinate system of the virtual camera 14 in the virtual space 11 is defined so as to be linked to the uvw field of view coordinate system of the HMD 120 in real space (actual coordinate system). Therefore, when the tilt of the HMD 120 changes, the tilt of the virtual camera 14 also changes accordingly. The virtual camera 14 can also move in the virtual space 11 in conjunction with the movement in real space of the user 5 wearing the HMD 120.

The processor 210 of the computer 200 determines the field of view 15 in the virtual space 11 based on the position and inclination (reference line of sight 16) of the virtual camera 14. The field of view 15 corresponds to the area of the virtual space 11 that is viewed by the user 5 wearing the HMD 120. In other words, the position of the virtual camera 14 can be said to be the viewpoint of the user 5 in the virtual space 11.

The line of sight of user 5 detected by gaze sensor 140 is the direction in the viewpoint coordinate system when user 5 views an object. The uvw field of view coordinate system of HMD 120 is equal to the viewpoint coordinate system when user 5 views monitor 130. The uvw field of view coordinate system of virtual camera 14 is linked to the uvw field of view coordinate system of HMD 120. Therefore, in a certain aspect, the HMD system 100 can consider the line of sight of user 5 detected by gaze sensor 140 to be the line of sight of user 5 in the uvw field of view coordinate system of virtual camera 14.

[User's gaze]
Determining the line of sight of the user 5 will be described with reference to Fig. 5. Fig. 5 is a top view of the head of a user 5 wearing an HMD 120 according to an embodiment.

In one aspect, the gaze sensor 140 detects the gaze of each of the right and left eyes of the user 5. In one aspect, when the user 5 is looking at something close, the gaze sensor 140 detects the gazes R1 and L1. In another aspect, when the user 5 is looking at something far away, the gaze sensor 140 detects the gazes R2 and L2. In this case, the angle that the gazes R2 and L2 make with respect to the roll axis w is smaller than the angle that the gazes R1 and L1 make with respect to the roll axis w. The gaze sensor 140 transmits the detection result to the computer 200.

When the computer 200 receives the detection values of the gazes R1 and L1 from the gaze sensor 140 as the gaze detection result, the computer 200 identifies the gaze point N1, which is the intersection of the gazes R1 and L1, based on the detection values. On the other hand, when the computer 200 receives the detection values of the gazes R2 and L2 from the gaze sensor 140, the computer 200 identifies the intersection of the gazes R2 and L2 as the gaze point. The computer 200 identifies the gaze N0 of the user 5 based on the position of the identified gaze point N1. For example, the computer 200 detects the direction of the line that passes through the gaze point N1 and the midpoint of the line connecting the right eye R and the left eye L of the user 5 as the gaze N0. The gaze N0 is the direction in which the user 5 is actually looking with both eyes. The gaze N0 corresponds to the direction in which the user 5 is actually looking with respect to the field of view 15.

In another aspect, the HMD system 100 may be equipped with a television broadcast receiving tuner. With such a configuration, the HMD system 100 can display television programs in the virtual space 11.

In yet another aspect, the HMD system 100 may be equipped with a communication circuit for connecting to the Internet or a calling function for connecting to a telephone line.

[View Area]
The field of view 15 will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a diagram showing a YZ cross section of the field of view 15 in the virtual space 11 as viewed from the X direction. Fig. 7 is a diagram showing an XZ cross section of the field of view 15 in the virtual space 11 as viewed from the Y direction.

As shown in FIG. 6, the field of view 15 in the YZ cross section includes an area 18. The area 18 is defined by the position of the virtual camera 14, the reference line of sight 16, and the YZ cross section of the virtual space 11. The processor 210 defines the range including the polar angle α centered on the reference line of sight 16 in the virtual space as the area 18.

As shown in FIG. 7, the field of view 15 in the XZ cross section includes area 19. Area 19 is defined by the position of virtual camera 14, reference line of sight 16, and the XZ cross section of virtual space 11. Processor 210 defines a range including azimuth angle β centered on reference line of sight 16 in virtual space 11 as area 19. Polar angles α and β are determined according to the position of virtual camera 14 and the inclination (direction) of virtual camera 14.

In one aspect, the HMD system 100 provides the user 5 with a field of view in the virtual space 11 by displaying a field of view image 17 on the monitor 130 based on a signal from the computer 200. The field of view image 17 is an image that corresponds to a portion of the panoramic image 13 that corresponds to the field of view area 15. When the user 5 moves the HMD 120 worn on the head, the virtual camera 14 also moves in conjunction with the movement. As a result, the position of the field of view area 15 in the virtual space 11 changes. As a result, the field of view image 17 displayed on the monitor 130 is updated to an image of the panoramic image 13 that is superimposed on the field of view area 15 in the direction in which the user 5 is facing in the virtual space 11. The user 5 can view the desired direction in the virtual space 11.

In this way, the inclination of the virtual camera 14 corresponds to the line of sight (reference line of sight 16) of the user 5 in the virtual space 11, and the position at which the virtual camera 14 is positioned corresponds to the viewpoint of the user 5 in the virtual space 11. Therefore, by changing the position or inclination of the virtual camera 14, the image displayed on the monitor 130 is updated and the field of view of the user 5 is moved.

While wearing the HMD 120, the user 5 can only view the panoramic image 13 displayed in the virtual space 11, without being able to see the real world. Therefore, the HMD system 100 can give the user 5 a high sense of immersion in the virtual space 11.

In one aspect, the processor 210 may move the virtual camera 14 in the virtual space 11 in conjunction with the movement in real space of the user 5 wearing the HMD 120. In this case, the processor 210 identifies the image area (field of view area 15) to be projected onto the monitor 130 of the HMD 120 based on the position and inclination of the virtual camera 14 in the virtual space 11.

In one aspect, the virtual camera 14 may include two virtual cameras, i.e., a virtual camera for providing an image for the right eye and a virtual camera for providing an image for the left eye. An appropriate parallax is set for the two virtual cameras so that the user 5 can recognize the three-dimensional virtual space 11. In another aspect, the virtual camera 14 may be realized by a single virtual camera. In this case, an image for the right eye and an image for the left eye may be generated from an image obtained by the single virtual camera. In this embodiment, the technical idea of the present disclosure is illustrated by assuming that the virtual camera 14 includes two virtual cameras and is configured so that the roll axis (w) generated by combining the roll axes of the two virtual cameras is adapted to the roll axis (w) of the HMD 120.

[controller]
An example of the controller 300 will be described with reference to Fig. 8. Fig. 8 is a diagram showing a schematic configuration of the controller 300 according to an embodiment.

As shown in FIG. 8, in one aspect, the controller 300 may include a right controller 300R and a left controller (not shown). The right controller 300R is operated by the right hand of the user 5. The left controller is operated by the left hand of the user 5. In one aspect, the right controller 300R and the left controller are configured symmetrically as separate devices. Thus, the user 5 can freely move both the right hand holding the right controller 300R and the left hand holding the left controller. In another aspect, the controller 300 may be an integrated controller that accepts operation from both hands. The right controller 300R will be described below.

The right controller 300R includes a grip 310, a frame 320, and a top surface 330. The grip 310 is configured to be held by the right hand of the user 5. For example, the grip 310 can be held by the palm and three fingers (middle finger, ring finger, and little finger) of the right hand of the user 5.

Grip 310 includes buttons 340, 350 and a motion sensor 420. Button 340 is located on the side of grip 310 and is operated by the middle finger of the right hand. Button 350 is located on the front of grip 310 and is operated by the index finger of the right hand. In one aspect, buttons 340, 350 are configured as trigger-type buttons. Motion sensor 420 is built into the housing of grip 310. If the movements of user 5 can be detected from around user 5 by a camera or other device, grip 310 does not need to include motion sensor 420.

The frame 320 includes a number of infrared LEDs 360 arranged along its circumference. During execution of a program that uses the controller 300, the infrared LEDs 360 emit infrared light in accordance with the progress of the program. The infrared light emitted from the infrared LEDs 360 can be used to detect the positions and attitudes (tilt, direction) of the right controller 300R and the left controller. In the example shown in FIG. 8, the infrared LEDs 360 are arranged in two rows, but the number of rows is not limited to that shown in FIG. 8. An arrangement in one row or three or more rows may be used.

The top surface 330 includes buttons 370, 380 and an analog stick 390. The buttons 370, 380 are configured as push buttons. The buttons 370, 380 are operated by the thumb of the right hand of the user 5. In a certain situation, the analog stick 390 is operated in any direction within 360 degrees from the initial position (neutral position). Such operations include, for example, operations for moving an object placed in the virtual space 11.

In one aspect, the right controller 300R and the left controller include a battery for driving the infrared LED 360 and other components. The battery may be, but is not limited to, a rechargeable battery, a button type, a dry cell battery, or the like. In another aspect, the right controller 300R and the left controller may be connected, for example, to a USB interface of the computer 200. In this case, the right controller 300R and the left controller do not require a battery.

As shown in states (A) and (B) of FIG. 8, for example, the yaw, roll, and pitch directions are defined for the right hand of user 5. When user 5 extends his thumb and index finger, the direction in which the thumb extends is defined as the yaw direction, the direction in which the index finger extends is defined as the roll direction, and the direction perpendicular to the plane defined by the axis of the yaw direction and the axis of the roll direction is defined as the pitch direction.

[Server hardware configuration]
Server 600 according to the present embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of a hardware configuration of server 600 according to an embodiment. Server 600 includes, as main components, a processor 610, a memory 620, a storage 630, an input/output interface 640, and a communication interface 650. Each component is connected to a bus 660.

The processor 610 executes a series of instructions contained in a program stored in the memory 620 or the storage 630 based on a signal provided to the server 600 or based on the satisfaction of a predetermined condition. In one aspect, the processor 610 is realized as a CPU, a GPU, an MPU, an FPGA, or other device.

Memory 620 temporarily stores programs and data. Programs are loaded, for example, from storage 630. Data includes data input to server 600 and data generated by processor 610. In one aspect, memory 620 is realized as RAM or other volatile memory.

Storage 630 permanently holds programs and data. Storage 630 is realized, for example, as a ROM, a hard disk drive, a flash memory, or other non-volatile storage device. Programs stored in storage 630 may include a program for providing a virtual space in HMD system 100, a simulation program, a game program, a user authentication program, and a program for realizing communication with computer 200. Data stored in storage 630 may include data and objects for defining the virtual space, etc.

In another aspect, the storage 630 may be realized as a removable storage device such as a memory card. In yet another aspect, instead of the storage 630 built into the server 600, a configuration may be used in which programs and data stored in an external storage device are used. With such a configuration, for example, in a situation where multiple HMD systems 100 are used, such as an amusement facility, it becomes possible to update programs and data collectively.

The input/output interface 640 communicates signals with input/output devices. In some aspects, the input/output interface 640 is realized using a USB, DVI, HDMI, or other terminal. The input/output interface 640 is not limited to the above.

The communication interface 650 is connected to the network 2 and communicates with the computer 200 connected to the network 2. In one aspect, the communication interface 650 is realized as, for example, a LAN or other wired communication interface, or a wireless communication interface such as WiFi, Bluetooth, NFC, or other wireless communication interface. The communication interface 650 is not limited to the above.

In one aspect, the processor 610 accesses the storage 630, loads one or more programs stored in the storage 630 into the memory 620, and executes a series of instructions contained in the programs. The one or more programs may include an operating system of the server 600, an application program for providing a virtual space, game software executable in the virtual space, and the like. The processor 610 may send a signal for providing the virtual space to the computer 200 via the input/output interface 640.

[HMD Control Device]
The control device of the HMD 120 will be described with reference to Fig. 10. In an embodiment, the control device is realized by a computer 200 having a known configuration. Fig. 10 is a block diagram showing the computer 200 according to an embodiment as a modular configuration.

As shown in FIG. 10, the computer 200 includes a control module 510, a rendering module 520, a memory module 530, and a communication control module 540. In one aspect, the control module 510 and the rendering module 520 are realized by the processor 210. In another aspect, multiple processors 210 may operate as the control module 510 and the rendering module 520. The memory module 530 is realized by the memory 220 or the storage 230. The communication control module 540 is realized by the communication interface 250.

The control module 510 controls the virtual space 11 provided to the user 5. The control module 510 defines the virtual space 11 in the HMD system 100 using virtual space data representing the virtual space 11. The virtual space data is stored in, for example, the memory module 530. The control module 510 may generate the virtual space data or acquire the virtual space data from the server 600 or the like.

The control module 510 places objects in the virtual space 11 using object data representing the objects. The object data is stored in the memory module 530, for example. The control module 510 may generate the object data or obtain the object data from the server 600 or the like. The objects may include, for example, an avatar object that is an avatar of the user 5, a character object, an operational object such as a virtual hand that is operated by the controller 300, landscapes including forests, mountains, and the like that are placed according to the progress of the game story, cityscapes, animals, and the like.

The control module 510 places in the virtual space 11 an avatar object of the user 5 of another computer 200 connected via the network 2. In one aspect, the control module 510 places in the virtual space 11 an avatar object of the user 5. In one aspect, the control module 510 places in the virtual space 11 an avatar object that resembles the user 5 based on an image including the user 5. In another aspect, the control module 510 places in the virtual space 11 an avatar object that has been selected by the user 5 from among multiple types of avatar objects (e.g., objects that resemble animals or deformed human objects).

The control module 510 determines the inclination of the HMD 120 based on the output of the HMD sensor 410. In another aspect, the control module 510 determines the inclination of the HMD 120 based on the output of the sensor 190 functioning as a motion sensor. The control module 510 detects the organs that make up the face of the user 5 (e.g., mouth, eyes, eyebrows) from the images of the face of the user 5 generated by the first camera 150 and the second camera 160. The control module 510 detects the movement (shape) of each detected organ.

The control module 510 detects the line of sight of the user 5 in the virtual space 11 based on a signal from the gaze sensor 140. The control module 510 detects the viewpoint position (coordinate value in the XYZ coordinate system) where the detected line of sight of the user 5 intersects with the celestial sphere of the virtual space 11. More specifically, the control module 510 detects the viewpoint position based on the line of sight of the user 5 defined in the uvw coordinate system and the position and inclination of the virtual camera 14. The control module 510 transmits the detected viewpoint position to the server 600. In another aspect, the control module 510 may be configured to transmit line of sight information representing the line of sight of the user 5 to the server 600. In such a case, the server 600 may calculate the viewpoint position based on the line of sight information received.

The control module 510 reflects the movement of the HMD 120 detected by the HMD sensor 410 in the avatar object. For example, the control module 510 detects that the HMD 120 has been tilted and tilts and positions the avatar object. The control module 510 reflects the detected movement of the facial organs in the face of the avatar object placed in the virtual space 11. The control module 510 receives gaze information of the other user 5 from the server 600 and reflects it in the gaze of the avatar object of the other user 5. In a certain aspect, the control module 510 reflects the movement of the controller 300 in the avatar object or the operation object. In this case, the controller 300 is equipped with a motion sensor, an acceleration sensor, or multiple light-emitting elements (e.g., infrared LEDs) for detecting the movement of the controller 300.

The control module 510 places an operation object in the virtual space 11 for receiving an operation by the user 5 in the virtual space 11. The user 5 operates, for example, an object placed in the virtual space 11 by operating the operation object. In one aspect, the operation object may include, for example, a hand object that is a virtual hand corresponding to the hand of the user 5. In one aspect, the control module 510 moves the hand object in the virtual space 11 in conjunction with the movement of the hand of the user 5 in the real space based on the output of the motion sensor 420. In one aspect, the operation object may correspond to the hand portion of an avatar object.

The control module 510 detects a collision when each of the objects placed in the virtual space 11 collides with another object. The control module 510 can detect, for example, the timing when a collision area of one object comes into contact with a collision area of another object, and performs a predetermined process when this detection is made. The control module 510 can detect the timing when objects are no longer in contact with each other, and performs a predetermined process when this detection is made. The control module 510 can detect when objects are in contact with each other. For example, when a control object comes into contact with another object, the control module 510 detects that the control object has come into contact with the other object, and performs a predetermined process.

In one aspect, the control module 510 controls the image display on the monitor 130 of the HMD 120. For example, the control module 510 places a virtual camera 14 in the virtual space 11. The control module 510 controls the position of the virtual camera 14 in the virtual space 11 and the tilt (direction) of the virtual camera 14. The control module 510 determines the field of view 15 according to the tilt of the head of the user 5 wearing the HMD 120 and the position of the virtual camera 14. The rendering module 520 generates a field of view image 17 to be displayed on the monitor 130 based on the determined field of view 15. The field of view image 17 generated by the rendering module 520 is output to the HMD 120 by the communication control module 540.

When the control module 510 detects speech from the user 5 using the microphone 170 from the HMD 120, it identifies the computer 200 to which voice data corresponding to the speech is to be sent. The voice data is sent to the computer 200 identified by the control module 510. When the control module 510 receives voice data from another user's computer 200 via the network 2, it outputs the voice (speech) corresponding to the voice data from the speaker 180.

The memory module 530 holds data that is used by the computer 200 to provide the virtual space 11 to the user 5. In one aspect, the memory module 530 holds space information, object information, and user information.

The spatial information holds one or more templates defined to provide the virtual space 11.

The object information includes a plurality of panoramic images 13 that constitute the virtual space 11, and object data for placing objects in the virtual space 11. The panoramic images 13 may include still images and moving images. The panoramic images 13 may include images of unreal space and images of real space. Images of unreal space include, for example, images generated by computer graphics.

The user information holds a user ID that identifies the user 5. The user ID may be, for example, an IP (Internet Protocol) address or a MAC (Media Access Control) address that is set in the computer 200 used by the user. In another aspect, the user ID may be set by the user. The user information includes a program for causing the computer 200 to function as a control device for the HMD system 100, etc.

The data and programs stored in the memory module 530 are input by the user 5 of the HMD 120. Alternatively, the processor 210 downloads the programs or data from a computer (e.g., server 600) operated by the operator providing the content, and stores the downloaded programs or data in the memory module 530.

The communication control module 540 can communicate with the server 600 and other information and communication devices via the network 2.

In one aspect, the control module 510 and the rendering module 520 can be realized using, for example, Unity (registered trademark) provided by Unity Technologies, Inc. In another aspect, the control module 510 and the rendering module 520 can be realized as a combination of circuit elements that realize each process.

Processing in computer 200 is realized by hardware and software executed by processor 210. Such software may be stored in advance on a hard disk or other memory module 530. The software may be stored on a CD-ROM or other computer-readable non-volatile data recording medium and distributed as a program product. Alternatively, the software may be provided as a downloadable program product by an information provider connected to the Internet or other network. Such software is read from the data recording medium by an optical disk drive or other data reading device, or downloaded from server 600 or other computer via communication control module 540, and then temporarily stored in a storage module. The software is read from the storage module by processor 210 and stored in RAM in the form of an executable program. Processor 210 executes the program.

[Control structure of HMD system]
The control structure of the HMD set 110 will be described with reference to Fig. 11. Fig. 11 is a sequence chart showing a part of the processing executed in the HMD set 110 according to an embodiment.

As shown in FIG. 11, in step S1110, the processor 210 of the computer 200, as the control module 510, identifies virtual space data and defines the virtual space 11.

In step S1120, the processor 210 initializes the virtual camera 14. For example, the processor 210 places the virtual camera 14 in a work area of the memory at a predefined center 12 in the virtual space 11, and directs the line of sight of the virtual camera 14 in the direction in which the user 5 is facing.

In step S1130, the processor 210, as the rendering module 520, generates field of view image data for displaying an initial field of view image. The generated field of view image data is output to the HMD 120 by the communication control module 540.

In step S1132, the monitor 130 of the HMD 120 displays a field of view image based on the field of view image data received from the computer 200. The user 5 wearing the HMD 120 can recognize the virtual space 11 by visually recognizing the field of view image.

In step S1134, the HMD sensor 410 detects the position and inclination of the HMD 120 based on the multiple infrared lights emitted from the HMD 120. The detection results are output to the computer 200 as motion detection data.

In step S1140, the processor 210 determines the field of view direction of the user 5 wearing the HMD 120 based on the position and tilt contained in the motion detection data of the HMD 120.

In step S1150, the processor 210 executes the application program and places objects in the virtual space 11 based on the instructions contained in the application program.

In step S1160, the controller 300 detects an operation by the user 5 based on a signal output from the motion sensor 420, and outputs detection data representing the detected operation to the computer 200. In another aspect, the operation of the controller 300 by the user 5 may be detected based on an image from a camera arranged around the user 5.

In step S1170, the processor 210 detects the operation of the controller 300 by the user 5 based on the detection data acquired from the controller 300.

In step S1180, the processor 210 generates field of view image data based on the operation of the controller 300 by the user 5. The generated field of view image data is output to the HMD 120 by the communication control module 540.

In step S1190, the HMD 120 updates the field of view image based on the received field of view image data and displays the updated field of view image on the monitor 130.

[Avatar Object]
Avatar objects according to the present embodiment will be described with reference to Figs. 12(A) and (B). Hereinafter, avatar objects of each user 5 of the HMD sets 110A and 110B will be described. Hereinafter, the user of the HMD set 110A will be referred to as user 5A, the user of the HMD set 110B as user 5B, the user of the HMD set 110C as user 5C, and the user of the HMD set 110D as user 5D. A is added to the reference symbol of each component related to the HMD set 110A, B is added to the reference symbol of each component related to the HMD set 110B, C is added to the reference symbol of each component related to the HMD set 110C, and D is added to the reference symbol of each component related to the HMD set 110D. For example, the HMD 120A is included in the HMD set 110A.

Figure 12 (A) is a schematic diagram showing a situation in which each HMD 120 provides a virtual space 11 to a user 5 in a network 2. Computers 200A to 200D provide virtual spaces 11A to 11D to users 5A to 5D, respectively, via HMDs 120A to 120D. In the example shown in Figure 12 (A), virtual space 11A and virtual space 11B are configured with the same data. In other words, computer 200A and computer 200B share the same virtual space. In virtual space 11A and virtual space 11B, there exist an avatar object 6A of user 5A and an avatar object 6B of user 5B. Although avatar object 6A in virtual space 11A and avatar object 6B in virtual space 11B each wear an HMD 120, this is for ease of explanation, and in reality, these objects do not wear an HMD 120.

In one aspect, the processor 210A may position a virtual camera 14A that captures a field of view image 17A of the user 5A at the eye position of the avatar object 6A.

Figure 12 (B) is a diagram showing a field of view image 17A of user 5A in Figure 12 (A). Field of view image 17A is an image displayed on monitor 130A of HMD 120A. This field of view image 17A is an image generated by virtual camera 14A. An avatar object 6B of user 5B is displayed in field of view image 17A. Although not specifically shown, avatar object 6A of user 5A is similarly displayed in the field of view image of user 5B.

In the state of FIG. 12(B), user 5A can communicate with user 5B through dialogue via virtual space 11A. More specifically, the voice of user 5A picked up by microphone 170A is transmitted to HMD 120B of user 5B via server 600 and output from speaker 180B provided on HMD 120B. The voice of user 5B is transmitted to HMD 120A of user 5A via server 600 and output from speaker 180A provided on HMD 120A.

The movements of user 5B (the movements of HMD 120B and controller 300B) are reflected by processor 210A in avatar object 6B placed in virtual space 11A. This allows user 5A to recognize the movements of user 5B through avatar object 6B.

Figure 13 is a sequence chart showing a part of the processing executed in the HMD system 100 according to the present embodiment. Although the HMD set 110D is not shown in Figure 13, the HMD set 110D operates in the same manner as the HMD sets 110A, 110B, and 110C. In the following description, A will be added to the reference number of each component related to the HMD set 110A, B will be added to the reference number of each component related to the HMD set 110B, C will be added to the reference number of each component related to the HMD set 110C, and D will be added to the reference number of each component related to the HMD set 110D.

In step S1310A, the processor 210A in the HMD set 110A acquires avatar information for determining the movement of the avatar object 6A in the virtual space 11A. The avatar information includes information about the avatar, such as, for example, movement information, face tracking data, and voice data. The movement information includes information indicating the temporal change in the position and tilt of the HMD 120A, and information indicating the hand movement of the user 5A detected by the motion sensor 420A, etc. The face tracking data includes data specifying the position and size of each part of the face of the user 5A. The face tracking data includes data indicating the movement of each organ constituting the face of the user 5A and gaze data. The voice data includes data indicating the voice of the user 5A acquired by the microphone 170A of the HMD 120A. The avatar information may include information specifying the avatar object 6A, or the user 5A associated with the avatar object 6A, and information specifying the virtual space 11A in which the avatar object 6A exists. The information specifying the avatar object 6A or the user 5A includes a user ID. The information for identifying the virtual space 11A in which the avatar object 6A exists includes a room ID. The processor 210A transmits the avatar information acquired as described above to the server 600 via the network 2.

In step S1310B, the processor 210B in the HMD set 110B acquires avatar information for determining the movement of the avatar object 6B in the virtual space 11B, similar to the processing in step S1310A, and transmits it to the server 600. Similarly, in step S1310C, the processor 210C in the HMD set 110C acquires avatar information for determining the movement of the avatar object 6C in the virtual space 11C, and transmits it to the server 600.

In step S1320, the server 600 temporarily stores the player information received from each of the HMD sets 110A, 110B, and 110C. The server 600 integrates the avatar information of all users (in this example, users 5A to 5C) associated with the common virtual space 11 based on the user ID and room ID, etc., included in each piece of avatar information. The server 600 then transmits the integrated avatar information to all users associated with the virtual space 11 at a predetermined timing. This executes a synchronization process. This synchronization process allows the HMD sets 110A, 110B, and 110C to share each other's avatar information at approximately the same timing.

Next, based on the avatar information transmitted from the server 600 to each of the HMD sets 110A to 110C, each of the HMD sets 110A to 110C executes the process of steps S1330A to S1330C. The process of step S1330A corresponds to the process of step S1180 in FIG. 11.

In step S1330A, the processor 210A in the HMD set 110A updates the information of the avatar objects 6B and avatar objects 6C of the other users 5B and 5C in the virtual space 11A. Specifically, the processor 210A updates the position, orientation, etc. of the avatar object 6B in the virtual space 11 based on the movement information included in the avatar information transmitted from the HMD set 110B. For example, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6B included in the object information stored in the memory module 530. Similarly, the processor 210A updates the information (position, orientation, etc.) of the avatar object 6C in the virtual space 11 based on the movement information included in the avatar information transmitted from the HMD set 110C.

In step S1330B, the processor 210B in the HMD set 110B updates the information of the avatar objects 6A and 6C of the users 5A and 5C in the virtual space 11B, similar to the processing in step S1330A. Similarly, in step S1330C, the processor 210C in the HMD set 110C updates the information of the avatar objects 6A and 6B of the users 5A and 5B in the virtual space 11C.

[Detailed configuration of modules of computer 200]
The details of the module configuration of the computer 200 will be described with reference to Fig. 14. Fig. 14 is a block diagram showing a detailed configuration of the modules of the computer 200 according to an embodiment. As shown in Fig. 14, the control module 510 includes a virtual object generation module 1421, a virtual camera control module 1422, an operation object control module 1423, an avatar object control module 1424, a motion detection module 1425, a collision detection module 1426, a virtual object control module 1427, and a learning module 1428. The memory module 530 stores a learned model 1429.

The virtual object generation module 1421 generates various virtual objects in the virtual space 11. In one aspect, the virtual objects may include, for example, landscapes including forests, mountains, and other objects, animals, etc., that are placed according to the progress of the game's story. In another aspect, the virtual objects may include avatar objects, control objects, stage objects, and UI (User Interface) objects.

The virtual camera control module 1422 controls the behavior of the virtual camera 14 in the virtual space 11. The virtual camera control module 1422 controls, for example, the placement position of the virtual camera 14 in the virtual space 11 and the orientation (tilt) of the virtual camera 14.

The operation object control module 1423 controls an operation object for receiving an operation from the user 5 in the virtual space 11. The user 5 operates the operation object to operate, for example, a virtual object placed in the virtual space 11. In one aspect, the operation object may include, for example, a hand object (virtual hand) corresponding to the hand of the user 5 wearing the HMD 120. In one aspect, the operation object may correspond to the hand portion of an avatar object described below.

The avatar object control module 1424 reflects the movement of the HMD 120 detected by the HMD sensor 410 in the avatar object. For example, the avatar object control module 1424 detects that the HMD 120 has been tilted and generates data for tilting and positioning the avatar object. In a certain aspect, the avatar object control module 1424 reflects the movement of the controller 300 in the avatar object. In this case, the controller 300 includes a motion sensor, an acceleration sensor, or a plurality of light-emitting elements (e.g., infrared LEDs) for detecting the movement of the controller 300. The avatar object control module 1424 reflects the movement of the facial organs detected by the movement detection module 1425 in the face of the avatar object placed in the virtual space 11. In other words, the avatar object control module 1424 reflects the facial movement of the user 5 in the avatar object.

The motion detection module 1425 detects the motion of the user 5. For example, the motion detection module 1425 detects the motion of the hand of the user 5 in response to the output of the controller 300. For example, the motion detection module 1425 detects the motion of the body of the user 5 in response to the output of a motion sensor attached to the body of the user 5. The motion detection module 1425 can also detect the movement of the facial organs of the user 5.

The collision detection module 1426 detects a collision when each of the virtual objects arranged in the virtual space 11 collides with another virtual object. The collision detection module 1426 can detect, for example, the timing when a certain virtual object comes into contact with another virtual object. The collision detection module 1426 can detect the timing when a certain virtual object and another virtual object leave a state of contact. The collision detection module 1426 can also detect a state in which a certain virtual object and another virtual object are in contact. For example, when a manipulation object and another virtual object come into contact with each other, the collision detection module 1426 detects that the manipulation object and the other object have come into contact with each other. The collision detection module 1426 executes a predetermined process based on these detection results.

The virtual object control module 1427 controls the behavior of virtual objects, excluding avatar objects, in the virtual space 11. As one example, the virtual object control module 1427 transforms a virtual object. As another example, the virtual object control module 1427 changes the placement position of a virtual object. As another example, the virtual object control module 1427 moves a virtual object.

The learning module 1428 performs machine learning on past actions (first actions) performed by the avatar object 6 in the virtual space 2611, thereby generating a learned model 1429 in which the action has been machine learned.

[Configuration of distribution system]
15 is a diagram showing an outline of the configuration of a distribution system 1500 according to the present embodiment. The distribution system 1500 includes a server 600, an HMD set 110B, user terminals 800A, 800C, and 800D, and a network 2. The HMD set 110B and each of the user terminals 800A, 800C, and 800D are configured to be able to communicate with the server 600 via the network 2. Hereinafter, the user terminals 800A, 800C, and 800D are also collectively referred to as user terminals 800. The number of user terminals 800 constituting the distribution system 1500 is not limited to three, and may be two or less, or four or more.

The user terminal 800 is a portable terminal device that can be carried by the user 5. The user terminal 800 is realized, for example, as a smartphone, a tablet terminal, or a notebook computer. Hereinafter, the user of the user terminal 800A will be referred to as user 5A, the user of the HMD set 110B as user 5B, the user of the user terminal 800C as user 5C, and the user of the user terminal 800D as user 5D. A will be added to the reference symbol of each component related to the user terminal 800A, B will be added to the reference symbol of each component related to the HMD set 110B, C will be added to the reference symbol of each component related to the user terminal 800C, and D will be added to the reference symbol of each component related to the user terminal 800D.

The distribution system 1500 is a system for streaming a program in which an avatar object 6B associated with a user 5B performs in a virtual space from an HMD set 110B to each user terminal 800. The user 5B advances the program of the avatar object 6B by controlling the avatar object 6B in the HMD set 110B. The user 5A watches the distributed program of the avatar object 6B through a user terminal 800A. The user 5C watches the distributed program of the avatar object 6B through a user terminal 800C. The user 5D watches the distributed program of the avatar object 6B through a user terminal 800D.

[Hardware configuration of user terminal]
16 is a block diagram showing an example of a hardware configuration of a user terminal 800 according to an embodiment. The user terminal 800 includes, as main components, a processor 710, a memory 720, a storage 730, an input/output interface 740, a communication interface 750, a touch screen 770, and a speaker 780. Each component is connected to a bus 760.

The processor 710 executes a series of instructions contained in a program stored in the memory 720 or storage 730 based on a signal provided to the user terminal 800 or based on the satisfaction of a predetermined condition. In one aspect, the processor 710 is realized as a CPU, GPU, MPU, FPGA, or other device.

Memory 720 temporarily stores programs and data. Programs are loaded, for example, from storage 730. Data includes data input to user terminal 800 and data generated by processor 710. In one aspect, memory 720 is realized as RAM or other volatile memory.

Storage 730 permanently holds programs and data. Storage 730 is realized, for example, as a ROM, a hard disk drive, a flash memory, or other non-volatile storage device. Programs stored in storage 730 may include a program for providing a virtual space in the user terminal 800, a simulation program, a game program, a user authentication program, and a program for realizing communication with the server 600. Data stored in storage 730 may include data and objects for defining the virtual space. In another aspect, storage 730 may be realized as a removable storage device such as a memory card.

The input/output interface 740 communicates signals with input/output devices. In one aspect, the input/output interface 740 is realized using a USB, DVI, HDMI, or other terminal. The input/output interface 740 is not limited to the above.

The communication interface 750 is connected to the network 2 and communicates with the server 600 connected to the network 2. In one aspect, the communication interface 750 is realized as, for example, a LAN or other wired communication interface, or a wireless communication interface such as WiFi, Bluetooth, NFC, or other wireless communication interface. The communication interface 750 is not limited to the above.

The touch screen 770 is an electronic component that combines an input section and a display section, not shown. The input section is, for example, a touch-sensitive device, and is configured, for example, by a touchpad. The display section 152 is, for example, configured by a liquid crystal display or an organic EL (Electro-Luminescence) display. The input section has a function of detecting the position where a user operation (mainly a physical contact operation such as a touch operation, a slide operation, a swipe operation, or a tap operation) is input on the input surface, and transmitting information indicating the position as an input signal. The input section may include a touch sensing section, not shown. The touch sensing section may adopt any method, such as a capacitive type or a resistive film type.

The speaker 780 converts the audio signal into audio and outputs it to the user 5. In another aspect, the user terminal 800 may include an earphone instead of the speaker 780.

In one aspect, the processor 710 accesses the storage 730, loads one or more programs stored in the storage 730 into the memory 720, and executes a series of instructions contained in the programs. The one or more programs may include an operating system for the user terminal 800, an application program for providing a virtual space, game software executable in the virtual space, etc.

Although not shown, the user terminal 800 may have one or more sensors for identifying the holding orientation of the user terminal 800. The sensor may be, for example, an acceleration sensor or an angular velocity sensor. If the user terminal 800 has a sensor, the processor 710 may identify the holding orientation of the user terminal 800 from the output of the sensor and perform processing according to the holding orientation. For example, when the user terminal 800 is held vertically, the processor 710 may display a vertical screen in which a vertically long image is displayed on the touch screen 770. On the other hand, when the user terminal 800 is held horizontally, the processor 710 may display a horizontal screen in which a horizontally long image is displayed on the touch screen. In this way, the processor 710 may be able to switch between a vertical screen display and a horizontal screen display according to the holding orientation of the user terminal 800.

[Module configuration of user terminal]
Fig. 17 is a block diagram showing a detailed configuration of modules of a user terminal 800 according to an embodiment. As shown in Fig. 17, the user terminal 800A includes a control module 810, a rendering module 820, a memory module 830, and a communication control module 840. In one aspect, the control module 810 and the rendering module 820 are realized by the processor 710. In another aspect, a plurality of processors 710 may operate as the control module 810 and the rendering module 820. The memory module 830 is realized by the memory 720 or the storage 730. The communication control module 840 is realized by the communication interface 750.

The basic functions of the control module 810, the rendering module 820, the memory module 530, and the communication control module 840 are the same as those of the control module 510, the rendering module 520, the memory module 530, and the communication control module 540 provided in the computer 200. Therefore, detailed descriptions of these will not be repeated.

As shown in FIG. 17, the control module 810 includes a virtual object generation module 1721, a virtual viewpoint control module 1722, an avatar object control module 1723, and a learning module 1724. The memory module 830 stores the learned model 1725.

Since the virtual object generation module 1721 has at least the same functions as the virtual object generation module 1421 provided in the computer 200, a detailed description thereof will not be repeated. The virtual viewpoint control module 1722 controls the behavior of the virtual viewpoint in the virtual space 11. The virtual viewpoint has a function equivalent to that of the virtual camera 14. The virtual viewpoint control module 1722 controls, for example, the placement position of the virtual viewpoint in the virtual space 11 and the orientation (tilt) of the virtual viewpoint. Since the avatar object control module 1723 has at least the same functions as the avatar object control module 1424 provided in the computer 200, a detailed description thereof will not be repeated.

Since learning module 1724 has at least the same functions as learning module 1428 included in computer 200, a detailed description thereof will not be repeated. Since trained model 1725 is the same as trained model 1429 stored in memory module 530 of computer 200, a detailed description thereof will not be repeated.

[Virtual space for performers (broadcasters)]
18A and 18B are diagrams showing a virtual space 11B and a field of view image 1817B according to an embodiment. In FIG. 18A, at least an avatar object 6B, a virtual camera 14B, and a panel object 1832 are arranged in a virtual space 11B for providing a virtual experience to a user 5B. A user 5B (first user) wears an HMD 120B on his head. The user 5B holds a right controller 300RB with a right hand (first part) constituting a part of the right side of the user 5B's body, and holds a left controller 300LB with a left hand (second part) constituting a part of the left side of the user 5B's body.

The HMD 120B includes a sensor 190 that functions as a motion sensor. The right controller 300RB and the left controller 300LB include a motion sensor 420. The user 5B further includes motion sensors 1841 to 1843. The motion sensor 1841 is attached to the waist of the user 5B by a belt 1844. The motion sensor 1842 is attached to the top of the right foot of the user 5B. The motion sensor 1843 is attached to the top of the left foot of the user 5B. The motion sensors 1841 to 1843 are connected to the computer 200B by wire or wirelessly.

In one aspect, a motion sensor worn by user 5B detects the arrival time and angle of a signal (e.g., an infrared laser) emitted from a base station (not shown). Processor 210B of computer 200B (hereinafter simply processor 210B) detects the position of the motion sensor relative to the base station based on the detection results of the motion sensor. Processor 210B may further normalize the position of the motion sensor relative to the base station using a specified point (e.g., the position of sensor 190 worn on the head) as a reference.

Avatar object 6B includes a virtual right hand 1831RB and a virtual left hand 1831LB. The virtual right hand 1831RB is a type of control object, and can move in virtual space 11B in response to the movement of the right hand of user 5B. The virtual left hand 1831LB is a type of control object, and can move in virtual space 11B in response to the movement of the left hand of user 5B.

The virtual space 11B shown in FIG. 18(A) is constructed by playing back program content in the computer 200B. The user 5B moves his/her body to make the avatar object 6B perform a performance. The computer 200B detects the movement of the user 5B based on the output of various motion sensors attached to the user 5B. In the virtual space 11B, the avatar object 6B performs a performance that reflects the movement of the user 5B in the real space according to the identified movement of the user 5B. When the avatar object 6B performs a performance according to the movement of the user 5B in the virtual space 11B, the avatar object 6B also performs the same performance in the virtual spaces 11A, 11C, and 11D that are synchronized with the virtual space 11B. In this way, the user 5B plays the role of a distributor who distributes the program by the avatar object 6B to the users 5A, 5C, and 5D, respectively.

The panel object 1832 is a virtual object that displays comments entered by a user 5A, who is a viewer of the program of the avatar object 6B, during the distribution of the program. The panel object 1832 may be a semi-transparent object that is set to be transparent.

In FIG. 18(A), virtual camera 14B is placed on the head of avatar object 6B. Virtual camera 14B defines field of view 15B according to the position and orientation of virtual camera 14B. Virtual camera 14B generates field of view image 1817B corresponding to field of view 15B and displays it on HMD 120B as shown in FIG. 18(B). By viewing field of view image 1817B, user 5B views a part of the virtual space from the viewpoint of avatar object 6B. This allows user 5B to obtain a virtual experience as if user 5B were avatar object 6B. Field of view image 1817B includes panel object 1832. Therefore, user 5B can view comments from users 5A and the like on the program by avatar object 6B in real time during the distribution of the program.

[Virtual space for viewers]
FIG. 19 is a diagram showing a virtual space 11A and a display surface of a user terminal 800A according to an embodiment. In FIG. 19(A), at least an avatar object 6B is arranged in a virtual space 11A for providing a virtual experience to a user 5A (second user). The virtual space 11A is synchronized with the virtual space 11B shown in FIG. 18. The user 5A does not wear an HMD 120A on his head, but holds a user terminal 800A in his left hand. In the example of FIG. 19, the user 5A watches a program (distributed video) performed by an avatar object 6B while viewing the screen of the user terminal 800A.

The virtual space 11A shown in FIG. 19(A) is constructed by playing program content on a user terminal 800A. In the virtual space 11A, an avatar object 6B performs as a live performer based on the movements of a user 5B. The user 5B watches the performance of the avatar object 6B as a live viewer through the screen of the user terminal 800A. At the same time, a user 5C watches the performance of the avatar object 6B as a live viewer through the screen of the user terminal 800C. Also, a user 5D watches the performance of the avatar object 6B as a live viewer through the screen of the user terminal 800D. In this way, the program of the avatar object 6B is streamed to a plurality of different users 5 simultaneously.

19(A), a virtual viewpoint 1951 is set at the center 12A of the virtual space 11A. The virtual viewpoint 1951 has the same function as the virtual camera 14A. The virtual viewpoint 1951 defines a field of view 15A according to the position and orientation of the virtual viewpoint 1951. The processor 710A generates a field of view image 1917A corresponding to the field of view 15A and displays it on the touch screen 770A as shown in FIG. 19(B). The field of view image 1917A includes at least an avatar object 6B performing a performance. By viewing the field of view image 1917A, the user 5A views the avatar object 6B and a part of the virtual space 11A in which the avatar object 6B appears. This allows the user 5A to have a virtual experience as if the avatar object 6B were an actual broadcaster.

[Get Dimension Data]
Fig. 20 is a diagram for explaining a method of acquiring dimensional data. The dimensional data is data that represents the dimensions of the body of the user 5B. Fig. 20(A) shows a state in which the user 5B faces forward, spreads both arms horizontally, and stands up. Hereinafter, the state shown in Fig. 20(A) is also referred to as a first posture. Fig. 20(B) shows a state in which the user 5B faces forward, places both arms on the sides of the thighs, and stands up. Hereinafter, the state shown in Fig. 20(B) is also referred to as a second posture.

In one aspect, the processor 210B prompts the user 5B to assume the first and second postures. As one example, the processor 210B displays the character in the first and second postures on the monitor 130B and displays a message instructing the user 5B to assume similar postures. As another example, the processor 210B may output a sound from the speaker 180B instructing the user 5B to assume the first and second postures.

The processor 210B detects the positions of the head, waist, hands, and feet of the user 5B from the motion sensors attached to the user 5B. Hereinafter, the positions of the parts of the user 5B detected by each motion sensor are also referred to as "position information." The processor 210B acquires position information of the head, waist, hands, and feet of the user 5B based on the output of the motion sensors attached to the user 5B in each of the two postures (first posture and second posture). This position information can be acquired as positions in the real coordinate system (x, y, z) as shown in FIG. 21.

The processor 210B calculates the dimension data of the user 5B from the position information corresponding to the two postures. In one embodiment, the processor 210B calculates the height, shoulder width, arm length, foot length, and height from head to shoulder of the user 5B as the dimension data, as shown in FIG. 22. The processor 210B may calculate the distance between the hands in the second posture as the shoulder width. The processor 210B may calculate the arm length as half the value obtained by subtracting the shoulder width from the distance between the hands in the first posture. The processor 210B may calculate the distance from the foot height to the head height as the height. The processor 210B may calculate the distance from the foot height to the waist height as the foot length. The processor 210B may calculate the height from the hand height to the head in the first posture as the height from the head to the shoulders.

Figure 23 is a flowchart showing the process for acquiring dimensional data. In step S2310, the processor 210B places a virtual camera 14B in the virtual space 11B. The processor 210B further outputs a field of view image 17B corresponding to the shooting range of the virtual camera 14B to the monitor 130B.

In step S2320, the processor 210B instructs the user 5B to assume the first posture. For example, the processor 210B realizes the processing of step S2320 by placing an object with the instruction written on it in the virtual space 11B. In step S2330, the processor 210B acquires position information corresponding to the first posture.

In step S2340, the processor 210B instructs the user 5B to assume the second posture. In step S2350, the processor 210B acquires position information corresponding to the second posture.

In step S2360, the processor 210B calculates dimensional data of the user 5B from the position information corresponding to the first posture and the position information corresponding to the second posture. The processor 210B stores the dimensional data in the storage 230B.

As described above, user 5B can easily input his/her own dimensions into computer 200B simply by assuming two postures. In other aspects, user 5B may input his/her own dimensions into computer 200B using an input device such as a keyboard.

[Joint rotation direction]
In one embodiment, the processor 210B estimates the rotation direction of the joints of the user 5B based on the outputs (position information) of six motion sensors attached to the user 5B and the dimension data. As an example, the processor 210B estimates the position of the shoulders based on the position information of the head, the shoulder width, and the height from the head to the shoulders. The processor 210B estimates the position of the elbow from the position information of the shoulders and the position information of the hands. This estimation can be performed by a known application using inverse kinematics.

In one embodiment, the processor 210B acquires the inclination (rotation direction) of the joints of the neck (head), waist, both wrists, and both ankles of the user 5B from six motion sensors. In addition, the processor 210B estimates the rotation direction of the joints of both shoulders, both elbows, both hips (base of the feet), and both knees based on inverse kinematics. As shown in FIG. 22, the processor 210B acquires or estimates the rotation direction of each joint in the uvw field of view coordinate system.

When the rotation direction is calculated based on position information and dimensional data, the processor 210B cannot accurately estimate the position of the shoulders, etc., when the user 5B is not facing forward (i.e., when the head and waist are facing in different directions). Therefore, in another embodiment, the computer 200B may estimate the rotation direction of the joints by further taking into account the inclination of parts of the user 5B detected by the motion sensor. For example, the computer 200B estimates the position of the shoulders based on the position information of the head, the inclination of the head, the inclination of the waist, the shoulder width, and the height from the head to the shoulders. With this configuration, the computer 200B can improve the accuracy of the rotation direction of the joints.

[Program distribution flow]
FIG. 25 is a sequence chart showing a part of the processing executed in the distribution system 1500 according to an embodiment. FIG. 26 is a diagram showing a virtual space 2611B and a field of view image 2617B according to an embodiment. FIG. 27 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In this embodiment, at least the HMD set 110B, the user terminal 800A, and the server 600 execute a series of processing for progressing the program of the avatar object 6B. A part or all of the processing of the HMD set 110B may be executed by the server 600 or the user terminal 800A. In the following, a series of processing for distributing the program of the avatar object 6B progressing in the virtual space 2611B from the HMD set 110B to the user terminal 800A will be described. The program of the avatar object 6B is also distributed to the user terminals 800C and 800D based on a similar series of processing.

In step S2501, the processor 210B defines a virtual space 2611B as shown in FIG. 26(A). This process corresponds to the process of step S1110 in FIG. 11. Specifically, the processor 210B identifies virtual space data, thereby defining a virtual space 2611B represented by the virtual space data. The virtual space 2611B is a virtual space in which the avatar object 6B performs a program. In other words, the virtual space 2611B is a virtual space in which a performance by the avatar object 6B is performed.

In step S2502, the processor 210B, as the virtual object generation module 1421, generates an avatar object 6B (first avatar) associated with the user 5B (first user) and places it in the virtual space 2611B. In step S2503, the processor 210B, as the virtual object generation module 1421, generates a virtual camera 14B and places it in the virtual space 2611B. In FIG. 26(A), the virtual camera 14B is placed at the head of the avatar object 6B. In step S2504, the processor 210B, as the virtual object generation module 1421, generates a panel object 1832 and places it in the virtual space 2611B. The panel object 1832 is a virtual object on which comments entered by the user 5A, who is a viewer of the program, are displayed. In FIG. 26(A), the panel object 1832 is placed in front of the avatar object 6B at a certain distance from the avatar object 6B. At this point, no comments about the program have been entered by user 5A or the like, so no comments are displayed on panel object 1832.

In step S2505, the processor 210B generates avatar information of the avatar object 6B and transmits it to the user terminal 800A via the server 600. In step S2506, the processor 210B, as the virtual camera control module 1422, determines the position and inclination of the virtual camera 14B in the virtual space 2611B according to the movement of the HMD 120B. More specifically, the processor 210B controls the field of view area 15B, which is the field of view from the virtual camera 14B (virtual viewpoint) in the virtual space 2611B, according to the posture of the head of the user 5B and the position of the virtual camera 14B in the virtual space 2611B. This process corresponds to a part of the process of step S1140 in FIG. 11. Since the virtual camera 14B is placed in the same position as the avatar object 6B, the position of the virtual camera 14B is synonymous with the position of the avatar object 6B. Furthermore, the field of view from the virtual camera 14B is synonymous with the field of view from the avatar object 6B.

In step S2507, the processor 210B displays the field of view image 2617B on the monitor 130B. Specifically, the processor 210B defines the field of view image 2617B corresponding to the field of view area 15B based on the movement of the HMD 120B (i.e., the position and tilt of the virtual camera 14B) and the virtual space data defining the virtual space 2611B. Defining the field of view image 2617 is synonymous with generating the field of view image 2617B. The processor 210B further displays the field of view image 2617B on the HMD 120B by outputting the field of view image 2617B to the monitor 130B of the HMD 120B. This process corresponds to the processes of steps S1180 and S1190 in FIG. 11.

For example, the processor 210B displays a field of view image 2617B corresponding to the virtual space 2611B shown in FIG. 26(A) on the monitor 130B as shown in FIG. 26(B). The field of view image 2617B includes a panel object 1832 on which no comments are displayed. By viewing the field of view image 2617B, the user 5B views a part of the virtual space 2611B from the viewpoint of the avatar object 6B. This allows the user 5B to obtain a virtual experience as if the user 5B were the avatar object 6B. By viewing the panel object 1832, the user 5B recognizes that no comments have been received for the program of the avatar object 6B at the present time.

In step S2521, the processor 710A defines a virtual space 2611A for providing a virtual experience to the user 5A. This process is basically the same as the process for defining the virtual space 2611B in step S2501, so detailed description will not be repeated. In FIG. 26(A), the virtual space 2611A includes an avatar object 6B. The virtual space 2611A is a virtual space that is partially synchronized with the virtual space 2611B. In step S2522, the processor 710A sets a virtual viewpoint 1951 in the virtual space 2611B. In FIG. 27(A), the virtual viewpoint 1951 is set to the center of the virtual space 2611A. In step S2523, the processor 710A receives avatar information of the avatar object 6B transmitted from the computer 200B. In step S2524, processor 710A, as virtual object generation module 1721, generates avatar object 6B based on the received avatar information and places it within viewing area 15A of virtual viewpoint 1951 in virtual space 2611A. Processor 710A does not receive information about panel object 1832 from computer 200B, and does not place panel object 1832 in virtual space 2611A. In this way, panel object 1832 is a virtual object that is visible to user 5B but not to user 5A.

In step S2525, the processor 710A, as the virtual viewpoint control module 1722, determines the position and inclination of the virtual viewpoint 1951 in the virtual space 2611A based on the operation of the user 5A on the touch screen 770A. The processor 710A controls the field of view 15A, which is the field of view from the virtual viewpoint 1951 in the virtual space 2611A, according to, for example, the direction of the flick operation of the user 5A on the touch screen 770A and the position of the virtual viewpoint 1951 in the virtual space 2611A.

In step S2526, the processor 710A generates a field of view image 2717A corresponding to the field of view area 15A and displays it on the touch screen 770A. Specifically, the processor 710A defines the field of view image 2717A corresponding to the field of view area 15A based on the position and inclination of the virtual viewpoint 1951 and the virtual space data defining the virtual space 2611A. Defining the field of view image 2717A is synonymous with generating the field of view image 2717A. The processor 710A further displays the field of view image 2717A on the touch screen 770A of the user terminal 800A by outputting the field of view image 2717A to the touch screen 770A.

For example, the processor 710A displays a field of view image 2717A corresponding to the virtual space 2611A shown in FIG. 27(A) on the touch screen 770A as shown in FIG. 27(B). As a result, the processor 710A provides the field of view image 2717A including the avatar object 6B to the user 5A. The user 5A watches the program of the avatar object 6B by viewing the field of view image 2717A. In step S2527, the processor 710A displays a comment field 1952 below the field of view image 2717A on the touch screen 770A. At this point, the user 5A has not yet input a comment. Therefore, the comment of the user 5A is not displayed in the comment field 1952.

28 is a diagram showing an example of the posture of user 5B according to an embodiment. FIG. 29 is a diagram showing a virtual space 2611B and a field of view image 2917B according to an embodiment. After the program starts, user 5B moves his/her body to assume the posture shown in FIG. 28, for example. The posture shown in FIG. 28 corresponds to the first performance. In step S2508, processor 210B detects the movement of user 5B to assume the posture shown in FIG. 28. In detail, processor 210B detects the positions of user 5B's head, waist, both hands, and both feet from each motion sensor attached to user 5B. Processor 210B calculates the rotation direction of the joints of user 5B based on the current position information of user 5B and the dimensional data of user 5B acquired in advance. In this way, detecting the current position information and calculating the rotation direction are synonymous with detecting the movement of user 5B.

In step S2509, the processor 210B controls the avatar object 6B as shown in FIG. 29(A) based on the detected movement of the user 5B. In detail, the processor 210B moves the avatar object 6B placed in the virtual space 2611B based on the current position information and rotation direction of the user 5B. The processor 210B moves the right upper arm of the avatar object 6B based on the rotation direction of the right shoulder, for example. The processor 210B further moves the position of the avatar object 6B in the virtual space 2611B based on the current position information (for example, current position information of the waist). In this way, the processor 210B reflects the movement of the user 5B in the real space in the avatar object 6B placed in the virtual space 2611B. In other words, the processor 210B causes the avatar object 6B to perform a first performance according to the movement of the user 5B.

The process for reflecting the movement of user 5B in avatar object 6B is not limited to the process according to the position information and rotation direction described above. For example, processor 210B may move avatar object 6B according to the movement of user 5B without calculating the rotation direction. For example, processor 210B may control the position of each part object of avatar object 6B corresponding to each part of user 5B so that it corresponds to the position of each part constituting the body of user 5B.

In step S2510, the processor 210B generates motion information in real time that represents the motion of the avatar object 6B when the first performance is performed, and transmits the avatar information of the avatar object 6B including this motion information in real time to the user terminal 800A via the server 600. In step S2511, the processor 210B updates the field of view image 2617B displayed on the monitor 130B. The processor 210B generates, for example, a field of view image 2917B corresponding to the virtual space 2611B shown in FIG. 29(A), and displays it on the monitor 130B as shown in FIG. 29(B). By viewing the field of view image 3017B, the user 5B recognizes that no comments have been received for the program immediately after the first performance is performed.

Figure 30 is a diagram showing the virtual space 2611A and the display surface of the user terminal 800A in an embodiment. In step S2528, the processor 710A receives the avatar information of the avatar object 6B transmitted from the computer 200B in real time. In step S2529, the processor 710A controls the action of the avatar object 6B belonging to the first category in the virtual space 2611A in real time, as shown in Figure 30 (A), based on the movement information of the avatar object 6B included in the received avatar information. In detail, the processor 710A causes the avatar object 6B to perform the first performance belonging to the first category in the virtual space 2611B, based on the movement information of the avatar object 6B included in the avatar information. In this way, the action belonging to the first category is an action that moves a part that constitutes a part of the body of the avatar object 6B. The behavior of the avatar object 6B in the virtual space 2611B is reflected in the avatar object 6B in the virtual space 2611A. In other words, the behavior of avatar object 6B is synchronized in virtual spaces 2611A and 2611B. In this manner, the program for avatar object 6B in virtual space 2611B is distributed to virtual space 2611A.

In step S2530, the processor 710A updates the field of view image 17A displayed on the touch screen 770A. The processor 710A generates, for example, a field of view image 3017A corresponding to the virtual space 2611A shown in FIG. 30(A) and displays it on the touch screen 770A as shown in FIG. 30(B). The user 5A can enjoy the first performance of the avatar object 6B by viewing the field of view image 3017A.

Although not shown, the processor 210B records the voice uttered by the user 5B using the microphone 170B. The processor 210B generates voice data representing the voice of the user 5B and transmits it to the server 600. The server 600 transmits the received voice data of the user 5B to the user terminal 800A by a synchronization process. The user terminal 800A outputs the voice represented by the received voice data of the user 5B to the speaker 780A. As a result of this series of processes, the user 5A can listen to the voice uttered by the user 5B during the live performance in real time.

(Details of embodiment 1)
FIG. 31 is a diagram showing expressions 3161 to 3163 that the avatar object 6B according to an embodiment can have. FIG. 31(A) shows a smiling expression 3161, FIG. 31(B) shows an angry expression 3162, and FIG. 31(C) shows a confused expression 3163. The avatar object 6B (first avatar) can create any one of the smiling expression 3161, the angry expression 3162, and the confused expression 3163 on the face of the avatar object 6B based on a desired operation by the user 5B (first user). The smiling expression 3161 is a facial expression that the avatar object 6B shows when it is laughing, the angry expression 3162 is a facial expression that the avatar object 6B shows when it is angry, and the confused expression 3163 is a facial expression that the avatar object 6B shows when it is in trouble.

These facial expressions are preset for operations performed on any of the buttons on the controller 300RB. For example, a smiley-face facial expression 3161 is set when button 340 is pressed, an angry facial expression 3162 is set when button 350 is pressed, and a confused facial expression 3163 is set when button 370 is pressed. The memory module 530B stores individual motion information representing each of the smiley-face facial expression 3161, the angry facial expression 3162, and the confused facial expression 3163. Hereinafter, the motion information of the avatar object 6B is synonymous with the movement information of the avatar object 6B. The motion information may be any information that specifies the motion of the skeleton of the avatar object 6. The processor 210B obtains the motion information corresponding to the button pressed by the user 5B from the memory module 530B, and causes the avatar object 6B to make the facial expression represented by the information.

Figure 32 is a diagram showing a virtual space 2611B and a field of view image 3217B according to an embodiment. First, a series of steps for machine learning an expression that is popular among viewers of a program of an avatar object 6B will be described below. In Figure 32 (A), an avatar object 6B and a panel object 1832 are placed in a virtual space 2611B. A user 5B presses a button 340 of a controller 300RB to make an avatar object 6B make a smiley face expression 3161. A processor 210B detects the pressing operation of the user 5B on the button 340 as a movement of the user 5B. The processor 210B makes an avatar object 6B make a smiley face expression 3161 corresponding to the pressing operation on the button 340 as shown in Figure 32 (A). The processor 210B transmits motion information representing the smiley face expression 3161 to a user terminal 800A via a server 600.

The processor 210B generates a field of view image 3217B corresponding to the virtual space 2611B shown in FIG. 32(A) and displays it on the monitor 130B, for example, as shown in FIG. 32(B). The field of view image 3217B includes a panel object 1832 on which nothing is displayed. The user 5B recognizes through the field of view image 3217B that no comments have yet been received from viewers in response to the smiling face 3161 of the avatar object 6B.

Figure 33 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. An avatar object 6B is placed in the virtual space 2611A shown in Figure 33 (A). The processor 710A receives motion information representing a smiley face 3161 from the HMD set 110B, and based on that information, causes the avatar object 6B to make a smiley face 3161 in the virtual space 2611A as shown in Figure 33 (B).

The processor 710A generates a field of view image 3317A corresponding to the virtual space 2611A shown in FIG. 33(A) and displays it on the touch screen 770A, for example, as shown in FIG. 33(B). The field of view image 3317A includes an avatar object 6B making a smiley-face expression 3161. The user 5A (second user) recognizes through the field of view image 3317A that the avatar object 6B has made the smiley-face expression 3161.

FIG. 34 is a diagram showing the display surface of a user terminal 800A according to an embodiment. A user 5A finds the smiling face 3161 of an avatar object 6B favorable, and inputs an operation to the user terminal 800A to give such an evaluation to the user 5B. In detail, the user 5A inputs a comment on the smiling face 3161 of the avatar object 6B to the user terminal 800A. The processor 710A displays a software keyboard for inputting a character string on the touch screen 770A, for example, based on an operation of the user 5B on the touch screen 770A. The user 5A inputs a comment to the user terminal 800A by tapping on the software keyboard. The processor 710A detects the input of the comment by the user 5A. As a result, the processor 710A identifies the comment 3453 input by the user 5A.

The processor 710A displays the identified comment 3453 in the comment field 1952 as shown in FIG. 34. By visually checking the comment 3453 displayed in the comment field 1952, the user 5A recognizes that the comment 3453 has been entered correctly. The processor 710A transmits the comment 3453 to the computer 200B via the server 600.

Figure 35 is a diagram showing a virtual space 2611B and a field of view image 3517B according to an embodiment. The processor 210B receives the comment 3453 sent from the user terminal 800A, and displays text 3553 related to the comment 3453 on the display surface of the panel object 1832, as shown in Figure 35 (A). The text includes the comment 3453 of the user 5A and the name of the user 5A. The processor 210B generates a field of view image 3517B corresponding to the virtual space 2611B shown in Figure 35 (A), and displays it on the monitor 130B, for example, as shown in Figure 35 (B). The user 5B recognizes through the field of view image 3517B that a comment has been input from the user 5A in response to the smiling face expression 3161.

FIG. 36 is a diagram showing a trained model 1429 according to an embodiment. The processor 210B identifies the comment 3453 input by the user 5A for the smiling expression 3161 made by the avatar object 6B based on the operation of the user 5B as an evaluation of the smiling expression 3161. The processor 210B associates the identified evaluation of the user 5A with the smiling expression 3161. Furthermore, the trained model 1429 corresponding to the learning result of the expression is generated by machine learning the smiling expression 3161 with which the evaluation is associated. The processor 210B generates the trained model 1429 in which, for example, as shown in FIG. 36, movement information representing expressions that the avatar object 6B can take is associated with the total evaluation for the expression and stored. In FIG. 36, the smiling expression 3161 is associated with the total evaluation of "100", the angry expression 3162 is associated with the total evaluation of "5", and the confused expression 3163 is associated with the total evaluation of "12". In this example, the expressions with the highest total ratings from viewers are the smiling expression 3161, the confused expression 3163, and the angry expression 3162, in that order.

Figure 37 is a sequence chart showing a part of the processing executed in a distribution system 1500 according to an embodiment. The following describes the flow of a series of processing steps when the avatar object 6B automatically creates a suitable facial expression based on the results of machine learning of facial expressions.

FIG. 38 is a diagram showing an example of a user's posture according to an embodiment. During the broadcast of a program, the user 5B moves his/her body to assume the posture shown in FIG. 38, for example. The posture shown in FIG. 38 corresponds to the second performance of the user 5B. In step S3701, the processor 210B detects the movement of the user 5B to assume the posture shown in FIG. 38. In step S3702, the processor 210B identifies information on a new motion that the avatar object 6B can perform based on the trained model 1429. Here, the new motion is a motion in which the avatar object 6B makes a facial expression. The processor 210B preferentially identifies motion information representing a facial expression associated with more evaluations among a plurality of motion information related to facial expressions stored in the trained model 1429. Here, among the smiley-faced expression 3161, the angry expression 3162, and the confused expression 3163, motion information representing the smiley-faced expression 3161 associated with the highest total evaluation of "100" is identified from the trained model 1429.

In step S3703, the processor 210B identifies control content applicable to the avatar object 6B based on the movement of the user 5B and the motion information acquired from the trained model 1429. Here, the processor 210B identifies the control content to create the smiley face expression 3161 represented by the acquired motion information while performing a second performance corresponding to the movement of the user 5B. Although not specifically illustrated, the processor 210B may control the avatar object 6B based on the identified control content in the virtual space 2611B. In step S3704, the processor 210B transmits the identified control content to the user terminal 800A via the server 600.

Figure 39 is a diagram showing virtual space 2611A and the display surface of user terminal 800A according to one embodiment. In step S3705, processor 710A receives control content transmitted from computer 200B. In step S3706, processor 710A controls avatar object 6B as shown in Figure 39 (A) based on the received control content. In detail, avatar object 6B is made to make a smiling face 3161 while performing the second performance. In this way, avatar object 6B can automatically make a smiling face 3161 that is highly evaluated by viewers, even without user 5B performing an operation to specify a facial expression.

The processor 710A generates a field of view image 3917A corresponding to the virtual space 2611A shown in FIG. 39(A) and displays it on the touch screen 770A, for example, as shown in FIG. 39(B). In the field of view image 3917A, avatar object 6B is displayed performing the second performance while making a smiley-face expression 3161. Through the field of view image 3917A, the user 5A enjoys the second performance of avatar object 6B. In particular, the user 5A will have an even more favorable impression of the program of avatar object 6B, because avatar object 6B is making the smiley-face expression 3161 that user 5A previously evaluated.

(Major Effects)
In this embodiment, the control content of the avatar object 6B is identified based on both the movement of the user 5B and the movement information regarding the new movement of the avatar object 6B identified based on the learning results of machine learning of the past movements of the avatar object 6B, so that the avatar object 6B can be more suitably controlled in the virtual space 2611.

In addition, because machine learning is performed on the preferred actions that the avatar object 6B should take based on the viewer's evaluation of the past actions of the avatar object 6B, it is possible to cause the avatar object 6B to perform actions that match the viewer's preferences.

(Modification)
FIG. 40 is a diagram showing a display surface of a user terminal 800A according to an embodiment. In FIG. 40(A), the processor 710A displays a field of view image 3317A on the touch screen 770A, and displays a UI display field 4054 below the field of view image 3317A on the touch screen 770A. The UI display field 4054 is an area for displaying various UI parts related to program viewing. In FIG. 40(A), the processor 710A displays a button 4055 in the UI display field 4054. The button 4055 is labeled with the text "Good". The button 4055 corresponds to a viewer's favorable evaluation of the program. The button 4055 has a function of accepting an input of an evaluation from the user 5A regarding the program of the avatar object 6B.

In FIG. 40(B), user 5B feels favorably about the smiling face 3161 made by avatar object 6B, and inputs an evaluation of smiling face 3161 to touch screen 770A by pressing button 4055 in response to smiling face 3161. Processor 710A detects user 5B's operation of button 4055, generates operation information representing the operation, and transmits it to computer 200B. Based on the operation information received from user terminal 800A, processor 210B determines that user 5B pressed button 4055 in response to avatar object 6B making smiling face 3161. In this way, processor 210B determines user 5A's evaluation of smiling face 3161.

In this example, the processor 210B can also appropriately identify the user 5A's evaluation of the actions that the avatar object 6B has performed in the past. This allows for effective machine learning of the actions of the avatar object 6B that are preferred by viewers of the program.

Figure 41 is a diagram showing the display surface of a user terminal 800A according to an embodiment. In Figure 41 (A), the processor 710A displays a field of view image 3317A on the touch screen 770A, and also displays a UI display field 4054 below the field of view image 3317A on the touch screen 770A. In Figure 41 (A), the processor 710A displays a diamond object 4156 in icon form in the UI display field 4054. The diamond object 4156 is a type of virtual object that is available to the user 5A for a fee. The diamond object 4156 is displayed in the UI display field 4054 in a manner that is capable of accepting an operation to grant the diamond object 4156 to the user 5B.

In FIG. 41(B), user 5A feels favorably towards the smiling face 3161 made by avatar object 6B, and inputs an operation on touch screen 770A to grant paid diamond object 4156 to user 5B in order to support the program distribution by user 5B. In detail, user 5A inputs a swipe operation on diamond object 4156 to move diamond object 4156 from UI display field 4054 to field of view image 3317A. Processor 710A detects the swipe operation of user 5B, and moves diamond object 4156 to the position of field of view image 3317A as shown in FIG. 41(B). In response to the movement of diamond object 4156, processor 710A generates object information representing diamond object 4156 and transmits it to computer 200B.

Figure 42 is a diagram showing a virtual space 2611B and a field of view image 4217B according to an embodiment. The processor 210B receives object information transmitted from the user terminal 800A, and determines based on the information that a diamond object 4156 has been granted from the user 5A to the user 5B. In response to this determination, the processor 210B displays a message 4134 indicating that the diamond object 4156 has been granted to the user 5B on the display surface of the panel object 1832 as shown in Figure 42 (A). The processor 210B further places the diamond object 4156 within the field of view area 15B as shown in Figure 42 (B).

Processor 210B generates field of view image 4217B corresponding to virtual space 2611B shown in FIG. 42(A) and displays it on monitor 130B, for example, as shown in FIG. 42(B). Field of view image 4217B includes panel object 1832 on which message 4134 is displayed, and diamond object 4156 granted to user 5B. User 5B recognizes through field of view image 4217B that he has received diamond object 4156 from user 5A. As a result of diamond object 4156 being granted to user 5B, a portion of the cost incurred by user 5A in purchasing diamond object 4156 is paid to the distributor of the program. This increases sales of the program.

The processor 210B identifies the diamond object 4156 given by the user 5A to the user 5B as the user 5A's evaluation of the smiling expression 3161. In this way, each time a gift object such as the diamond object 4156 is given to the user 5B by a viewer in response to the avatar object 6B making a particular expression, the evaluation associated with that expression becomes higher. In this example as well, the processor 210B can appropriately identify the user 5A's evaluation of actions previously performed by the avatar object 6B. This allows for effective machine learning of actions that are preferred by viewers of the program.

The processor 210B can sell the machine-learned smiling expression 3161 to the user 5A, etc. The user 5A can act as a distributor and distribute a program by the avatar object 6A to the user 5B, etc. In this case, when the processor 210A detects the movement of the user 5A during the distribution of the program by the avatar object 6A, the processor 210A identifies the control content applicable to the avatar object 6A based on the movement and the machine-learned smiling expression 3161 purchased from the user 5B. This allows the avatar object 6A to automatically create the smiling expression 3161 without the user 5A having to perform an operation to control the facial expression of the avatar object 6A.

The processor 210B can perform machine learning to learn any movement of the avatar object 6B based on the movement of the user 5B, and sell the movement information representing the movement. For example, the distribution system 1500 enables the buying and selling of standard movements such as dance and sports (swings and pitching forms), and movements for playing musical instruments.

[Embodiment 2]
FIG. 43 is a diagram showing a user 5B and an avatar object 6B according to an embodiment. In FIG. 43(A), the user 5B is breathing naturally with both hands hanging down. In this case, the movement of the user 5B is small, so there is a possibility that the avatar object 6B cannot be moved normally simply based on the detected movement of the user 5B. Therefore, the processor 210B first generates movement information (first information) regarding the movement of the avatar object 6B corresponding to the detected movement of the user 5B, and then generates corrected movement information (second information) representing the corrected corrected movement (first movement) of the avatar object 6B by correcting the movement information using a predetermined correction parameter. The corrected movement here is a movement in which the avatar object 6B largely moves its shoulders and arms up and down, as shown in FIG. 43(A).

Figure 44 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. In the virtual space 2611A shown in Figure 44 (A), an avatar object 6B is placed within the field of view 15A. The processor 710A receives correction operation information generated in the HMD set 110B from the computer 200B. Based on the received correction operation information, the processor 210B causes the avatar object 6B to perform a correction operation in the virtual space 2611A, as shown in Figure 44 (A). The correction operation here is the same as the correction operation shown in Figure 43 (B).

The processor 710A generates a field of view image 4417A corresponding to the virtual space 2611A shown in FIG. 44(A) and displays it on the touch screen 770A, for example, as shown in FIG. 44(B). The processor 210B displays, together with the field of view image 4417, a button 4055 for giving a favorable rating to the program to the user 5B on the touch screen 770A. The field of view image 4417A shows the avatar object 6B raising and lowering both shoulders and both arms. The user 5A finds this movement unnatural, so does not press the button 4055 when viewing the field of view image 4417A.

Figure 45 is a diagram showing a user 5B and an avatar object 6B according to an embodiment. In Figure 45 (A), the user 5B continues to perform a natural breathing motion with both hands hanging down after the avatar object 6B performs the correction motion shown in Figure 43 (A). The processor 210B determines whether the evaluation of the correction motion performed by the avatar object 6B is equal to or higher than a threshold. Here, it is determined that the evaluation of the correction motion shown in Figure 44 is below a predetermined threshold, corresponding to the user 5A not pressing the button 4055. If the evaluation of the correction motion is below the threshold, the processor 210B adjusts the correction parameters. The adjustment method is not particularly limited. Here, the correction parameters are adjusted so that the degree to which the movement of the user 5B is emphasized is reduced compared to before the adjustment.

The processor 210B generates motion information regarding the motion of the avatar object 6B corresponding to the motion of the user 5B shown in FIG. 45(A), and generates corrected motion information of the avatar object 6B by correcting the motion information using the adjusted correction parameters. The corrected motion represented by this corrected motion information is the motion of the avatar object 6B slightly raising and lowering both shoulders and both arms, as shown in FIG. 45(B), which is a natural motion that matches well with the motion of the user 5B shown in FIG. 45(A).

Figure 46 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. In the virtual space 2611A shown in Figure 46 (A), an avatar object 6B is placed within the viewing area 15A. The processor 710A receives correction action information representing the correction action of the avatar object 6B shown in Figure 45 (B) from the computer 200B. Based on the received correction action information, the processor 210B causes the avatar object 6B to perform the correction action in the virtual space 2611A, as shown in Figure 46 (A). The first action here is the same as the action shown in Figure 45 (B).

The processor 710A generates a field of view image 4617A corresponding to the virtual space 2611A shown in FIG. 46(A) and displays it on the touch screen 770A as shown in FIG. 46(B), for example. The processor 210B displays a button 4055 on the touch screen 770A together with the field of view image 4617. The field of view image 4617A shows the avatar object 6B slightly raising and lowering both shoulders and arms. The user 5A finds this movement natural, so by pressing the button 4055 when viewing the field of view image 4617A, the user 5A gives a favorable evaluation to the movement of the avatar object 6B shown in FIG. 46(B) to the user 5B.

The processor 210B identifies an evaluation given to the first action of the avatar object 6B in the user terminal 800A. The processor 210B performs machine learning on the correction parameters used to generate the correction action information representing the correction action with which the evaluation is associated, thereby generating a trained model 1429 in which the correction parameters are stored.

Figure 47 is a diagram showing a user 5B and an avatar object 6B according to an embodiment. In Figure 47 (A), the user 5B is breathing naturally with both hands close together in front of the body after the correction parameters have been machine-learned. The processor 210B generates motion information of the avatar object 6B corresponding to the motion of the user 5B shown in Figure 47 (A), and further identifies learned correction parameters for correcting the motion information from the learned model 1429. The processor 210B corrects the motion information using the identified correction parameters to generate corrected motion information for making the avatar object 6B perform the corrected motion shown in Figure 47 (B). This corrected motion is a motion in which the avatar object 6B slightly raises and lowers both shoulders and both arms with both hands close together in front of the body as shown in Figure 47 (B), which is a natural motion that matches well with the motion of the user 5B shown in Figure 47 (A).

Figure 48 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. In the virtual space 2611A shown in Figure 48 (A), an avatar object 6B is placed within the field of view 15A. The processor 710A receives correction action information representing the correction action of the avatar object 6B shown in Figure 47 (B) from the computer 200B. Based on the received correction action information, the processor 210B causes the avatar object 6B to perform the correction action in the virtual space 2611A, as shown in Figure 48 (A). The first action here is the same as the action shown in Figure 47 (B).

The processor 710A generates a field of view image 4817A corresponding to the virtual space 2611A shown in FIG. 48(A) and displays it on the touch screen 770A, for example, as shown in FIG. 48(B). The field of view image 4817A shows the avatar object 6B slightly moving both shoulders and both arms up and down. The user 5A feels this movement as natural.

(Major Effects)
In this embodiment, the correction parameters are adjusted if the evaluation of the correction action of the avatar object 6B is below the threshold, and by repeating such adjustments, it is possible to eventually obtain correction parameters that will cause the evaluation of the correction action to exceed the threshold. In this way, by applying the correction parameters to various movements of the user 5A, it becomes possible for the user 5A to visually recognize the movements of the avatar object 6B as natural. As a result, it is possible to make the avatar object 6B behave naturally in the virtual space 2611, which can further increase the satisfaction of the viewer (user 5A) with the program.

(Modification)
When the processor 210B does not detect the movement of the user 5B for a certain period of time, the processor 210B can cause the avatar object 6B to execute a prescribed action. In this case, the action may be a past action of the avatar object 6B that has been learned by machine learning.

[Embodiment 3]
Figure 49 is a diagram showing a user 5B and an avatar object 6B according to an embodiment. In Figure 49(A), the user 5B brings his/her right hand holding the controller 300RB close to the head of the user 5B. The processor 210B operates the virtual right hand 1831RB of the avatar object 6B in the virtual space 2611B, as shown in Figure 49(B), based on the movement of the right hand of the user 5B shown in Figure 49(A). In Figure 49(B), the virtual right hand 1831RB does not stop at the side of the face of the avatar object 6B, but penetrates into the inside of the face of the avatar object 6B.

The processor 210B determines whether the virtual right hand 1831RB, which is moved as shown in FIG. 49(B), is embedded in the avatar object 6B. Specifically, when it is detected that the avatar object 6B and the virtual right hand 1831RB at least partially overlap each other based on the positional relationship between the two, it determines that the virtual right hand 1831RB is embedded in the avatar object 6B. Here, it is determined that the virtual right hand 1831RB is embedded. The processor 210B does not machine learn the movement information representing the movement of the virtual right hand 1831RB embedded in the avatar object 6B.

Figure 50 is a diagram showing a user 5B and an avatar object 6B according to one embodiment. In Figure 50(A), the user 5B brings his/her right hand holding the controller 300RB close to a position slightly away from the head of the user 5B. Based on the movement of the right hand of the user 5B shown in Figure 50(A), the processor 210B moves the virtual right hand 1831RB of the avatar object 6B in the virtual space 2611B as shown in Figure 50(B). In Figure 50(B), the virtual right hand 1831RB stops at a position slightly away from the face of the avatar object 6B and does not penetrate into the face of the avatar object 6B.

Figure 51 is a diagram showing a trained model 1429 according to an embodiment. The processor 210B determines whether the virtual right hand 1831RB, which is moved as shown in Figure 50(B), is embedded into the avatar object 6B. Here, it is determined that the virtual right hand 1831RB is not embedded. Based on the result of this determination, the processor 210B learns motion information representing the motion of the virtual right hand 1831RB shown in Figure 50(B), thereby generating a trained model 1429 in which the motion information is stored, as shown in Figure 51. In Figure 51, motion information "α" associated with "no embedding" is stored in the trained model 1429.

Figure 52 is a diagram showing a user 5B and an avatar object 6B according to an embodiment. After learning the motion information, the user 5B brings the right hand holding the controller 300RB close to the head of the user 5B as shown in Figure 52 (A). This is the same as the motion of the user 5B shown in Figure 49 (A). When the processor 210B detects the motion of the right hand of the user 5B shown in Figure 52 (A), it accesses the trained model 1429 and identifies machine-learned motion information α from among multiple pieces of motion information related to the motion of the virtual right hand 1831RB. Based on the detected motion of the right hand of the user 5B and the motion information α identified from the trained model 1429, the processor 210B identifies the control content that prevents the virtual right hand 1831RB from penetrating the avatar object 6B. Here, the movement of avatar object 6B represented by the movement information α, i.e., the movement of bringing virtual right hand 1831RB very close to the head of avatar object 6B, is identified as the control content of avatar object 6B corresponding to the movement of user 5B. Based on the identified control content, processor 210B controls virtual right hand 1831RB as shown in FIG. 52(B). As a result, when user 5B performs a movement in which virtual right hand 1831RB was embedded into avatar object 6B as shown in FIG. 49(B) before learning movement information α, virtual right hand 1831RB does not embed into avatar object 6B as shown in FIG. 52(B) after learning movement information α.

[Embodiment 4]
In this embodiment, the user 5B performs a setting on the HMD set 110B to have the avatar object 6B speak a converted voice (second voice) obtained by converting the voice (first voice) instead of the voice uttered by the user 5B. When starting the setting, the user 5B utters a predetermined voice toward the microphone 170A of the HMD set 110B. When detecting the movement of the user 5B, the processor 210B detects the voice uttered by the user 5B through the microphone 170A. The processor 210B converts the detected voice of the user 5B into a converted voice using a conversion parameter (first parameter) for voice conversion. The default values of the conversion parameters are stored in advance in the memory module 530B.

The processor 210B determines whether the second voice has the same sound quality as the reference voice. The reference voice is a voice having a sound quality that the user 5B wants the avatar object 6B to speak in place of his or her own voice. Before starting the setup, the user 5B obtains voice data representing the reference voice and stores it in the HMD set 110B, and can listen to the reference voice by playing back the voice data. The processor 210B determines whether the sound quality of the second voice and the reference voice is the same based on the voice data representing the second voice and the voice data representing the reference voice. The method of determination is not particularly limited.

If the processor 210B determines that the two have the same sound quality, it generates a learning result for the voice conversion by machine learning the conversion parameters used in the voice conversion. In detail, the processor 210B generates a learned model 1429 in which the machine-learned conversion parameters are stored as the learning result. If the processor 210B determines that the two do not have the same sound quality, it does not learn the conversion parameters used in the voice conversion. Instead, it corrects the conversion parameters to obtain new conversion parameters. The processor 210B uses the new conversion parameters to perform the conversion process of the user 5B's voice and the process of determining the identity of the sound quality between the converted voice and the reference voice. At that time, the voice conversion and the process of determining the identity are repeatedly performed while changing the degree of correction of the conversion parameters each time the sound quality is determined to be inconsistent. If the processor 210B generates the learned model 1429 in which the machine-learned conversion parameters are stored, it ends the voice conversion setting process. The processor 210B may notify the user 5B that the setting has ended.

After the conversion parameters are machine-learned, the processor 210B identifies the conversion parameters stored in the trained model 1429 as information related to a new action that the avatar object 6B can perform. Each time the user 5B speaks a sound while the program is in progress, the processor 210B detects the sound, then converts the sound using the conversion parameters to generate a converted sound for speaking, and has the avatar object 6B speak the converted sound in the virtual space 2611B. At that time, the processor 210B outputs the converted sound from the speaker 180B.

The processor 210B transmits the identified conversion parameters to the user terminal 800A via the server 600. The processor 210B also generates voice data representing the detected voice of the user 5B and transmits the voice data to the user terminal 800A via the server 600. The processor 710A does not cause the avatar object 6B to speak the voice of the user 5B represented by the received voice data, but generates a converted voice by converting the voice of the user 5B using the received conversion parameters, and causes the avatar object 6B to speak the converted voice in the virtual space 2611A.

(Major Effects)
In this embodiment, conversion parameters for converting the voice of user 5B into a voice having the same sound quality as the reference voice are learned, and the parameters are used to automatically convert the voice spoken by user 5B during program distribution into a voice having the same sound quality as the reference voice and have avatar object 6B speak the voice. This allows user 5B to have avatar object 6B speak a voice having a desired sound quality that user 5B prefers, thereby further improving the interest of the program by avatar object 6B. Furthermore, distribution system 1500 can motivate user 5 who is not very confident in his or her own voice to have a program distributed using avatar object 6.

(Modification)
The processor 210B can sell the converted voice obtained by converting the voice of the user 5B using the machine-learned conversion parameters to any user 5 other than the user 5B. For example, if the user 5A purchases the converted voice, the converted voice functions as a reference voice for the user 5A. By listening to the reference voice, the user 5A can set the above-mentioned conversion parameters using, for example, the HMD set 110A. In this way, when a program using an avatar object 6A associated with the user 5A is distributed, the user 5A can have the object speak a voice with the same sound quality as the voice spoken by the avatar object 6B in place of the voice of the user 5A.

When user 5B sings during a program broadcast, processor 210B can detect the singing voice of user 5B and automatically correct the pitch of the singing voice. Processor 210B can further cause avatar object 6B to speak the singing voice of user 5B after pitch correction, thereby causing avatar object 6B to sing a song with an appropriate pitch. Such pitch correction can also be performed by processor 710A in user terminal 800A.

The processor 210B can make the voice spoken by the user 5B sound as if the avatar object 6B is singing in the virtual space 2611B. Such processing can also be executed by the processor 710A in the user terminal 800A.

The processor 210B can make the avatar object 6B make a facial expression corresponding to the voice of the user 5B. The processor 210B, for example, detects various parameters (waveform, frequency, voice pitch, voice depth) related to the detected voice of the user 5B, and identifies a facial expression corresponding to the parameters. For example, when the waveform of the voice of the user 5B is a waveform of the first pattern, the processor 210B identifies the smiley face expression 3161 corresponding to the waveform from the memory module 530B and reflects it on the face of the avatar object 6B. This allows the avatar object 6B to automatically make an appropriate facial expression that matches the voice uttered by the user 5B, so that the user 5B does not need to perform any operation to make the avatar object 6B make an appropriate facial expression that matches the voice uttered by the user 5B.

[Embodiment 5]
53 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In FIG. 53(A), an avatar object 6B arranged in the virtual space 2611A makes a confused expression 3163. The processor 710A identifies the fact that the avatar object 6B makes the confused expression 3163 as an event that has occurred in the virtual space 2611A. The processor 710A outputs information corresponding to the identified event to the virtual space 2611A. In FIG. 53(A), the processor 710A reproduces a sound 5371 of "Gaah" corresponding to the confused expression 3163 in the virtual space 2611A as a sound generated by the avatar object 6B.

The processor 710A generates a field of view image 5317A corresponding to the virtual space 2611A shown in FIG. 53(A) and displays it on the touch screen 770A, for example, as shown in FIG. 53(B). The processor 210B further causes audio 5371 to be output from the speaker 780A. The user 5A listens to the audio 5371 coming from the speaker 780A while viewing the field of view image 5317A. In this way, the user 5B hears the audio 5371 that appropriately matches the confused expression 3163 of the avatar object 6B.

Figure 54 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. In Figure 54(A), avatar object 6B is making a confused expression 3163 based on the operation of user 5B, similar to Figure 53(A). Processor 710A identifies the event that avatar object 6B is making a confused expression 3163, and outputs text 5472 "Gah" as information corresponding to the event into field of view 15A in virtual space 2611A.

Processor 710A generates field of view image 5417A corresponding to virtual space 2611A shown in FIG. 54(A) and displays it on touch screen 770A, for example, as shown in FIG. 54(B). Since avatar object 6B and text 5472 are included in field of view area 15A in FIG. 54(A), field of view image 5417A shown in FIG. 54 includes avatar object 6B and text 5472 superimposed on avatar object 6B. User 5A recognizes through field of view image 5417A that when avatar object 6B makes confused expression 3163, text 5472 saying "Gah," which appropriately matches confused expression 3163, is displayed as a caption in the program.

Figure 55 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In Figure 54(A), avatar object 6B is making a confused expression 3163 based on the operation of user 5B, similar to Figure 53(A). Processor 710A identifies the event that avatar object 6B is making a confused expression 3163, and reflects special effect 5573 in virtual space 2611A as information corresponding to the event. Special effect 5573 is an effect of darkening at least a portion of virtual space 2611A. In Figure 55(A), as a result of special effect 5573 being output to virtual space 2611A, the entire viewing area 15A is darkened.

Processor 710A generates field of view image 5517A corresponding to virtual space 2611A shown in FIG. 55(A) and displays it on touch screen 770A, for example, as shown in FIG. 55(B). Because special effect 5573 is reflected within field of view area 15A, field of view image 5517A reflecting special effect 5573 throughout is displayed. User 5B recognizes through field of view image 5617A that when avatar object 6B makes a confused expression 3163, special effect 5573, a blackout that appropriately matches the confused expression 3163, is reflected in virtual space 2611A.

(Major Effects)
According to each example of this embodiment, various appropriate information corresponding to an event occurring in the virtual space 2611A is automatically output to the virtual space 2611A, thereby making the program of the avatar object 6B in the virtual space 2611A more exciting.

[Embodiment 6]
FIG. 56 is a sequence chart showing a part of the process executed in the HMD set 110B according to an embodiment. FIG. 57 is a diagram showing a virtual space 2611B and a field of view image 17B according to an embodiment. In the example of FIG. 57, the processor 210B defines a virtual space 2611B including an avatar object 6B associated with the user 5B and an avatar object 5706 controlled independently of the user 5B. The avatar object 5706 is a virtual object having an appearance imitating a human body like the avatar object 6B, and has a virtual left hand 5231L and a virtual right hand 5231R. In the virtual space 2611B, the avatar object 5706 is arranged in the field of view area 15B with its back surface facing the front of the avatar object 6B.

In FIG. 57(A), avatar object 5706 is in a predetermined pose. Processor 210B generates field of view image 5317A corresponding to virtual space 2611B shown in FIG. 57(A) and displays it on monitor 130B, for example, as shown in FIG. 53(B). User 5B views avatar object 5706 placed in virtual space 2611B by viewing field of view image 5317B.

Figure 58 is a diagram showing a virtual space 2611B and a field of view image 5817B according to an embodiment. In S5601, when an avatar object 5706 is placed in the field of view area 15B, the processor 210B causes the avatar object 5706 to perform a sample action (first action) as shown in Figure 58 (A). Here, the sample action is an action of moving the virtual left hand 5231L close to the front of the face of the avatar object 5706. The memory module 530B stores in advance action information representing the sample action by the avatar object 5706, and the processor 210B causes the avatar object 5706 to automatically perform the sample action based on the action information. In this way, the avatar object 5706 is an object that is automatically controlled without being linked to the movement of each user, including the user 5B.

Processor 210B generates field of view image 5317B corresponding to virtual space 2611B shown in FIG. 58(A) and displays it on monitor 130B, for example, as shown in FIG. 58(B). By viewing field of view image 5817B, user 5B recognizes that avatar object 5706 has performed a model action that user 5B should follow. While viewing the model action by avatar object 5706, user 5B moves his/her left hand to imitate the action. In detail, user 5B moves his/her left hand until it is close to the front of his/her face, similar to avatar object 5706.

Figure 59 is a diagram showing a virtual space 2611B and a field of view image 5917B according to one embodiment. In step S5602, the processor 210B detects the movement of the user 5B imitating the sample movement while the avatar object 5706 is performing the sample movement. In step S5603, the processor 210B causes the avatar object 6B to perform the same movement as the sample movement when the avatar object 5706 is positioned within the field of view area 15, based on the detected movement of the user 5B. Since the detected movement of the user 5B corresponds to the sample movement, the processor 210B can cause the avatar object 6B to perform the same movement as the sample movement, based on the detected movement of the user 5B.

Processor 210B generates field of view image 5917B corresponding to virtual space 2611B shown in FIG. 59(A) and displays it on monitor 130B, for example, as shown in FIG. 59(B). By visually checking field of view image 5917B, user 5B recognizes that virtual left hand 1831LB of avatar object 6B has moved in the same manner as virtual left hand 5231L of avatar object 5706, and has therefore moved close to the face of avatar object 6B (sensibly, that of user 5B).

Figure 60 is a diagram showing a virtual space 2611A and the display surface of a user terminal 800A according to one embodiment. The processor 810A defines the virtual space 2611A including an avatar object 6B. The processor 210B does not transmit information about the avatar object 5706 to the user terminal 810, and the processor 810A does not place the avatar object 5706 in the virtual space 2611A. Therefore, the avatar object 5706 is not displayed on the display surface of the user terminal 810. In this way, the avatar object 5706 is an object that is visible to the user 5B in the virtual space 2611B, but is not visible to the user 5A in the virtual space 2611A.

The processor 210B generates motion information representing a first motion performed by the avatar object 6B, and transmits it to the user terminal 800A via the server 600. The processor 710A receives the transmitted motion information, and causes the avatar object 6B placed in the virtual space 2611A to perform the first motion based on the motion information, as shown in FIG. 60(A). In detail, the processor 810A moves the virtual left hand 1831LB of the avatar object 6B to close to the front of the face of the avatar object 6B in the virtual space 2611A. This causes the motion of the avatar object 6B in the virtual space 2611A to be synchronized with the motion of the avatar object 6B in the virtual space 2611B.

The processor 710A generates a field of view image 6017A corresponding to the virtual space 2611A shown in FIG. 60(A) and displays it on the touch screen 770A, for example, as shown in FIG. 60(B). The user 5A recognizes through the field of view image 6017A that the virtual left hand 1831LB of the avatar object 6B has moved close to the face of the avatar object 6B.

(Major Effects)
In this embodiment, the user 5B can perform a desired movement to make the avatar object 6B perform a desired movement while referring to the movement of the model avatar object 5706 in real time. This eliminates the need for the user 5B to worry that the movement of the avatar object 6B does not match the movement of the user 5B. Furthermore, the user 5B can proceed with the program with more peace of mind, which allows the program to proceed more smoothly.

[Embodiment 7]
Fig. 61 is a sequence chart showing a part of the process executed in the HMD set 110B according to an embodiment. Fig. 62 is a diagram showing a virtual space 2611B and a field of view image 5717B according to an embodiment. In the example of Fig. 62, the processor 210B defines a virtual space 2611B including an avatar object 6B and a panel object 1832. The panel object 1832 is placed in the field of view area 15B at a certain distance from the front of the avatar object 6B, with its display surface facing the front of the avatar object 6B. When the virtual space 2611B is defined in the manner shown in Fig. 61, nothing is displayed on the panel object 1832.

The processor 210B generates a field of view image 5717B corresponding to the virtual space 2611B shown in FIG. 62(A) and displays it on the monitor 130B, for example, as shown in FIG. 62(B). The field of view image 6317B includes a panel object 1832 that is placed in the upper region within the field of view image 6317B. The user 5B views the panel object 1832, with nothing displayed on the display surface, through the field of view image 5717B.

FIG. 63 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In the example of FIG. 63, the processor 710A defines a virtual space 2611A that includes an avatar object 6B but does not include a panel object 1832. The behavior of the avatar object 6B in the virtual space 2611A is synchronized with the behavior of the avatar object 6B in the virtual space 2611B. The processor 210B does not transmit information about the panel object 1832 to the user terminal 800A, and the processor 710A does not place the panel object 1832 in the virtual space 2611A. The processor 710A does not display the panel object 1832 on the display surface of the user terminal 800, regardless of which direction the virtual viewpoint 1951 is facing in the virtual space 2611A.

The processor 710A generates a field of view image 6317A corresponding to the virtual space 2611A shown in FIG. 63(A), and displays it on the display surface of the touch screen 770A, for example, as shown in FIG. 63(B). The avatar object 6B is displayed in the field of view image 6317A so as to face the right end of the display surface of the user terminal 800A. When displaying the field of view image 6317A, the processor 710A further displays a UI display field 4054 below the field of view image 6317A on the touch screen 770A. The processor 710A further displays an enemy object 6357 in an icon state in the UI display field 4054. The enemy object 6357 is a type of virtual object that can be placed in the virtual space 2611. When the enemy object 6357 is displayed in the UI display field 4054, it functions as a UI part that can accept a selection operation of the enemy object 6357 by the user 5A. The enemy object 6357 is associated with an action (first action) that the user 5A requests of the avatar object 6B. In FIG. 63, the action associated with the enemy object 6357 is a combat action in which "avatar object 6B fights enemy object 6357."

Figure 64 is a diagram showing the display surface of a user terminal 800A in one embodiment. As shown in Figure 64, a user 5A performs an operation on the touch screen 770A to assign an enemy object 6357 to an avatar object 6B. In Figure 64, this operation is a swipe operation in which the enemy object 6357 is moved from the UI display field 4054 to the field of view image 6317A while being touched by the user 5B. When the processor 210B detects this swipe operation, as shown in Figure 64, the processor 210B moves the enemy object 6357 from the UI display field 4054 to the field of view image 6317A on the display surface of the touch screen 770A.

65 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In response to the enemy object 6357 moving onto the field of view image 6417A, the processor 710A places the enemy object 6357 in the virtual space 2611A as shown in FIG. 65. In FIG. 65, the enemy object 6357 is placed in the field of view area 15A so as to face the avatar object 6B in the virtual space 2611A. The processor 710A generates a field of view image 6517A corresponding to the virtual space 2611A shown in FIG. 65(A), and displays it on the touch screen 770A, for example, as shown in FIG. 65(B). By viewing the field of view image 6517A, the user 5A recognizes that the enemy object 6357 has been placed in the virtual space 2611A so as to face the avatar object 6B as a result of the swipe operation of the user 5A.

Figure 66 is a diagram showing a virtual space 2611B and a field of view image 6617B according to one embodiment. In response to an enemy object 6357 being placed in the virtual space 2611A, the processor 710A generates information about the enemy object 6357 and transmits it to the server 600. The information includes at least information representing the position and orientation of the enemy object 6357, and combat actions associated with the enemy object 6357. The server 600 transmits the information received from the user terminal 800A to the computer 200B. The processor 210B receives the information transmitted from the server 600.

In step S6101, the processor 200B places the enemy object 6357 in the virtual space 2611B as shown in FIG. 66 based on the information on the enemy object 6357 received from the server 600. In FIG. 66, the enemy object 6357 is placed in the field of view 15B and faces the avatar object 6B. When placing the enemy object 6357 in the virtual space 2611B, the processor 210B displays the combat action associated with the enemy object 6357 in the virtual space 2611B. In FIG. 66(A), the combat action is displayed as text in a manner explaining the combat action to the user 5B on the display surface of the panel object 1832 placed in the field of view 15B.

Processor 210B generates field of view image 6617B corresponding to virtual space 2611B shown in FIG. 66(A) and displays it on monitor 130B, for example, as shown in FIG. 66(B). Field of view image 6617B includes enemy object 6357 and panel object 1832. In field of view image 6617B, text 6658 explaining the combat action requested by user 5A to avatar object 6B is displayed on panel object 1832. User 5B recognizes through field of view image 6617B that enemy object 6357 has appeared in front of avatar object 6B and that user 5A has requested avatar object 6B to combat enemy object 6357.

Figure 67 is a diagram showing a virtual space 2611B and a field of view image 6717B according to one embodiment. In step S6502, in response to the placement of the enemy object 6357, the processor 210B places a weapon object 6759 in the virtual space 2611B as shown in Figure 67, in association with the avatar object 6B. The weapon object 6759 is a type of virtual object that can be used by the avatar object 6B to attack the enemy object 6357. The weapon object 6759 is associated with the avatar object 6B by being held by the virtual right hand 1831RB.

FIG. 68 is a diagram showing a virtual space 2611B and a field of view image 6817B according to an embodiment. After the avatar object 6B holds the weapon object 6759 in the virtual right hand 1831RB, the user 5B inputs an operation to the HMD set 110B to make the avatar object 6B fight the enemy object 6357 in response to a request from the user 5A. Here, the user 5B moves the right hand of the user 5B so as to slash the enemy object 6357 with the weapon object 6759. In step S6503, the processor 210B detects the movement of the right hand by the user 5B. In step S6504, the processor 210B causes the avatar object 6B to perform a combat action against the enemy object 6357, as shown in FIG. 68(A), based on the detected movement of the user 5B. In FIG. 68, the processor 210B causes the avatar object 6B to perform a combat action of attacking the enemy object 6357 with the weapon object 6759.

The processor 210B generates a field of view image 6817B corresponding to the virtual space 2611B shown in FIG. 68(A) and displays it on the monitor 130B, for example, as shown in FIG. 68(B). The field of view image 6817B shows the enemy object 6357 being slashed with the weapon object 6759. The user 5B recognizes through the field of view image 6817B that he has been able to control the avatar object 6B to comply with the request of the user 5A.

Figure 69 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. In response to having the avatar object 6B hold the weapon object 6759 as shown in Figure 67, the processor 210B generates information about the weapon object 6759 and transmits it to the server 600. The server 600 transfers the information about the weapon object 6759 to the user terminal 800A. The processor 710A receives the information from the server 600 and, based on the information, has the avatar object 6B hold the weapon object 6759 in the virtual right hand 1831RB in the virtual space 2611A as shown in Figure 69 (A). The processor 710A generates a field of view image 6917A corresponding to the virtual space 2611A shown in Figure 69 (A) and displays it on the touch screen 770A as shown in Figure 69 (B), for example. User 5A understands through field of view image 6917A that avatar object 6B is grasping weapon object 6759 to fight enemy object 6357.

Figure 70 is a diagram showing a virtual space 2611A and a display surface of a user terminal 800A according to an embodiment. The processor 210B generates motion information representing the motion of the avatar object 6B shown in Figure 68, and transmits it to the user terminal 800A via the server 600. The processor 210B receives the motion information of the avatar object 6B, and based on the information, causes the avatar object 6B to perform a combat motion against an enemy object 6357 in the virtual space 2611A as shown in Figure 70 (A). In Figure 70, the avatar object 6B in the virtual space 2611A performs a combat motion to attack the enemy object 6357 using a weapon object 6759 in conjunction with the motion of the avatar object 6B in the virtual space 2611B shown in Figure 68.

The processor 710A generates a field of view image 7017A corresponding to the virtual space 2611A shown in FIG. 70(A) and displays it on the touch screen 770A, for example, as shown in FIG. 70(B). Through the field of view image 7017A, the user 5A understands that the avatar object 6B has responded to the user 5A's request by fighting the enemy object 6357.

(Major Effects)
In this embodiment, the avatar object 6B can be caused to execute an action requested by the user 5A, so that the interest of the user 5A in the virtual space 2611 can be improved.

Although the embodiment of the present disclosure has been described above, the technical scope of the present invention should not be interpreted as being limited by the description of this embodiment. This embodiment is merely an example, and it will be understood by those skilled in the art that various modifications of the embodiment are possible within the scope of the invention described in the claims. The technical scope of the present invention should be determined based on the scope of the invention described in the claims and its equivalents.

Each embodiment and its various modifications may be applied to other embodiments or other modifications as long as they do not contradict each other.

The "machine learning" in each embodiment may be, for example, deep learning. For example, the processor 210B can generate a trained model 1429 in which the smiling expression 3161 has been deep-learned by deep learning the smiling expression 3161 associated with the evaluation from the viewer.

The HMD set 110 may be configured to integrate the HMD 120, computer 200, HMD sensor 410, and motion sensor 420.

[Additional Notes]
The contents related to one aspect of the present invention are as follows.

(Item 1) A program has been described. According to an aspect of the present disclosure, the program is executed by a computer (200B) having a processor (210B) and a memory (220B). The program causes the processor to execute the steps of: defining a virtual space (2611B) including an avatar (avatar object 6B) associated with a first user (user 5B); detecting the movement of the first user (S3701); identifying information on a new movement that the avatar may perform based on a learning result (trained model 1429) obtained by machine learning a past first movement of the avatar based on artificial intelligence (S3702); identifying control content applicable to the avatar based on the detected movement of the first user and movement information on the identified movement (S3704); and controlling the avatar based on the identified control content.

(Item 2) In (Item 1), the program causes the processor to execute a step of identifying an evaluation for a first action, and a step of generating a learning result by machine learning the first action associated with the identified evaluation, and in the step of identifying information, from among the multiple pieces of action information, preferentially identifying action information relating to an action associated with a larger number of evaluations.

(Item 3) In (Item 2), in the step of identifying the evaluation, the evaluation is identified based on an operation performed by the second user in response to the first action on a button (4055) displayed in the virtual space.
(Item 4) In (Item 2), in the step of identifying an evaluation, a comment (3533) input by the second user in response to the action is identified as the evaluation.

(Item 5) In (Item 2), in the step of identifying an evaluation, an object (4156) given by the second user to the first user in response to an action performed by the avatar is identified as an evaluation of the action.

(Item 6) In any of items 2 to 5, the first action is an action of the avatar making a facial expression, and in the step of generating a learning result, the facial expression to which an evaluation has been assigned (smiling expression 3161) is machine-learned, in the step of identifying information, the machine-learned facial expression is identified from among a plurality of facial expressions, and in the step of identifying control content, control content indicating a new facial expression to be made by the avatar is identified based on the detected movement of the first user and the identified facial expression.

(Item 7) In any of items 2 to 5, the program causes the processor to execute the steps of: generating second information about the first information regarding the movement of the avatar corresponding to the detected movement of the first user by correcting the first information with the first parameter; causing the avatar to perform the first movement based on the generated second information; and adjusting the first parameter if the identified evaluation is below a threshold value.

(Item 8) In (Item 1), the avatar has a control object, and the program causes the processor to execute a step of operating the control object (virtual right hand 1831RB) in a virtual space based on the detected movement of the first user, a step of determining whether the operated control object has become embedded in the avatar, and a step of generating a learning result by machine learning motion information related to the motion of the control object when it is determined that the object has not become embedded. In the step of identifying information, machine-learned motion information is identified from multiple pieces of motion information related to the motion of the control object, and in the step of identifying control content, control content that prevents the control object from becoming embedded in the avatar is identified based on the detected movement of the first user and the identified motion information.

(Item 9) In any of items 1 and 2, in the step of detecting the movement of the first user, a first voice uttered by the first user is detected, and the program causes the processor to execute the steps of converting the detected first voice into a second voice using a first parameter for voice conversion, determining whether the second voice has the same sound quality as a reference voice, and, if it is determined that the second voice has the same sound quality, generating a learning result by machine learning the first parameter. In the step of identifying information, the machine-learned first parameter is identified as information, and in the step of controlling, a third voice is generated by converting the detected voice of the first user using the identified first parameter, and the third voice is spoken by an avatar.

(Item 10) In any of (Item 2) to (Item 9), in the step of generating a learning result, the first action is deep-learned.

(Item 11) In any of (Item 1) to (Item 10) that executes the above, the program causes the processor to identify an event that has occurred in the virtual space, and output information corresponding to the identified event to the virtual space.

(Item 12) In (Item 11), in the notification step, audio corresponding to the identified event is played in the virtual space.

(Item 13) In (Item 11), in the notification step, text corresponding to the identified event is displayed in the virtual space.

(Item 14) In (Item 11), in the notification step, a special effect corresponding to the identified event is reflected in the virtual space.

(Item 15) In any of items 1 to 14, the program causes the processor to execute a step of selling the identified control content.

(Item 16) An information processing device has been described. According to an aspect of the present disclosure, the information processing device (computer 200B) includes a memory unit (storage 230B) that stores a program executed by the information processing device, and a control unit (processor 210B) that controls the operation of the information processing device by executing the program. The control unit defines a virtual space (2611B) including an avatar (avatar object 6B) associated with a first user (user 5B), detects the movement of the first user, identifies information on a new action that the avatar may perform based on a learning result (trained model 1429) obtained by machine learning the past first action of the avatar based on artificial intelligence, identifies control content applicable to the avatar based on the detected movement of the first user and the action information related to the identified action, and controls the avatar based on the identified control content.

(Item 17) A method for executing a program has been described. According to an aspect of the present disclosure, the program is executed by a computer (200B) having a processor (210B) and a memory (220B). The method includes a step (S2501) in which the processor defines a virtual space (2611B) including an avatar (avatar object 6B) associated with a first user (user 5B), a step (S3701) in which the processor detects the movement of the first user, a step (S3702) in which the processor determines information on a new movement that the avatar may perform based on a learning result (trained model 1429) obtained by machine learning the past first movement of the avatar based on artificial intelligence, a step (S3704) in which the processor determines control content applicable to the avatar based on the detected movement of the first user and the movement information related to the determined movement, and a step (S3704) in which the processor determines control content applicable to the avatar based on the determined control content.

(Item 18) A program has been described. According to an aspect of the present disclosure, the program is executed by a computer (user terminal 200B) having a processor (210B) and a memory (220B). The program causes the processor to execute the steps of: defining a virtual space (virtual space 2611B) including a first avatar (avatar object 6B) associated with a first user (user 5B) and a second avatar (avatar object 5706); causing the second avatar to perform a first action when the second avatar is placed within the field of view (field of view area 15B) of the first avatar; detecting the movement of the first user while the second avatar is performing the first action; and causing the first avatar to perform the first action when the second avatar is placed within the field of view based on the detected movement of the first user.

(Item 19) A program has been described. According to an aspect of the present disclosure, the program is executed by a computer (user terminal 200B) having a processor (210B) and a memory (220B). The program causes the processor to execute the steps of: defining a virtual space (virtual space 2611B) including an avatar (avatar object 6B) associated with a first user (user 5B); arranging an object (enemy object 6357) associated with an action requested of the first avatar by a second user (user 5A) in the virtual space based on an operation by the second user; detecting the movement of the first user; and causing the avatar to perform an action on the object based on the detected movement of the user.

In the above embodiment, a virtual space (VR space) in which the user is immersed by the HMD has been described as an example, but a see-through HMD may be used as the HMD. In this case, a field of view image in which a part of the image constituting the virtual space is synthesized in the real space visually recognized by the user through the see-through HMD may be output to provide the user with a virtual experience in an augmented reality (AR) space or a mixed reality (MR) space. In this case, an action may be exerted on a target object in the virtual space based on the movement of the user's hand instead of the operation object. Specifically, the processor may specify coordinate information of the position of the user's hand in the real space, and may define the position of the target object in the virtual space in relation to the coordinate information in the real space. This allows the processor to grasp the positional relationship between the user's hand in the real space and the target object in the virtual space, and to execute processing corresponding to the above-mentioned collision control between the user's hand and the target object. As a result, it becomes possible to exert an action on the target object based on the movement of the user's hand.

2 network, 5, 5A, 5B, 5C, 5D user, 6, 6A, 6B, 6C, 6D avatar object, 11, 11A, 11B, 11C, 11D virtual space, 12 center, 13 panoramic image, 14, 14A, 14B virtual camera, 15, 15A, 15B, 15C field of view area, 16 reference line of sight, 17, 17A, 17B field of view image, 18, 19 area, 100 HMD system, 110, 110A, 110B, 110C, 110D HMD set, 120, 120A, 120B, 120C, HMD, 130, 130A, 130B, 130C monitor, 140 gaze sensor, 150 first camera, 160 second camera, 170, 170A, 170B Microphone, 180, 180A, 180B Speaker, 190 Sensor, 200, 200A, 200B Computer, 210, 210A, 210B, 210C, 210D, 610, 710, 710A Processor, 220, 620, 720 Memory, 230, 230A, 230B, 630, 730 Storage, 240, 640, 740 Input/output interface, 250, 650, 750 Communication interface, 260, 660, 760 Bus, 300, 300B Controller, 300R Right controller, 300L Left controller, 310 Grip, 320 Frame, 330
Top surface, 340, 340, 350, 370, 380 Button, 360 Infrared LED, 390 Analog stick, 410 HMD sensor, 420, 420A Motion sensor, 430, 430A Display, 510 Control module, 520 Rendering module, 530 Memory module, 540 Communication control module, 600
Server, 700 External device, 770, 770A Touch screen, 780A Speaker, 1421 Virtual object generation module, 1422 Virtual camera control module, 1423 Operation object control module, 1424 Avatar object control module, 1425 Motion detection module, 1426 Collision detection module, 1427 Virtual object control module, 1428 Learning module, 1429 Learned model, 1831LB Virtual left hand, 1831RB Virtual right hand, 1832 Panel object, 1841, 1842, 1843 Motion sensor, 1844 Belt, 1951 Virtual viewpoint, 1952 Comment section, 2611A, 2611B Virtual space 1817B, 1917A, 2617, 2617B, 2717A, 2917B, 3017A, 3017B, 3217B, 3317A, 3517B, 3917A, 4117A, 4217B, 4417, 4417A, 4617, 4617A, 4817A, 5317A, 5317B, 5417A, 5517A, 5617A, 5717B, 5817B, 5917B, 6017A, 6317A, 6317B, 6417A, 6517A, 6617B, 6717B, 6817B, 6917A, 7017A Field of view image 3161 Smiley face expression, 3162 Angry face expression, 3163 Confused expression, 3453, 3533
Comment, 4054 UI display column, 4134 Message, 4156 Diamond object, 5371 Sound, 5472, 6658 Text, 5573 Special effects, 6357
Enemy object, 6759 Weapon object

Claims (6)

A program executed by a computer having a processor and a memory,
The program causes the processor to:
A definition step of defining a virtual space including an avatar associated with a first user;
detecting a movement of the first user;
a control step of determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and determining an evaluation for the plurality of pieces of motion information ;
In the control step, a piece of motion information related to a motion associated with a larger number of the evaluations is determined as having a higher priority among the plurality of pieces of motion information ;
The program , in the step of identifying the evaluation, identifies the evaluation based on an operation performed by a second user on a button displayed in the virtual space in response to the plurality of pieces of motion information. A program executed by a computer having a processor and a memory,
The program causes the processor to:
A definition step of defining a virtual space including an avatar associated with a first user;
detecting a movement of the first user;
a control step of determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and determining an evaluation for the plurality of pieces of motion information;
In the control step, a piece of motion information related to a motion associated with a larger number of the evaluations is determined as having a higher priority among the plurality of pieces of motion information;
In the step of identifying the evaluation, a comment input by a second user in response to the plurality of pieces of motion information is identified as the evaluation. A program executed by a computer having a processor and a memory,
The program causes the processor to:
A definition step of defining a virtual space including an avatar associated with a first user;
detecting a movement of the first user;
a control step of determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and determining an evaluation for the plurality of pieces of motion information;
In the control step, a piece of motion information related to a motion associated with a larger number of the evaluations is determined as having a higher priority among the plurality of pieces of motion information;
In the step of identifying an evaluation, an object given to the first user by a second user in response to the plurality of pieces of action information is identified as the evaluation for the action. A definition means for defining a virtual space including an avatar associated with a first user;
A detection means for detecting a movement of the first user;
a control means for determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and for controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and a determination unit that determines an evaluation for the plurality of pieces of motion information . The control unit determines, among the plurality of pieces of motion information, motion information related to the motion associated with a larger number of the evaluations, with a higher priority.
The system is configured such that the means for identifying the evaluation performs control to identify the evaluation based on an operation performed by a second user on a button displayed in the virtual space in response to the multiple pieces of action information . A definition means for defining a virtual space including an avatar associated with a first user;
A detection means for detecting a movement of the first user;
a control means for determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and for controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and a determination means for determining an evaluation for the plurality of pieces of motion information.
The control means determines, among the plurality of pieces of motion information, motion information relating to the motion associated with a larger number of the evaluations, with higher priority;
The identification means for identifying the evaluation performs control to identify a comment input by a second user in response to the plurality of pieces of motion information as the evaluation. A definition means for defining a virtual space including an avatar associated with a first user;
A detection means for detecting a movement of the first user;
a control means for determining information on a new action that the avatar can perform based on a plurality of pieces of action information of the avatar, and for controlling the avatar based on the detected movement of the first user and the action information on the determined action;
and a determination means for determining an evaluation for the plurality of pieces of motion information.
The control means determines, among the plurality of pieces of motion information, motion information relating to the motion associated with a larger number of the evaluations, with higher priority;
The system includes a means for identifying an evaluation, the means performing control to identify an object given to the first user by a second user in response to the plurality of pieces of action information as the evaluation for the action.

Images