JP7479219B2 - Program, method, and terminal device - Google Patents

Description

The present invention relates to a program, a method, and a terminal device.

Conventionally, there have been systems for displaying virtual characters controlled by an actor. Some of these systems display decorative objects that are requested to be displayed by a user and are attached to the virtual character.

Japanese Patent No. 6420930

However, in conventional systems, only parts of a virtual character that were requested by the user were decorated according to that request. This meant that the user could not get a sense of reality, for example, when the display mode of a specific part of a virtual character changed spontaneously, and there was room for improvement.

The present invention was conceived in light of this situation, and its purpose is to provide a program, method, and terminal device that can give the user a sense of the reality of a virtual character.

According to one aspect of an embodiment disclosed herein, a program executed in a terminal device having a processor causes the processor to execute the steps of receiving first display data for enabling the display of an image of a virtual space including a first character that moves in conjunction with the movements of an actor, and displaying an image of the first character moving in the virtual space based on the first display data, wherein a predetermined first object is associated with the first character, and the display manner of the first object is changed according to a predetermined rule.

The present invention allows users to feel the reality of the virtual character.

FIG. 1 is a diagram showing an example of an overview of a system according to an embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of a user terminal according to an embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of a server according to an embodiment. 1 is a block diagram showing an example of a hardware configuration of a game playing terminal according to an embodiment. FIG. 2 is a block diagram showing an example of a hardware configuration of a distribution terminal according to an embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of a user terminal, a server, and an HMD set according to an embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of a distribution terminal according to an embodiment. 11 is a flowchart illustrating a part of a process executed in a user terminal and a game playing terminal according to an embodiment. 1A to 1C are diagrams illustrating an example of a virtual space provided to a player and a field of view image viewed by the player according to an embodiment. A figure showing an example of a virtual space provided to a user of a user terminal and a field of view image viewed by the user according to one embodiment. FIG. 11 is a diagram showing another example of a field of view image visually recognized by a user of a user terminal. A figure showing yet another example of a field of view image visually recognized by a user of a user terminal. 1 is a flowchart illustrating a part of a process executed in a game playing terminal according to an embodiment. 10 is a flowchart illustrating a part of a process executed in a user terminal according to an embodiment. 11 is a flowchart illustrating a part of a process executed in a server according to an embodiment. FIG. 2 illustrates an example of a list of users who have joined a game, according to one embodiment. 11 is a flowchart illustrating a part of a process executed in a distribution terminal according to an embodiment. FIG. 13 shows a specific example of a screen displayed on a distribution terminal according to an embodiment. FIG. 13 is a diagram showing another specific example of a screen displayed on a distribution terminal according to an embodiment. FIG. 13 illustrates one example of voice input by a player according to one embodiment. 13A to 13C are diagrams showing yet another specific example of a screen displayed on a distribution terminal according to an embodiment and an overview of distribution of operation instruction data. FIG. 13 illustrates another example of voice input by a player according to one embodiment. 13A to 13C are diagrams showing yet another specific example of a screen displayed on a distribution terminal according to an embodiment and an overview of distribution of operation instruction data. A diagram showing an overview of the transmission of game progress information from a game playing terminal to a user terminal according to an embodiment. 10 is a flowchart illustrating a part of a process executed in a user terminal according to an embodiment. FIG. 13 is a diagram showing a specific example of video playback. FIG. 11 is a diagram showing another specific example of video playback. 13A is a flowchart showing an example of a display mode control process on the game playing terminal side, and FIG. 13B is a flowchart showing an example of a display mode setting process on the user terminal side. (A) is a diagram showing an example of the Nth live streaming video, (B) is a diagram showing an example of the N+1th live streaming video, and (C) is a diagram showing an example of the N+2th live streaming video. FIG. 1A is a diagram showing an example of a live streaming video in normal viewing mode, and FIG. 1B is a diagram showing an example of a live streaming video in AR viewing mode.

The system disclosed herein is a system for providing a game to multiple users. The system will be described below with reference to the drawings. Note that the present invention is not limited to these examples, but is indicated by the claims, and it is intended that all modifications within the meaning and scope equivalent to the claims are included in the present invention. In the following description, the same elements in the description of the drawings will be given the same reference numerals, and redundant description will not be repeated.

<Overview of System 1 Operation>
FIG. 1 is a diagram showing an overview of a system 1 according to this embodiment. The system 1 includes a plurality of user terminals 100 (computers), a server 200, a game play terminal 300 (external device, second external device), and a distribution terminal 400 (external, first external device). Note that in FIG. 1, user terminals 100A to 100C, in other words, three user terminals 100, are shown as an example of the plurality of user terminals 100, but the number of user terminals 100 is not limited to the example shown. In this embodiment, when there is no need to distinguish between the user terminals 100A to C, they are referred to as "user terminals 100". The user terminals 100, the game play terminals 300, and the distribution terminals 400 are connected to the server 200 via a network 2. The network 2 is composed of the Internet and various mobile communication systems constructed by wireless base stations not shown. Examples of such mobile communication systems include so-called 3G and 4G mobile communication systems, LTE (Long Term Evolution), and wireless networks (such as Wi-Fi (registered trademark)) that can be connected to the Internet via a predetermined access point.

(Game Overview)
In this embodiment, a game that is mainly played by a user of a game play terminal 300 will be described as an example of a game provided by the system 1 (hereinafter, this game). Hereinafter, the user of the game play terminal 300 will be referred to as a "player". As an example, the player (actor) progresses through the game by operating a character that appears in the game. In addition, in this game, the user of the user terminal 100 plays a role of supporting the player's progress through the game. Details of this game will be described later. Note that the game provided by the system 1 may be a game in which multiple users participate, and is not limited to this example.

(Game Play Terminal 300)
The game play terminal 300 progresses the game in response to input operations by the player. Furthermore, the game play terminal 300 sequentially distributes information generated by the player's game play (hereinafter, game progress information) to the server 200 in real time.

(Server 200)
The server 200 transmits game progress information (second data) received in real time from the game playing terminal 300 to the user terminal 100. The server 200 also mediates the transmission and reception of various types of information between the user terminal 100, the game playing terminal 300, and the distribution terminal 400.

(Distribution terminal 400)
The distribution terminal 400 generates action instruction data (first data) in response to an input operation by a user of the distribution terminal 400, and distributes the action instruction data to the user terminal 100 via the server 200. The action instruction data is data for playing a video on the user terminal 100, and specifically, data for causing a character appearing in the video to perform a motion.

In this embodiment, as an example, the user of the distribution terminal 400 is a player of the game. Also, as an example, the video played on the user terminal 100 based on the action instruction data is a video of a character operated by the player in the game moving. "Movement" refers to moving at least a part of the character's body, and includes speaking. Therefore, the action instruction data in this embodiment includes, for example, voice data for making the character speak, and motion data for moving the character's body.

As an example, the action instruction data is transmitted to the user terminal 100 after the game ends. Details of the action instruction data and the video that is played based on the action instruction data will be described later.

(User terminal 100)
The user terminal 100 receives game progress information in real time, and generates and displays a game screen using the information. In other words, the user terminal 100 reproduces the game screen of the game being played by the player by real-time rendering. This allows the user of the user terminal 100 to view the same game screen as the player views while playing the game, at approximately the same timing as the player.

The user terminal 100 also generates information to assist the player in progressing through the game in response to input operations by the user, and transmits the information to the game play terminal 300 via the server 200. Details of the information will be described later.

The user terminal 100 also receives action instruction data from the distribution terminal 400, and generates and plays a video (image) using the action instruction data. In other words, the user terminal 100 renders and plays the action instruction data.

<Hardware configuration of System 1>
Fig. 2 is a diagram showing the hardware configuration of the user terminal 100. Fig. 3 is a diagram showing the hardware configuration of the server 200. Fig. 4 is a diagram showing the hardware configuration of the game playing terminal 300. Fig. 5 is a diagram showing the hardware configuration of the distribution terminal 400.

(User terminal 100)
In this embodiment, as an example, the user terminal 100 is realized as a smartphone, but the user terminal 100 is not limited to a smartphone. For example, the user terminal 100 may be realized as a feature phone, a tablet computer, a laptop computer (a so-called notebook computer), or a desktop computer. The user terminal 100 may also be a game device suitable for playing games.

As shown in FIG. 2, the user terminal 100 includes a processor 10, a memory 11, a storage 12, a communication interface (IF) 13, an input/output IF 14, a touch screen 15 (display unit), a camera 17, and a distance sensor 18. These components of the user terminal 100 are electrically connected to each other via a communication bus. Note that the user terminal 100 may include an input/output IF 14 to which a display (display unit) configured separately from the user terminal 100 main body can be connected instead of or in addition to the touch screen 15.

Also, as shown in FIG. 2, the user terminal 100 may be configured to be able to communicate with one or more controllers 1020. The controller 1020 establishes communication with the user terminal 100 according to a communication standard such as Bluetooth (registered trademark). The controller 1020 may have one or more buttons, etc., and transmits output values based on user input operations on the buttons, etc., to the user terminal 100. The controller 1020 may also have various sensors such as an acceleration sensor and an angular velocity sensor, and transmits output values of the various sensors to the user terminal 100.

In addition, instead of or in addition to the user terminal 100 being equipped with the camera 17 and distance measurement sensor 18, the controller 1020 may have the camera 17 and distance measurement sensor 18.

For example, when starting a game, the user terminal 100 desirably has the user using the controller 1020 input user identification information, such as the user's name or login ID, via the controller 1020. This enables the user terminal 100 to link the controller 1020 with the user, and to identify which user the output value belongs to based on the source (controller 1020) of the received output value.

When the user terminal 100 communicates with multiple controllers 1020, each user can hold a controller 1020 and realize a multi-play with the single user terminal 100 without communicating with other devices such as the server 200 via the network 2. In addition, the user terminals 100 can realize a multi-play locally by connecting to each other via a wireless standard such as a wireless LAN (Local Area Network) standard (connecting to communicate without going through the server 200). When the above-mentioned multi-play is realized locally by a single user terminal 100, the user terminal 100 may further include at least some of the various functions of the server 200, which will be described later. In addition, when the above-mentioned multi-play is realized locally by multiple user terminals 100, the multiple user terminals 100 may have various functions of the server 200, which will be described later, distributed among them.

Note that even when the above-mentioned multiplay is realized locally, the user terminal 100 may communicate with the server 200. For example, information indicating the results of a game, such as the score or win/loss, may be associated with user identification information and transmitted to the server 200.

The controller 1020 may also be configured to be detachable from the user terminal 100. In this case, a connection part for the controller 1020 may be provided on at least one of the faces of the housing of the user terminal 100. When the user terminal 100 and the controller 1020 are connected by a wire via the connection part, the user terminal 100 and the controller 1020 transmit and receive signals via the wire.

As shown in FIG. 2, the user terminal 100 may accept the attachment of a storage medium 1030 such as an external memory card via the input/output IF 14. This allows the user terminal 100 to read programs and data recorded on the storage medium 1030. The program recorded on the storage medium 1030 is, for example, a game program.

The user terminal 100 may store in the memory 11 of the user terminal 100 a game program acquired by communicating with an external device such as the server 200, or may store in the memory 11 a game program acquired by reading it from the storage medium 1030.

As described above, the user terminal 100 includes a communication IF 13, an input/output IF 14, a touch screen 15, a camera 17, and a distance sensor 18 as examples of mechanisms for inputting information to the user terminal 100. Each of the above-mentioned components as input mechanisms can be considered as an operation unit configured to accept input operations by the user.

For example, when the operation unit is composed of at least one of the camera 17 and the distance measurement sensor 18, the operation unit detects an object 1010 in the vicinity of the user terminal 100 and identifies an input operation from the detection result of the object. As an example, a user's hand or a marker of a predetermined shape is detected as the object 1010, and the input operation is identified based on the color, shape, movement, or type of the object 1010 obtained as the detection result. More specifically, when the user's hand is detected from an image captured by the camera 17, the user terminal 100 identifies and accepts a gesture (a series of movements of the user's hand) detected based on the captured image as the user's input operation. Note that the captured image may be a still image or a video.

Alternatively, if the operation unit is configured with a touch screen 15, the user terminal 100 identifies and accepts a user operation performed on the input unit 151 of the touch screen 15 as a user input operation. Alternatively, if the operation unit is configured with a communication IF 13, the user terminal 100 identifies and accepts a signal (e.g., an output value) transmitted from the controller 1020 as a user input operation. Alternatively, if the operation unit is configured with an input/output IF 14, the user terminal 100 identifies and accepts a signal output from an input device (not shown) connected to the input/output IF 14 and different from the controller 1020 as a user input operation.

(Server 200)
The server 200 may be, for example, a general-purpose computer such as a workstation or a personal computer. The server 200 includes a processor 20, a memory 21, a storage 22, a communication IF 23, and an input/output IF 24. These components of the server 200 are electrically connected to each other by a communication bus.

(Game Play Terminal 300)
The game play terminal 300 may be, for example, a general-purpose computer such as a personal computer. The game play terminal 300 comprises a processor 30, a memory 31, a storage 32, a communication IF 33, and an input/output IF 34. These components of the game play terminal 300 are electrically connected to each other by a communication bus.

As shown in FIG. 4, the game play terminal 300 according to this embodiment is, for example, included in an HMD (Head Mounted Display) set 1000. In other words, it can be expressed that the HMD set 1000 is included in the system 1, and it can also be expressed that the player plays the game using the HMD set 1000. Note that the device on which the player plays the game is not limited to the HMD set 1000. For example, the device may be any device that allows the player to virtually experience the game. The device may also be realized as a smartphone, a feature phone, a tablet computer, a laptop computer (a so-called notebook computer), or a desktop computer. The device may also be a game device suitable for game play.

The HMD set 1000 includes, in addition to the game play terminal 300, an HMD 500, an HMD sensor 510, a motion sensor 520, a display 530, and a controller 540. The HMD 500 includes a monitor 51, a gaze sensor 52, a first camera 53, a second camera 54, a microphone 55, and a speaker 56. The controller 540 may include the motion sensor 520.

The HMD 500 is worn on the player's head and can provide the player with a virtual space during operation. More specifically, the HMD 500 displays an image for the right eye and an image for the left eye on the monitor 51. When each eye of the player views the respective image, the player can recognize the image as a three-dimensional image based on the parallax between the two eyes. The HMD 500 can include both a so-called head-mounted display equipped with a monitor, and a head-mounted device to which a smartphone or other terminal equipped with a monitor can be attached.

Monitor 51 is realized, for example, as a non-transparent display device. In one aspect, monitor 51 is disposed on the main body of HMD 500 so as to be positioned in front of both eyes of the player. Therefore, when the player visually recognizes the three-dimensional image displayed on monitor 51, he or she can become immersed in the virtual space. In one aspect, the virtual space includes, for example, images of a background, objects that the player can control, and menus that the player can select. In one aspect, monitor 51 can be realized as a liquid crystal monitor or an organic EL (Electro Luminescence) monitor that is provided on a so-called smartphone or other information display terminal.

In another aspect, the monitor 51 may be realized as a transmissive display device. In this case, the HMD 500 may be an open type such as glasses, rather than a closed type that covers the player's eyes as shown in FIG. 1. The transmissive monitor 51 may be temporarily configurable as a non-transmissive display device by adjusting its transmittance. The monitor 51 may include a configuration that simultaneously displays a portion of an image that constitutes the virtual space and the real space. For example, the monitor 51 may display an image of the real space captured by a camera mounted on the HMD 500, or the transmittance of a portion of the monitor may be set high to make the real space visible.

In one aspect, the monitor 51 may include a sub-monitor for displaying an image for the right eye and a sub-monitor for displaying an image for the left eye. In another aspect, the monitor 51 may be configured to display an image for the right eye and an image for the left eye as a single unit. In this case, the monitor 51 includes a high-speed shutter. The high-speed shutter operates to alternately display an image for the right eye and an image for the left eye so that the image is recognized by only one of the eyes.

In one aspect, the HMD 500 includes multiple light sources (not shown). Each light source is realized, for example, by an LED (Light Emitting Diode) that emits infrared rays. The HMD sensor 510 has a position tracking function for detecting the movement of the HMD 500. More specifically, the HMD sensor 510 reads multiple infrared rays emitted by the HMD 500 and detects the position and tilt of the HMD 500 in real space.

In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and inclination of the HMD 500 by performing image analysis processing using image information of the HMD 500 output from the camera.

In another aspect, the HMD 500 may include a sensor (not shown) as a position detector instead of or in addition to the HMD sensor 510. The HMD 500 may use the sensor to detect the position and inclination of the HMD 500 itself. For example, if the sensor is an angular velocity sensor, a geomagnetic sensor, or an acceleration sensor, the HMD 500 may use any of these sensors instead of the HMD sensor 510 to detect the position and inclination of the HMD 500 itself. As an example, if the sensor included in the HMD 500 is an angular velocity sensor, the angular velocity sensor detects the angular velocity of the HMD 500 around three axes in real space over time. The HMD 500 calculates the change in angle of the HMD 500 around the three axes over time based on each angular velocity, and further calculates the inclination of the HMD 500 based on the change in angle over time.

The gaze sensor 52 detects the direction in which the gaze of the player's right eye and left eye is directed. In other words, the gaze sensor 52 detects the player's gaze. Detection of the gaze direction is achieved, for example, by a known eye tracking function. The gaze sensor 52 is achieved by a sensor having the eye tracking function. In one aspect, the gaze sensor 52 preferably includes a sensor for the right eye and a sensor for the left eye. The gaze sensor 52 may be, for example, a sensor that irradiates the player's right eye and left eye with infrared light and detects the rotation angle of each eyeball by receiving reflected light from the cornea and iris in response to the irradiated light. The gaze sensor 52 can detect the player's gaze based on each detected rotation angle.

The first camera 53 photographs the lower part of the player's face. More specifically, the first camera 53 photographs the player's nose, mouth, etc. The second camera 54 photographs the player's eyes, eyebrows, etc. The housing of the HMD 500 on the player's side is defined as the inside of the HMD 500, and the housing of the HMD 500 on the opposite side from the player is defined as the outside of the HMD 500. In one aspect, the first camera 53 may be disposed on the outside of the HMD 500, and the second camera 54 may be disposed on the inside of the HMD 500. Images generated by the first camera 53 and the second camera 54 are input to the game play terminal 300. In another aspect, the first camera 53 and the second camera 54 may be realized as a single camera, and the player's face may be photographed by this single camera.

The microphone 55 converts the player's speech into an audio signal (electrical signal) and outputs it to the game play terminal 300. The speaker 56 converts the audio signal into sound and outputs it to the player. In another aspect, the HMD 500 may include earphones instead of the speaker 56.

The controller 540 is connected to the game play terminal 300 by wire or wirelessly. The controller 540 accepts input of commands from the player to the game play terminal 300. In one aspect, the controller 540 is configured to be held by the player. In another aspect, the controller 540 is configured to be attached to a part of the player's body or clothing. In yet another aspect, the controller 540 may be configured to output at least one of vibration, sound, and light based on a signal transmitted from the game play terminal 300. In yet another aspect, the controller 540 accepts operations from the player to control the position and movement of an object placed in a virtual space.

In one aspect, the controller 540 includes multiple light sources. Each light source is realized, for example, by an LED that emits infrared light. The HMD sensor 510 has a position tracking function. In this case, the HMD sensor 510 reads multiple infrared rays emitted by the controller 540 and detects the position and tilt of the controller 540 in real space. In another aspect, the HMD sensor 510 may be realized by a camera. In this case, the HMD sensor 510 can detect the position and tilt of the controller 540 by performing image analysis processing using image information of the controller 540 output from the camera.

In one aspect, the motion sensor 520 is attached to the player's hand and detects the movement of the player's hand. For example, the motion sensor 520 detects the rotation speed, number of rotations, etc. of the hand. The detected signal is sent to the game play terminal 300. The motion sensor 520 is provided, for example, in the controller 540. In one aspect, the motion sensor 520 is provided, for example, in the controller 540 configured to be held by the player. In another aspect, for safety in the real space, the controller 540 is attached to something that is not easily blown away by being worn on the player's hand, such as a glove type. In yet another aspect, a sensor that is not worn by the player may detect the movement of the player's hand. For example, a signal from a camera that captures the player may be input to the game play terminal 300 as a signal representing the player's action. The motion sensor 520 and the game play terminal 300 are connected to each other wirelessly, as an example. In the case of wireless communication, the communication form is not particularly limited, and for example, Bluetooth or other known communication methods are used.

The display 530 displays an image similar to the image displayed on the monitor 51. This allows users other than the player wearing the HMD 500 to view the same image as the player. The image displayed on the display 530 does not need to be a three-dimensional image, and may be an image for the right eye or an image for the left eye. Examples of the display 530 include a liquid crystal display and an organic EL monitor.

The game play terminal 300 moves the character operated by the player based on various information acquired from each part of the HMD 500, the controller 540, and the motion sensor 520, and progresses the game. Here, "movement" includes moving each part of the body, changing posture, changing facial expression, moving, speaking, touching or moving objects placed in the virtual space, and using weapons, tools, etc. held by the character. That is, in this game, when the player moves each part of his body, the character also moves each part of his body in the same way as the player. Also, in this game, the character speaks the contents of what the player speaks. In other words, in this game, the character is an avatar object that acts as an alter ego of the player. As an example, at least a part of the character's movements may be executed by input to the controller 540 by the player.

In this embodiment, the motion sensors 520 are attached to the player's hands, both feet, waist, and head, as an example. The motion sensors 520 attached to the player's hands may be provided on the controller 540, as described above. The motion sensor 520 attached to the player's head may be provided on the HMD 500. The motion sensors 520 may also be attached to the user's elbows and knees. By increasing the number of motion sensors 520 attached to the player, the player's movements can be more accurately reflected in the character. Also, instead of attaching motion sensors 520 to each part of the body, the player may wear a suit to which one or more motion sensors 520 are attached. In other words, the method of motion capture is not limited to the example in which the motion sensors 520 are used.

(Distribution terminal 400)
The distribution terminal 400 may be a mobile terminal such as a smartphone, a personal digital assistant (PDA), or a tablet computer, or may be a so-called stationary terminal such as a desktop personal computer.

As shown in FIG. 5, the distribution terminal 400 includes a processor 40, a memory 41, a storage 42, a communication IF 43, an input/output IF 44, and a touch screen 45. Note that instead of or in addition to the touch screen 45, the distribution terminal 400 may include an input/output IF 44 to which a display (display unit) configured separately from the distribution terminal 400 main body can be connected.

The controller 1021 may have a physical input mechanism such as one or more buttons, levers, sticks, wheels, etc. The controller 1021 transmits to the streaming terminal 400 an output value based on an input operation input to the input mechanism by an operator of the streaming terminal 400 (a player in this embodiment). The controller 1021 may also have various sensors such as an acceleration sensor and an angular velocity sensor, and may transmit the output values of the various sensors to the streaming terminal 400. The above-mentioned output values are received by the streaming terminal 400 via the communication IF 43.

The distribution terminal 400 may be equipped with a camera and a distance measurement sensor (both not shown). Instead of or in addition to the distribution terminal 400 being equipped with a camera and a distance measurement sensor, the controller 1021 may have the camera and distance measurement sensor.

As described above, the distribution terminal 400 includes a communication IF 43, an input/output IF 44, and a touch screen 45 as an example of a mechanism for inputting information to the distribution terminal 400. Each of the above-mentioned units as an input mechanism can be considered as an operation unit configured to accept input operations by the user.

When the operation unit is configured with a touch screen 45, the distribution terminal 400 identifies and accepts a user operation performed on the input unit 451 of the touch screen 45 as a user input operation. Alternatively, when the operation unit is configured with a communication IF 43, the distribution terminal 400 identifies and accepts a signal (e.g., an output value) transmitted from the controller 1021 as a user input operation. Alternatively, when the operation unit is configured with an input/output IF 44, the distribution terminal 400 identifies and accepts a signal output from an input device (not shown) connected to the input/output IF 44 as a user input operation.

<Hardware components of each device>
Processors 10, 20, 30, and 40 control the overall operation of user terminal 100, server 200, game play terminal 300, and distribution terminal 400, respectively. Processors 10, 20, 30, and 40 include a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and a GPU (Graphics Processing Unit). Processors 10, 20, 30, and 40 read programs from storages 12, 22, 32, and 42, respectively, which will be described later. Then, processors 10, 20, 30, and 40 load the read programs into memories 11, 21, 31, and 41, which will be described later, respectively. Processors 10, 20, and 30 execute the loaded programs.

Memories 11, 21, 31, and 41 are main storage devices. Memories 11, 21, 31, and 41 are composed of storage devices such as ROM (Read Only Memory) and RAM (Random Access Memory). Memory 11 provides a working area for processor 10 by temporarily storing programs and various data read by processor 10 from storage 12 described later. Memory 11 also temporarily stores various data generated while processor 10 is operating according to the program. Memory 21 provides a working area for processor 20 by temporarily storing various programs and data read by processor 20 from storage 22 described later. Memory 21 also temporarily stores various data generated while processor 20 is operating according to the program. Memory 31 provides a working area for processor 30 by temporarily storing various programs and data read by processor 30 from storage 32 described later. Memory 31 also temporarily stores various data generated while processor 30 is operating according to the program. The memory 41 provides a working area for the processor 40 by temporarily storing the programs and various data that the processor 40 reads from the storage 42 (described later). The memory 41 also temporarily stores various data that the processor 40 generates while it is operating according to the programs.

In this embodiment, the program executed by processors 10 and 30 may be a game program for the game. In this embodiment, the program executed by processor 40 may be a distribution program for realizing the distribution of action instruction data. Furthermore, processor 10 may further execute a viewing program for realizing the playback of video.

In this embodiment, the program executed by the processor 20 may be at least one of the game program, distribution program, and viewing program described above. The processor 20 executes at least one of the game program, distribution program, and viewing program in response to a request from at least one of the user terminal 100, the game play terminal 300, and the distribution terminal 400. Note that the distribution program and the viewing program may be executed in parallel.

In other words, the game program may be a program that realizes a game through cooperation between the user terminal 100, the server 200, and the game playing terminal 300. The distribution program may be a program that realizes the distribution of action instruction data through cooperation between the server 200 and the distribution terminal 400. The viewing program may be a program that realizes the playback of a video through cooperation between the user terminal 100 and the server 200.

Storages 12, 22, 32, and 42 are auxiliary storage devices. Storages 12, 22, 32, and 42 are configured with storage devices such as flash memory or HDDs (Hard Disk Drives). Storages 12 and 32 store, for example, various data related to games. Storage 42 stores various data related to the distribution of action instruction data. Storage 12 stores various data related to the playback of videos. Storage 22 may store at least some of the various data related to games, the distribution of action instruction data, and the playback of videos.

The communication IFs 13, 23, 33, and 43 respectively control the transmission and reception of various data in the user terminal 100, the server 200, the game play terminal 300, and the distribution terminal 400. The communication IFs 13, 23, 33, and 43 control, for example, communication via a wireless LAN (Local Area Network), internet communication via a wired LAN, wireless LAN, or mobile phone network, and communication using short-range wireless communication, etc.

The input/output IFs 14, 24, 34, and 44 are interfaces through which the user terminal 100, the server 200, the game playing terminal 300, and the distribution terminal 400 accept data input and output data, respectively. The input/output IFs 14, 24, 34, and 44 may input and output data via a USB (Universal Serial Bus) or the like. The input/output IFs 14, 24, 34, and 44 may include physical buttons, a camera, a microphone, a speaker, a mouse, a keyboard, a display, a stick, a lever, and the like. The input/output IFs 14, 24, 34, and 44 may also include a connection section for transmitting and receiving data to and from peripheral devices.

Touch screen 15 is an electronic component that combines input unit 151 and display unit 152 (display). Touch screen 45 is an electronic component that combines input unit 451 and display unit 452. Input units 151 and 451 are, for example, touch-sensitive devices, and are configured, for example, by a touchpad. Display units 152 and 452 are, for example, configured by a liquid crystal display or an organic EL (Electro-Luminescence) display.

The input units 151 and 451 have a function of detecting the position where a user operation (mainly a physical contact operation such as a touch operation, a slide operation, a swipe operation, or a tap operation) is input on the input surface, and transmitting information indicating the position as an input signal. The input units 151 and 451 may be provided with a touch sensing unit (not shown). The touch sensing unit may be of any type, such as a capacitive type or a resistive film type.

Although not shown, the user terminal 100 and the distribution terminal 400 may each include one or more sensors for identifying the holding posture of the user terminal 100 and the distribution terminal 400. The sensor may be, for example, an acceleration sensor or an angular velocity sensor.

If the user terminal 100 and the distribution terminal 400 are equipped with sensors, the processors 10 and 40 can respectively identify the holding orientation of the user terminal 100 and the distribution terminal 400 from the output of the sensor and perform processing according to the holding orientation. For example, when the user terminal 100 and the distribution terminal 400 are held vertically, the processors 10 and 40 can respectively set the display units 152 and 452 to display a vertically long image in a vertical screen display. On the other hand, when the user terminal 100 and the distribution terminal 400 are held horizontally, the processors 10 and 40 can respectively set the display units 152 and 452 to display a horizontally long image in a horizontal screen display. In this way, the processors 10 and 40 can respectively switch between a vertical screen display and a horizontal screen display depending on the holding orientation of the user terminal 100 and the distribution terminal 400.

<Functional Configuration of System 1>
Fig. 6 is a block diagram showing the functional configuration of the user terminal 100, the server 200, and the HMD set 1000 included in the system 1. Fig. 7 is a block diagram showing the functional configuration of the distribution terminal 400 shown in Fig. 6.

The user terminal 100 functions as an input device that accepts input operations from the user, and as an output device that outputs game images and sounds. The user terminal 100 functions as a control unit 110 and a storage unit 120 through cooperation of the processor 10, memory 11, storage 12, communication IF 13, input/output IF 14, touch screen 15, etc.

The server 200 has a function of mediating the transmission and reception of various information between the user terminal 100, the HMD set 1000, and the distribution terminal 400. The server 200 functions as a control unit 210 and a storage unit 220 through cooperation of the processor 20, memory 21, storage 22, communication IF 23, input/output IF 24, etc.

The HMD set 1000 (game playing terminal 300) functions as an input device that accepts input operations from the player, as an output device that outputs game images and sounds, and as a function that transmits game progress information in real time to the user terminal 100 via the server 200. The HMD set 1000 functions as a control unit 310 and a memory unit 320 through cooperation of the processor 30, memory 31, storage 32, communication IF 33, input/output IF 34, HMD 500, HMD sensor 510, motion sensor 520, controller 540, etc. of the game playing terminal 300.

The distribution terminal 400 has a function of generating action instruction data and transmitting the action instruction data to the user terminal 100 via the server 200. The distribution terminal 400 functions as a control unit 410 and a storage unit 420 through cooperation of the processor 40, memory 41, storage 42, communication IF 43, input/output IF 44, touch screen 45, etc.

(Data stored in the memory unit of each device)
The storage unit 120 stores a game program 131 (program), game information 132, and user information 133. The storage unit 220 stores a game program 231, game information 232, user information 233, and a user list 234. The storage unit 320 stores a game program 331, game information 332, and user information 333. The storage unit 420 stores a user list 421, a motion list 422, and a delivery program 423 (program, second program).

The game programs 131, 231, and 331 are game programs executed by the user terminal 100, the server 200, and the HMD set 1000, respectively. The game is realized by each device operating in cooperation based on the game programs 131, 231, and 331. The game programs 131 and 331 may be stored in the storage unit 220 and downloaded to the user terminal 100 and the HMD set 1000, respectively. In this embodiment, the user terminal 100 renders data received from the distribution terminal 400 based on the game program 131 and plays a video. In other words, the game program 131 is also a program for playing a video using video instruction data distributed from the distribution terminal 400. The program for playing the video may be different from the game program 131. In this case, the storage unit 120 stores the program for playing the video separately from the game program 131.

Game information 132, 232, 332 is data that the user terminal 100, server 200, and HMD set 1000 refer to when executing a game program. User information 133, 233, 333 is data related to the account of the user of the user terminal 100. Game information 232 is game information 132 of each user terminal 100 and game information 332 of the HMD set 1000. User information 233 is user information 133 of each user terminal 100 and user information of the player included in user information 333. User information 333 is user information 133 of each user terminal 100 and user information of the player.

User list 234 and user list 421 are lists of users who have participated in the game. User list 234 and user list 421 may include a list of users who have participated in the most recent game play by the player, as well as a list of users who have participated in each game play prior to that game play. Motion list 422 is a list of multiple motion data created in advance. Motion list 422 is, for example, a list in which motion data is associated with information identifying each motion (for example, a motion name). Distribution program 423 is a program for realizing the distribution of action instruction data for playing a video on user terminal 100 to user terminal 100.

(Functional configuration of server 200)
The control unit 210 executes a game program 231 stored in the storage unit 220 to comprehensively control the server 200. For example, the control unit 210 mediates the transmission and reception of various types of information between the user terminal 100, the HMD set 1000, and the distribution terminal 400.

The control unit 210 functions as a communication intermediation unit 211, a log generation unit 212, and a list generation unit 213 according to the description of the game program 231. The control unit 210 can also function as other functional blocks (not shown) to mediate the sending and receiving of various information related to game play and distribution of action instruction data, and to support the progress of the game.

The communication intermediation unit 211 mediates the transmission and reception of various information between the user terminal 100, the HMD set 1000, and the distribution terminal 400. For example, the communication intermediation unit 211 transmits game progress information received from the HMD set 1000 to the user terminal 100. The game progress information includes data indicating the movement of the character operated by the player, the parameters of the character, items and weapons held by the character, enemy characters, and other information. The server 200 transmits the game progress information to the user terminals 100 of all users participating in the game. In other words, the server 200 transmits common game progress information to the user terminals 100 of all users participating in the game. As a result, the game progresses in the user terminals 100 of all users participating in the game in the same way as in the HMD set 1000.

Furthermore, for example, the communication intermediation unit 211 transmits information received from one of the user terminals 100 to assist the player in progressing through the game to the other user terminals 100 and the HMD set 1000. As an example, the information may be item information indicating an item that allows the player to progress through the game advantageously and is provided to the player (character). The item information includes information indicating the user who provided the item (user name, user ID, etc.). Furthermore, the communication intermediation unit 211 may mediate the distribution of action instruction data from the distribution terminal 400 to the user terminal 100.

The log generating unit 212 generates a log of the game progress based on the game progress information received from the HMD set 1000. The list generating unit 213 generates a user list 234 after the game play ends. Although details will be described later, each user in the user list 234 is associated with a tag indicating the content of the support provided by that user to the player. The list generating unit 213 generates a tag based on the game progress log generated by the log generating unit 212 and associates it with the corresponding user. Note that the list generating unit 213 may associate the content of the support provided by each user to the player, which is input by the game administrator or the like using a terminal device such as a personal computer, with the corresponding user as a tag. This makes the content of the support provided by each user more detailed. Note that when a user participates in a game, the user terminal 100 transmits information indicating the user to the server 200 based on the user's operation. For example, the user terminal 100 transmits the user ID input by the user to the server 200. In other words, the server 200 holds information indicating each user for all users participating in the game. The list generation unit 213 can use this information to generate the user list 234.

(Functional configuration of the HMD set 1000)
The control unit 310 executes the game program 331 stored in the storage unit 320 to comprehensively control the HMD set 1000. For example, the control unit 310 progresses the game in accordance with the game program 331 and the player's operation. Furthermore, while the game is progressing, the control unit 310 communicates with the server 200 as necessary to transmit and receive information. The control unit 310 may transmit and receive information directly to and from the user terminal 100 without going through the server 200.

The control unit 310 functions as an operation reception unit 311, a display control unit 312, a UI control unit 313, an animation generation unit 314, a game progression unit 315, a virtual space control unit 316, and a reaction processing unit 317 according to the description of the game program 331. Depending on the nature of the game being executed, the control unit 310 can also function as other functional blocks (not shown) for the purpose of controlling characters appearing in the game, etc.

The operation reception unit 311 detects and accepts input operations by the player. The operation reception unit 311 accepts signals input from the HMD 500, motion sensor 520, controller 540, etc., determines what type of input operation has been performed, and outputs the results to each element of the control unit 310.

The UI control unit 313 controls user interface (hereinafter, UI) images displayed on the monitor 51, display 530, etc. The UI images are tools that allow the player to input to the HMD set 1000 the information required for the progression of the game, or to obtain information from the HMD set 1000 that is output during the progression of the game. Examples of UI images include, but are not limited to, icons, buttons, lists, menu screens, etc.

The animation generation unit 314 generates animations showing the motion of various objects based on the control modes of the various objects. For example, the animation generation unit 314 may generate animations that show an object (e.g., a player's avatar object) moving as if it were actually there, moving its mouth, changing its facial expression, etc.

The game progression unit 315 progresses the game based on the game program 331, input operations by the player, and the actions of the avatar object in response to the input operations. For example, the game progression unit 315 performs a predetermined game process when the avatar object performs a predetermined action. For example, the game progression unit 315 may receive information representing an operation of a user on the user terminal 100, and perform game processing based on the user's operation. The game progression unit 315 also generates game progression information according to the progress of the game and transmits it to the server 200. The game progress information is transmitted to the user terminal 100 via the server 200. As a result, the progress of the game on the HMD set 1000 is shared on the user terminal 100. In other words, the progress of the game on the HMD set 1000 and the progress of the game on the user terminal 100 are synchronized.

The virtual space control unit 316 performs various controls related to the virtual space provided to the player in accordance with the progress of the game. As an example, the virtual space control unit 316 generates various objects and places them in the virtual space. The virtual space control unit 316 also places a virtual camera in the virtual space. The virtual space control unit 316 also operates the various objects placed in the virtual space in accordance with the progress of the game. The virtual space control unit 316 also controls the position and tilt of the virtual camera placed in the virtual space in accordance with the progress of the game.

The display control unit 312 outputs to the monitor 51 and the display 530 a game screen that reflects the results of the processing performed by each of the above-mentioned elements. The display control unit 312 may display an image based on the field of view from a virtual camera placed in a virtual space as a game screen on the monitor 51 and the display 530. The display control unit 312 may also include an animation generated by the animation generation unit 314 in the game screen. The display control unit 312 may also draw the above-mentioned UI image controlled by the UI control unit 313 by superimposing it on the game screen.

The reaction processing unit 317 accepts feedback from the user of the user terminal 100 regarding the reaction of the player to the gameplay of the player, and outputs this to the player. In this embodiment, for example, the user terminal 100 can create a comment (message) addressed to an avatar object based on an input operation by the user. The reaction processing unit 317 accepts comment data of the comment and outputs it. The reaction processing unit 317 may display text data corresponding to the user's comment on the monitor 51 or the display 530, or may output audio data corresponding to the user's comment from a speaker (not shown). In the former case, the reaction processing unit 317 may draw an image corresponding to the text data (i.e., an image including the content of the comment) superimposed on the game screen.

(Functional configuration of user terminal 100)
The control unit 110 executes the game program 131 stored in the storage unit 120 to comprehensively control the user terminal 100. For example, the control unit 110 progresses the game in accordance with the game program 131 and the user's operation. Furthermore, while the game is progressing, the control unit 110 communicates with the server 200 as necessary to transmit and receive information. The control unit 110 may transmit and receive information directly to the HMD set 1000 without going through the server 200.

The control unit 110 functions as an operation reception unit 111, a display control unit 112, a UI control unit 113, an animation generation unit 114, a game progression unit 115, a virtual space control unit 116, and a video playback unit 117 according to the description of the game program 131. The control unit 110 can also function as other functional blocks (not shown) for the progression of the game according to the nature of the game being executed.

The operation reception unit 111 detects and accepts user input operations on the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the actions performed by the user on the console via the touch screen 15 and other input/output IF 14, and outputs the results to each element of the control unit 110.

For example, the operation reception unit 111 receives an input operation on the input unit 151, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation reception unit 111 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc. as the type of input operation. Furthermore, when a continuously detected input is interrupted, the operation reception unit 111 detects that the contact input from the touch screen 15 has been released.

The UI control unit 113 controls UI images to be displayed on the display unit 152 to construct a UI in response to at least one of the user's input operations and the received game progress information. The UI images are tools that allow the user to input to the user terminal 100 the information required for the progress of the game, or tools that allow the user to obtain information from the user terminal 100 that is output during the progress of the game. Examples of UI images include, but are not limited to, icons, buttons, lists, and menu screens.

The animation generation unit 114 generates animations showing the motion of various objects based on the control modes of the various objects.

The game progression unit 115 progresses the game based on the game program 131, the received game progress information, and input operations by the user. When the game progression unit 115 performs a predetermined game process based on an input operation by the user, it transmits information related to the game process to the HMD set 1000 via the server 200. This causes the predetermined game process to be shared in the HMD set 1000. In other words, the progress of the game in the HMD set 1000 and the progress of the game in the user terminal 100 are synchronized. The predetermined game process is, for example, a process of providing an item to an avatar object, and in this example, the information related to the game process is the item information described above.

The virtual space control unit 116 performs various controls related to the virtual space provided to the user according to the progress of the game. As an example, the virtual space control unit 116 generates various objects and places them in the virtual space. The virtual space control unit 116 also places a virtual camera in the virtual space. The virtual space control unit 116 also operates the various objects placed in the virtual space according to the progress of the game, specifically, according to the received game progress information. The virtual space control unit 316 also controls the position and tilt of the virtual camera placed in the virtual space according to the progress of the game, specifically, according to the received game progress information.

The display control unit 112 outputs to the display unit 152 a game screen reflecting the results of the processing performed by each of the above-mentioned elements. The display control unit 112 may display, as a game screen, an image based on the field of view from a virtual camera placed in a virtual space provided to the user on the display unit 152. The display control unit 112 may also include an animation generated by the animation generation unit 114 in the game screen. The display control unit 112 may also draw the above-mentioned UI image controlled by the UI control unit 113 by superimposing it on the game screen. In any case, the game screen displayed on the display unit 152 is the same game screen as the game screen displayed on the other user terminals 100 and the HMD set 1000.

The video playback unit 117 analyzes (renders) the action instruction data received from the distribution terminal 400 and plays the video.

(Functional configuration of distribution terminal 400)
The control unit 410 performs overall control of the distribution terminal 400 by executing a program (not shown) stored in the storage unit 420. For example, the control unit 410 generates action instruction data according to the program and an operation by a user (a player in this embodiment) of the distribution terminal 400, and distributes the action instruction data to the user terminal 100. Furthermore, the control unit 410 communicates with the server 200 as necessary to transmit and receive information. The control unit 410 may transmit and receive information directly to and from the user terminal 100 without going through the server 200.

The control unit 410 functions as a communication control unit 411, a display control unit 412, an operation reception unit 413, a voice reception unit 414, a motion identification unit 415, and an action instruction data generation unit 416 according to the program description. The control unit 410 can also function as other functional blocks (not shown) for generating and distributing action instruction data.

The communication control unit 411 controls the transmission and reception of information to and from the server 200 or the user terminal 100 via the server 200. As an example, the communication control unit 411 receives a user list 421 from the server 200. Also, as an example, the communication control unit 411 transmits operation instruction data to the user terminal 100.

The display control unit 412 outputs to the display unit 452 various screens reflecting the results of the processing executed by each element. As an example, the display control unit 412 displays a screen including the received user list 234. As an example, the display control unit 412 also displays a screen including a motion list 422 for allowing the player to select motion data for moving the avatar object, which is included in the action instruction data to be distributed.

The operation reception unit 413 detects and accepts input operations by the player to the input unit 151. The operation reception unit 111 determines what kind of input operation has been performed from the action performed by the player on the touch screen 45 and the console via other input/output IF 44, and outputs the result to each element of the control unit 410.

For example, the operation reception unit 413 receives an input operation on the input unit 451, detects the coordinates of the input position of the input operation, and identifies the type of the input operation. The operation reception unit 413 identifies, for example, a touch operation, a slide operation, a swipe operation, a tap operation, etc. as the type of input operation. Furthermore, when a continuously detected input is interrupted, the operation reception unit 413 detects that the contact input from the touch screen 45 has been released.

The audio receiving unit 414 receives audio generated around the distribution terminal 400 and generates audio data for the audio. As an example, the audio receiving unit 414 receives audio spoken by a player and generates audio data for the audio.

The motion identification unit 415 identifies the motion data selected by the player from the motion list 422 in response to the player's input operation.

The motion instruction data generating unit 416 generates motion instruction data. As an example, the motion instruction data generating unit 416 generates motion instruction data that includes the generated voice data and the identified motion data.

The functions of the HMD set 1000, the server 200, and the user terminal 100 shown in FIG. 6 and the functions of the distribution terminal 400 shown in FIG. 7 are merely examples. Each of the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400 may have at least a part of the functions of the other devices. Furthermore, a device other than the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400 may be a component of the system 1, and the other device may execute a part of the processing in the system 1. That is, the computer that executes the game program in this embodiment may be any of the HMD set 1000, the server 200, the user terminal 100, and the distribution terminal 400, as well as other devices, or may be realized by a combination of these multiple devices.

<Virtual space control processing>
Fig. 8 is a flowchart showing an example of the flow of a control process for a virtual space provided to a player and a virtual space provided to a user of a user terminal 100. Fig. 9 is a diagram showing a virtual space 600A provided to a player and a field of view image visually recognized by the player according to an embodiment. Fig. 10 is a diagram showing a virtual space 600B provided to a user of a user terminal 100 and a field of view image visually recognized by the user according to an embodiment. Hereinafter, when there is no need to distinguish between the virtual spaces 600A and 600B, they will be referred to as "virtual space 600".

In step S1, the processor 30, as the virtual space control unit 316, defines the virtual space 600A shown in FIG. 9. The processor 30 defines the virtual space 600A using virtual space data (not shown). The virtual space data may be stored in the game play terminal 300, may be generated by the processor 30 based on the game program 331, or may be acquired by the processor 30 from an external device such as the server 200.

As an example, the virtual space 600 has a spherical structure that covers the entire 360-degree area from a point defined as the center. In order to avoid complicating the explanation, Figures 9 and 10 show an example of the upper half of the celestial sphere of the virtual space 600.

In step S2, the processor 30, functioning as the virtual space control unit 316, places an avatar object 610 (character) in the virtual space 600A. The avatar object 610 is an avatar object associated with the player, and moves according to the player's input operation.

In step S3, the processor 30, as the virtual space control unit 316, places other objects in the virtual space 600A. In the example of FIG. 9, the processor 30 places objects 631 to 634. The other objects may include, for example, character objects (so-called non-player characters, NPCs) that act according to the game program 331, control objects such as virtual hands, and objects that imitate animals, plants, artificial objects, natural objects, etc. that are placed as the game progresses.

In step S4, the processor 30, as the virtual space control unit 316, places a virtual camera 620A in the virtual space 600A. As an example, the processor 30 places the virtual camera 620A at the head position of the avatar object 610.

In step S5, the processor 30 displays the field of view image 650 on the monitor 51 and the display 530. The processor 30 defines a field of view area 640A, which is the field of view from the virtual camera 620A in the virtual space 600A, according to the initial position and inclination of the virtual camera 620A. The processor 30 then defines a field of view image 650 corresponding to the field of view area 640A. The processor 30 outputs the field of view image 650 to the monitor 51 and the display 530, thereby causing the field of view image 650 to be displayed on the HMD 500 and the display 530.

In the example of Figure 9, as shown in Figure 9(A), a portion of object 634 is included in field of view area 640A, so field of view image 650 includes a portion of object 634 as shown in Figure 9(B).

In step S6, the processor 30 transmits the initial placement information to the user terminal 100 via the server 200. The initial placement information is information that indicates the initial placement positions of various objects in the virtual space 600A. In the example of FIG. 9, the initial placement information includes information on the initial placement positions of the avatar object 610 and objects 631 to 634. The initial placement information can also be expressed as one piece of game progress information.

In step S7, the processor 30, as the virtual space control unit 316, controls the virtual camera 620A in accordance with the movement of the HMD 500. Specifically, the processor 30 controls the direction and inclination of the virtual camera 620A in accordance with the movement of the HMD 500, i.e., the position of the player's head. As described below, when the player moves his/her head (changes the position of the head), the processor 30 moves the head of the avatar object 610 in accordance with this movement. The processor 30 controls the direction and inclination of the virtual camera 620A, for example, so that the direction of the line of sight of the avatar object 610 and the direction of the line of sight of the virtual camera 620A coincide with each other. In step S8, the processor 30 updates the field of view image 650 in accordance with the change in the direction and inclination of the virtual camera 620A.

In step S9, the processor 30, as the virtual space control unit 316, moves the avatar object 610 in response to the movement of the player. As an example, the processor 30 moves the avatar object 610 in the virtual space 600A in response to the player's movement in real space. The processor 30 also moves the head of the avatar object 610 in the virtual space 600A in response to the player's movement of the head in real space.

In step S10, the processor 30, as the virtual space control unit 316, moves the virtual camera 620A so that it follows the avatar object 610. In other words, the virtual camera 620A is always at the position of the head of the avatar object 610, even if the avatar object 610 moves.

Processor 30 updates field of view image 650 in response to the movement of virtual camera 620A. That is, processor 30 updates field of view area 640A in response to the pose of the player's head and the position of virtual camera 620A in virtual space 600A. As a result, field of view image 650 is updated.

In step S11, the processor 30 transmits action instruction data for the avatar object 610 to the user terminal 100 via the server 200. The action instruction data here includes at least any of the following during the virtual experience (e.g., during game play): motion data capturing the player's actions, audio data of the player's voice, and operation data indicating the content of the input operation on the controller 540. When the player is playing a game, the action instruction data is transmitted to the user terminal 100 as, for example, game progress information.

The processing of steps S7 to S11 is continuously and repeatedly executed while the player is playing the game.

In step S21, the processor 10 of the user terminal 100 of the user 3 defines the virtual space 600B shown in FIG. 10 as the virtual space control unit 116. The processor 10 defines the virtual space 600B using virtual space data (not shown). The virtual space data may be stored in the user terminal 100, may be generated by the processor 10 based on the game program 131, or may be acquired by the processor 10 from an external device such as the server 200.

In step S22, the processor 10 receives the initial placement information. In step S23, the processor 10, as the virtual space control unit 116, places various objects in the virtual space 600B according to the initial placement information. In the example of FIG. 10, the various objects are the avatar object 610 and objects 631 to 634.

In step S24, the processor 10, as the virtual space control unit 116, places a virtual camera 620B in the virtual space 600B. As an example, the processor 10 places the virtual camera 620B at the position shown in FIG. 10(A).

In step S25, the processor 10 displays the field of view image 660 on the display unit 152. The processor 10 defines a field of view area 640B, which is the field of view from the virtual camera 620B in the virtual space 600B, according to the initial position and inclination of the virtual camera 620B. The processor 10 then defines a field of view image 660 corresponding to the field of view area 640B. The processor 10 outputs the field of view image 660 to the display unit 152, thereby causing the display unit 152 to display the field of view image 660.

In the example of Figure 10, as shown in Figure 10(A), avatar object 610 and object 631 are included in field of view area 640B, so field of view image 660 includes avatar object 610 and object 631 as shown in Figure 10(B).

In step S26, the processor 10 receives the action instruction data. In step S27, the processor 10, as the virtual space control unit 116, moves the avatar object 610 in the virtual space 600B in accordance with the action instruction data. In other words, the processor 10 plays a video of the avatar object 610 moving by real-time rendering.

In step S28, the processor 10, as the virtual space control unit 116, controls the virtual camera 620B in response to the user's operation received by the operation receiving unit 111. In step S29, the processor 10 updates the field of view image 660 in response to changes in the position of the virtual camera 620B in the virtual space 600B and the direction and inclination of the virtual camera 620B. Note that in step S28, the processor 10 may automatically control the virtual camera 620B in response to the movement of the avatar object 610, for example, the movement of the avatar object 610 or a change in the direction. For example, the processor 10 may automatically move the virtual camera 620B or change the direction and inclination so that the avatar object 610 is always photographed from the front. Also, as an example, the processor 10 may automatically move the virtual camera 620B or change the direction and inclination so that the avatar object 610 is always photographed from the rear in response to the movement of the avatar object 610.

In this way, in virtual space 600A, avatar object 610 moves in response to the player's movements. Action instruction data indicating this movement is transmitted to user terminal 100. In virtual space 600B, avatar object 610 moves in response to the received action instruction data. As a result, avatar object 610 performs similar movements in virtual space 600A and virtual space 600B. In other words, user 3 can use user terminal 100 to visually recognize the movement of avatar object 610 in response to the player's movement.

＜Game Overview＞
11 is a diagram showing another example of a field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of a game screen of a game (this game) executed by the system 1 and being played by a player.

As an example, this game is a game in which an avatar object 610 that controls a weapon such as a gun or a knife and multiple enemy objects 671 that are NPCs appear in a virtual space 600, and the avatar object 610 fights the enemy objects 671. Various game parameters such as the vitality of the avatar object 610, the number of usable magazines, the number of bullets remaining in the gun, and the number of remaining enemy objects 671 are updated as the game progresses.

In this game, a plurality of stages are prepared, and the player can clear each stage by achieving a predetermined achievement condition associated with the stage. The predetermined achievement condition may include, for example, a condition that is achieved by defeating all enemy objects 671 that appear, defeating a boss object among the enemy objects 671 that appear, acquiring a predetermined item, reaching a predetermined position, and the like. The achievement condition is defined in the game program 131. In this game, the player clears a stage when an achievement condition is achieved in accordance with the content of the game, in other words, the victory of the avatar object 610 over the enemy object 671 (win or loss between the avatar object 610 and the enemy object 671) is determined. In contrast, for example, if the game executed by the system 1 is a racing game or the like, the ranking of the avatar object 610 is determined when the condition of reaching the goal is achieved.

In this game, in order to share a virtual space between the HMD set 1000 and multiple user terminals 100, game progress information is live-streamed to multiple user terminals 100 at predetermined time intervals. As a result, a field of view image of the field of view area defined by the virtual camera 620B corresponding to the user terminal 100 is displayed on the touch screen 15 of the user terminal 100 while watching the game. In addition, parameter images representing the vitality of the avatar object 610, the number of available magazines, the number of bullets remaining in the gun, the number of enemy objects 671 remaining, etc. are superimposed and displayed on the upper right and upper left sections of the field of view image. This field of view image can also be expressed as a game screen.

As described above, the game progress information includes motion data capturing the player's actions, audio data of the voice spoken by the player, and operation data indicating the content of input operations on the controller 540. These pieces of data are information for identifying the position, posture, orientation, etc. of the avatar object 610, information for identifying the position, posture, orientation, etc. of the enemy object 671, and information for identifying the positions of other objects (e.g., obstacle objects 672, 673). The processor 10 analyzes (renders) the game progress information to identify the position, posture, orientation, etc. of each object.

Game information 132 includes data on various objects such as avatar object 610, enemy object 671, obstacle objects 672, 673, etc. Processor 10 uses this data and the analysis results of the game progress information to update the position, posture, orientation, etc. of each object. As a result, the game progresses, and each object in virtual space 600B moves in the same way as each object in virtual space 600A. Specifically, in virtual space 600B, each object including avatar object 610 behaves based on the game progress information, regardless of whether or not a user operates user terminal 100.

As an example, UI images 701 and 702 are displayed superimposed on the field of view image on the touch screen 15 of the user terminal 100. UI image 701 is a UI image that accepts an operation for displaying on the touch screen 15 a UI image 711 that accepts an item insertion operation from the user 3 to support the avatar object 610. UI image 702 is a UI image that accepts an operation for displaying on the touch screen 15 a UI image (described later) that accepts an operation from the user 3 to input and send a comment for the avatar object 610 (in other words, the player 4). The operation accepted by UI images 701 and 702 may be, for example, an operation of tapping on UI images 701 and 702.

When UI image 701 is tapped, UI image 711 is displayed superimposed on the field of view image. UI image 711 includes, for example, UI image 711A depicting a magazine icon, UI image 711B depicting a first aid box icon, UI image 711C depicting a traffic cone icon, and UI image 711D depicting a barricade icon. An item insertion operation corresponds to, for example, tapping on any of the UI images.

As an example, when UI image 711A is tapped, the number of remaining bullets in the gun used by avatar object 610 increases. When UI image 711B is tapped, the stamina of avatar object 610 is restored. When UI images 711C and 711D are tapped, obstacle objects 672, 673 that impede the movement of enemy object 671 are placed in the virtual space. One of obstacle objects 672, 673 may be more likely to impede the movement of enemy object 671 than the other.

The processor 10 transmits item input information indicating that an item input operation has been performed to the server 200. The item input information includes at least information for identifying the type of item specified by the item input operation. The item input information may also include other information related to the item, such as information indicating the position where the item is placed. The item input information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

Figure 12 is a diagram showing another example of a field of view image displayed on the user terminal 100. Specifically, it is a diagram showing an example of a game screen of the present game, and is a diagram for explaining communication between a player and the user terminal 100 during game play.

In the example of FIG. 12(A), the user terminal 100 causes the avatar object 610 to make an utterance 691. Specifically, the user terminal 100 causes the avatar object 610 to make an utterance 691 in accordance with the voice data included in the game progress information. The content of the utterance 691 is "I'm out of bullets!" uttered by player 4. In other words, the content of the utterance 691 informs each user that the magazine is empty and there is only one bullet loaded in the gun, and therefore they are about to lose the means to attack the enemy object 671.

Note that in FIG. 12(A), speech bubbles are used to visually indicate the speech of the avatar object 610, but in reality, audio is output from the speaker of the user terminal 100. Note that along with the audio output, the speech bubble shown in FIG. 12(A) (i.e., a speech bubble containing the text of the audio content) may be displayed in the field of view image. This is also true for speech 692, which will be described later.

When a tap operation on UI image 702 is received, the user terminal 100 displays UI images 705 and 706 (message UIs) superimposed on the field of view image, as shown in FIG. 12(B). UI image 705 is a UI image that displays a comment for the avatar object 610 (in other words, the player). UI image 706 is a UI image that receives a comment sending operation from the user 3 to send the input comment.

As an example, when the user terminal 100 receives a tap operation on the UI image 705, it displays a UI image (not shown, hereinafter simply referred to as "keyboard") resembling a keyboard on the touch screen 15. The user terminal 100 displays text in response to the user's input operation on the keyboard on the UI image 705. In the example of FIG. 12(B), the text "I'll send you a magazine" is displayed on the UI image 705.

After inputting text, for example, when the user terminal 100 receives a tap operation on the UI image 706, the user terminal 100 transmits comment information including information indicating the input content (text content) and information indicating the user to the server 200. The comment information is transmitted to other user terminals 100 and the HMD set 1000 via the server 200.

UI image 703A is a UI image showing the username of the user who sent the comment, and UI image 704A is a UI image showing the content of the comment sent by that user. In the example of FIG. 12(B), a user with the username "BBBBB" has sent comment information with the content "Danger!" using his/her own user terminal 100, causing UI image 703A and UI image 704A to be displayed. UI image 703A and UI image 704A are displayed on the touch screens 15 of all user terminals 100 participating in the game, and on the monitor 51 of the HMD 500. Note that UI images 703A and 704A may be a single UI image. In other words, a single UI image may include the username and the content of the comment.

In the example of FIG. 12(C), a user with the user name "AAAAA", who is a user of the user terminal 100 shown in FIG. 12, has entered and sent a comment as described above, causing UI images 703B and 704B to be displayed on the touch screen 15. UI image 703B includes the user name "AAAAA", and UI image 704B includes the comment "I'll send you a magazine!" that was entered in the example of FIG. 12(B).

The example in FIG. 12(C) is a field of view image 611 after user "AAAAA" further inputs a tap operation on UI image 701, causes UI image 711 to be displayed on touch screen 15, and inputs a tap operation on UI image 711A. That is, as a result of item insertion information indicating a magazine being transmitted from user terminal 100 of user "AAAAA" to other user terminals 100 and HMD set 1000, user terminal 100 and HMD set 1000 place effect object 674 (described below) in virtual space 600. As an example, user terminal 100 and HMD set 1000 execute an effect related to effect object 674 after the elapsed time indicated in the item insertion information has elapsed, and execute a process to activate the effect of the item object.

In the example of FIG. 12(D), the number of magazines increases from 0 to 1 by executing a process to activate the effect of the item object. As a result, the player says "Thank you!" to the user "AAAAA", and the voice data of this utterance is transmitted to each user terminal 100. As a result, each user terminal 100 outputs the voice "Thank you!" as the utterance 692 of the avatar object 610.

As described above, in this game, communication between users and avatar object 610 is realized by outputting the spoken voice of avatar object 610 based on the player's speech and by inputting comments by each user.

(Game Progression Processing in the Game Playing Terminal 300)
FIG. 13 is a flow chart showing an example of the flow of a game progression process executed on the game playing terminal 300.

In step S31, the processor 30, as the game progression unit 315, progresses the game based on the game program 331 and the player's movements. In step S32, the processor 30 generates game progress information and distributes it to the user terminals 100. Specifically, the processor 30 transmits the generated game progress information to each user terminal 100 via the server 200.

In step S33, when the processor 30 receives the item insertion information (YES in S33), in step S34, the processor 30 places the item object in the virtual space 600A based on the item insertion information. As an example, the processor 30 places an effect object 674 in the virtual space 600A before placing the item object (see FIG. 11(C)). The effect object 674 may be, for example, an object resembling a gift box. As an example, the processor 30 may execute an effect related to the effect object 674 after the elapsed time indicated in the item insertion information has elapsed. The effect may be, for example, an animation of the lid of the gift box opening. After the execution of the animation, the processor 30 executes a process to activate the effect of the item object. For example, in the example of FIG. 11(D), an obstacle object 673 is placed.

After the animation is executed, the processor 30 may place an item object corresponding to the tapped UI image in the virtual space 600A. For example, when a tap operation is performed on the UI image 711A, the processor 30 places a magazine object indicating a magazine in the virtual space 600A after the animation is executed. Also, when a tap operation is performed on the UI image 711B, the processor 30 places a first aid box object indicating a first aid box in the virtual space 600A after the animation is executed. For example, when the avatar object 610 moves to the position of the magazine object or the first aid box object, the processor 30 may execute a process to activate the effect of the magazine object or the first aid box object.

The processor 30 continues and repeats the processing of steps S31 to S34 until the game ends. When the game ends, for example, when the player inputs a predetermined input operation to end the game (YES in step S35), the processing shown in FIG. 13 ends.

(Game Progression Processing in User Terminal 100)
FIG. 14 is a flowchart showing an example of the flow of a game progression process executed on the user terminal 100.

In step S41, the processor 10 receives game progress information. In step S42, the processor 10, as the game progression unit 115, progresses the game based on the game progress information.

In step S43, when the processor 10 accepts an item insertion operation by the user 3 (YES in step S43), in step S44, the processor 10 consumes the virtual currency and places the effect object 674 in the virtual space 600B. Here, the virtual currency may be purchased (charged for the game) by the user 3 performing a predetermined operation on the processor 10 before or during participation in the game, or may be granted to the user 3 when a predetermined condition is met. The predetermined condition may be something that requires participation in the game, such as clearing a quest in the game, or something that does not require participation in the game, such as answering a questionnaire. The amount of virtual currency (amount of virtual currency owned) is, for example, stored in the user terminal 100 as game information 132.

In step S45, the processor 10 transmits item insertion information to the server 200. The item insertion information is transmitted to the game play terminal 300 via the server 200.

After placing the effect object 674, when a predetermined time has elapsed, the processor 10 places an item object in the virtual space 600A. In the example of FIG. 11, an obstacle object 673 is placed. That is, when the user 3 inputs a tap operation on the UI image 711C, a predetermined amount of virtual currency is consumed and the obstacle object 673 is placed.

The processor 10 continues and repeats the processing of steps S41 to S45 until the game ends. When the game ends, for example, when the player performs a predetermined input operation to end the game, or when the user 3 performs a predetermined input operation to leave the game (YES in step S46), the processing shown in FIG. 14 ends.

(Game Progression Processing in Server 200)
FIG. 15 is a flowchart showing an example of the flow of a game progression process executed by the server 200.

In step S51, the processor 20 receives game progress information from the game play terminal 300. In step S52, the processor 20, as the log generation unit 212, updates the game progress log (hereinafter, the play log). Note that the play log is generated by the processor 20, for example, when initial placement information is received from the game play terminal 300.

In step S53, the processor 20 transmits the received game progress information to each user terminal 100.

In step S54, if item input information is received from any of the user terminals 100 (YES in step S54), in step S55, the processor 20 updates the play log as the log generating unit 212. In step S56, the processor 20 transmits the received item input information to the game play terminal 300.

The processor 20 continues and repeats the processing of steps S51 to S56 until the game ends. When the game ends, for example when information indicating that the game has ended is received from the game play terminal 300 (YES in step S57), in step S58, the processor 20, as the list generation unit 213, generates a list of users who participated in the game (user list 234) from the play log. The processor 20 stores the generated user list 234 in the server 200.

Figure 16 is a diagram showing a specific example of a user list 234. The "User" column stores information (e.g., username) indicating each user who participated in the game. The "Tag" column stores information (tags) generated based on the assistance provided by each user to the player. In the example of Figure 16, among the tags stored in the "Tag" column, those without quotation marks are information automatically generated by the processor 20, and those with quotation marks are information manually entered by the game operator.

In the example of FIG. 16, the user "AAAAA" is associated with the information magazine, 10th floor, boss, and "defeat boss by gifting magazine." This indicates that, for example, in a boss battle on stage 10th floor, the user "AAAAA" inserted a magazine, and the avatar object 610 defeated the boss with bullets from the inserted magazine.

Furthermore, the user "BBBBB" is associated with the information: first aid kit, 3rd floor, weak enemy, and "recovered just before game over." This indicates that, for example, in a battle with a weak enemy on stage 3F, the user "BBBBB" throws in a first aid kit, and as a result, the avatar object 610's vitality is recovered just before it reaches 0 (game over).

Furthermore, the user "CCCCC" is associated with the information "barricade, 5F, weak enemies, and "two zombies held back by barricade." This indicates that, for example, in a battle with weak enemies on stage 5F, the user "CCCCC" put in a barricade (obstacle object 672 in FIG. 11), and as a result, succeeded in holding back two weak enemies.

In the example of FIG. 16, one support provided is associated with the username of each user 3, but the username of a user 3 who provided support multiple times is associated with a tag for each of the multiple support instances. It is preferable that each tag is distinguished in the user list 234. This allows a player who refers to the user list 421 using the distribution terminal 400 after the game ends to accurately understand the content of each support.

<Distribution of operation instruction data>
(Distribution process in distribution terminal 400)
Fig. 17 is a flowchart showing an example of the flow of a distribution process executed by distribution terminal 400. Fig. 18 is a diagram showing a specific example of a screen displayed on distribution terminal 400. Fig. 19 is a diagram showing another specific example of a screen displayed on the distribution terminal.

In step S61, the processor 40, as the operation receiving unit 413, receives a first operation for displaying a list of users who have participated in the game (user list 234). The download screen 721 shown in FIG. 18(A) is a screen for downloading the user list 234 from the server 200 and displaying it on the display unit 452. As an example, the download screen 721 is a screen that is displayed immediately after an operation to start an application that executes the distribution process shown in FIG. 17 is input to the distribution terminal 400.

Download screen 721 includes UI images 722 and 723, as an example. UI image 722 accepts an operation for downloading user list 234, i.e., the first operation described above. The first operation may be, for example, an operation of tapping UI image 722. UI image 723 accepts an operation for closing an application. The operation may be, for example, an operation of tapping UI image 723.

When a tap operation on the UI image 722 is received, in step S62, the processor 40, as the communication control unit 411, acquires (receives) the user list 234 from the server 200. In step S63, the processor 40, as the display control unit 412, causes the display unit 452 to display the user list 234. Specifically, the processor 40 causes the display unit 452 to display a user list screen generated based on the user list 234. As an example, the user list screen may be the user list screen 731 shown in FIG. 18 (B). The user list screen 731 consists of record images corresponding to each record in the user list 234. In the example of FIG. 18 (B), record images 732A to 732C are described as record images, but the number of record images is not limited to three. In the example of FIG. 18(B), if the number of records in the user list 234 is greater than three (i.e., the number of users participating in the game is greater than three), the player can display other record images on the display unit 452, for example, by inputting an operation to scroll the screen (e.g., a drag operation or a flick operation) on the touch screen 45.

As an example, record images 732A-732C include user names 733A-733C, tag information 734A-734C, and icons 735A-735C, respectively. Hereinafter, when there is no need to distinguish between record images 732A-732C, user names 733A-733C, tag information 734A-734C, and icons 735A-735C, they will be referred to as "record image 732," "user name 733," "tag information 734," and "icon 735," respectively.

The user name 733 is information that indicates each user who participated in the game, and is stored in the "user" column in the user list 234. The tag information 734 is information that indicates a tag that is associated with each piece of information that indicates each user who participated in the game in the user list 234. For example, the record image 732A includes "AAAAA" as the user name 733A. Therefore, the record image 732A includes, as the tag information 734A, "Magazine, 10F, Boss, 'Defeat the boss by gifting a magazine'," which is associated with "AAAAA" in the user list 234. The icon 735 is, for example, an image that the user has set in advance.

The processor 40 may store the received user list in the distribution terminal 400 (user list 421 in FIG. 7). The download screen 721 may include a UI image (not shown) for displaying the user list 421 on the display unit 452. In this example, when the UI image is tapped, the processor 40 does not download the user list 234, but reads out the user list 421, generates a user list screen from the user list 421, and displays it on the display unit 452.

In step S64, the processor 40, as the operation receiving unit 413, receives a second operation for selecting one of the users included in the user list screen 731. As an example, the second operation may be an operation of tapping one of the record images 732 on the user list screen 731. In the example of FIG. 18(B), the player inputs a tap operation on the record image 732A. That is, the player selects the user "AAAAA" as the user to whom the action instruction data is to be distributed.

When a tap operation on the record image 732 is received, in step S65, the processor 40, as the display control unit 412, causes the display unit 452 to display the motion list 422. Specifically, the processor 40 causes the display unit 452 to display a motion list screen generated based on the motion list 422. As an example, the motion list screen may be the motion list screen 741 shown in FIG. 19. The motion list screen 741 is made up of record images corresponding to each record in the motion list 422. In the example of FIG. 19, record images 742A to 742C are shown as record images, but the number of record images is not limited to three. In the example of FIG. 19, if the number of records in the motion list 422 is more than four, the player can display other record images on the display unit 452 by inputting an operation to scroll the screen (e.g., a drag operation or a flick operation) to the touch screen 45.

As an example, the record images 742A to 742C include motion names 743A to 743C, motion images 744A to 744C, and UI images 745A to 745C, respectively. Hereinafter, when there is no need to distinguish between the record images 742A to 742C, the motion names 743A to 743C, the motion images 744A to 744C, and the UI images 745A to 745C, they will be referred to as "record image 7432", "motion name 743", "
"Motion image 744" and "UI image 745".

The motion name 743 is information for identifying a motion stored in the motion list 422. The motion image 744 is an image generated from the motion data associated with each motion name in the motion list 422. As an example, the processor 40 includes an image of the avatar object 610 taking an initial posture in each motion data in the record image 742 as the motion image 744. The motion image 744 may be a UI image that accepts a predetermined operation by the player (e.g., a tap operation on the motion image 744). When the processor 40 accepts the predetermined operation, it may play a motion video in which the avatar object 610 moves based on the motion data. When the motion video ends, the processor 40 may automatically redisplay the motion list screen 741.

Note that instead of the motion image 744, the record image 742 may include a UI image including the text "motion playback," for example.

In step S66, the processor 40, as the operation receiving unit 413, receives a third operation for selecting a motion. As an example, the third operation may be a tap operation on the UI image 745. In other words, the UI image 745 receives an operation for selecting motion data corresponding to each record image 742. Having received the third operation, the processor 40, as the motion identifying unit 415, identifies the motion data selected by the player.

In step S67, the processor 40, acting as the display control unit 412 and the voice receiving unit 414, receives voice input from the player while playing back a motion video in which the avatar object 610 moves based on the selected motion data.

Figure 20 is a diagram showing a specific example of voice input by player 4. As shown in Figure 20, player 4 is inputting speech 820A while playing back motion video 810A. This speech 820A is addressed to user 3 (hereinafter, user 3A) whose user name is "AAAAA." That is, in the example of Figure 20, player 4 selects user 3A (first user) in step S64 and creates action instruction data addressed to user 3A. It should be noted that the user terminal 100 used by user 3A is user terminal 100A.

Since the speech voice 820A is addressed to the user 3A, it is based on the content of the support that the user 3A provided to the avatar object 610 (in other words, the player 4). Specifically, the user 3A inserted a magazine in the boss battle on stage 10F, and the avatar object 610 defeated the boss with the bullets from the inserted magazine. Therefore, the speech voice 820A is "Thank you for giving me the magazine during the boss battle! Your timing was perfect, too! Thanks to AAAAAA, I was able to clear it!" In this way, it is preferable that the speech voice includes the content of the support that the user 3 provided in the game and gratitude to the user 3.

In one situation, the player 4 creates a speech content addressed to the user 3 before starting voice input, i.e., before inputting the third operation to the distribution terminal 400. In another situation, the speech content addressed to the user 3 may be automatically generated by the processor 40. The processor 40 may also display a tag associated with the user 3 selected by the second operation superimposed on the motion video 810A.

The processor 40 converts the received voice into voice data. In step S68, the processor 40, as the motion instruction data generator 416, generates motion instruction data including the voice data and the motion data of the selected motion.

In step S69, the processor 40, as the communication control unit 411, distributes the generated action instruction data to the user terminal 100 (first computer) of the selected user 3 (user 3A in the example of FIG. 20). FIG. 21 is a diagram showing yet another specific example of a screen displayed on the distribution terminal 400. After executing step S68, the processor 40, as the display control unit 412, causes the display unit 452 to display a distribution screen. As an example, the distribution screen may be the distribution screen 751 shown in FIG. 21(A). The distribution screen 751 includes a UI image 752 and a motion image 753A. Furthermore, the distribution screen 751 may include information indicating the user to which the action instruction data is to be distributed, as shown in FIG. 21(A).

The UI image 752 accepts an operation to distribute the action instruction data to the selected user 3. The operation may be, for example, a tap operation on the UI image 752. The motion image 753A is a UI image that accepts an operation to play a video based on the generated action instruction data, i.e., a video based on the action instruction data generated for the user 3A. The operation may be, for example, a tap operation on the motion image 753A. Note that the UI image that accepts an operation to play the generated video is not limited to the motion image 753A. For example, it may be a UI image that includes the text "Play video." The processor 40 may automatically redisplay the distribution screen 751 when the video ends.

It is preferable that the distribution screen 751 further includes a UI image that accepts an operation to return to receiving voice input. The operation may be, for example, a tap operation on the UI image. By including the UI image on the distribution screen 751, the player 4 can retry voice input if the voice input fails, for example, if the player 4 makes a mistake in what he or she wants to say. Note that the UI image may be a UI image that accepts an operation to return to selecting motion data.

When a tap operation on the UI image 752 is received, the processor 40 transmits the action instruction data to the server 200 together with information indicating the user 3A. The server 200 identifies the user terminal 100 to which the action instruction data is to be transmitted based on the information indicating the user 3A, and transmits the action instruction data to the identified user terminal 100 (i.e., the user terminal 100A).

When the transmission of the operation instruction data is completed, the processor 40 may, for example, cause the display unit 452 to display a distribution completion screen 761 shown in FIG. 21(B). For example, the distribution completion screen 761 includes UI images 762 and 763. Furthermore, as shown in FIG. 21(B), the distribution completion screen 761 may include text indicating that the transmission of the operation instruction data has been completed.

The UI image 762 accepts an operation to start creating action instruction data addressed to another user 3. The operation may be, for example, an operation of tapping the UI image 762. When the processor 40 accepts the tap operation, it causes the display unit 452 to display the user list screen again. That is, when the tap operation is accepted, the distribution process returns to step S63. At this time, the processor 40 may generate a user list screen based on the user list 421 stored in the distribution terminal 400 and cause the display unit 452 to display the user list screen. The UI image 763 accepts an operation to end the application. The operation may be, for example, an operation of tapping the UI image 763. When the operation is accepted, the distribution process ends.

In the example described with reference to Figures 20 and 21, as shown in Figure 21 (C), the distribution terminal 400 transmits action instruction data for a video addressed to user 3A (user 3 whose user name is "AAAA") only to user terminal 100A.

FIG. 22 is a diagram showing another specific example of voice input by player 4. As shown in FIG. 22, player 4 inputs speech 820B while playing back motion video 810B. This speech 820B is addressed to user 3 (hereinafter, user 3B) whose user name is "BBBBB." That is, in the example of FIG. 22, player 4 inputs a tap operation on record image 732B corresponding to user 3B in step S64, and creates action instruction data addressed to user 3B. It is assumed that the user terminal 100 used by user 3B is user terminal 100B.

Since the speech voice 820B is addressed to user 3B, it is based on the content of the support that user 3B provided to the avatar object 610 (in other words, player 4). Specifically, in a battle with a weak enemy on stage 3F, user "BBBBB" threw in a first aid kit, and as a result, the avatar object 610's strength recovered just before it reached 0 (the game would be over). Therefore, the speech voice 820B is "Thanks to the first aid kit that BBBBB gave me, I didn't get a game over on the 3rd floor. Thank you so much!".

FIG. 23 is a diagram showing yet another specific example of a screen displayed on the distribution terminal 400. The distribution screen 751 shown in FIG. 23(A) includes a UI image 752 and a motion image 753B. When a tap operation is received, the motion image 753B plays a video based on the action instruction data generated for the user 3B.

When a tap operation on the UI image 752 is received, the processor 40 transmits the action instruction data to the server 200 together with information indicating the user 3B. The server 200 identifies the user terminal 100 to which the action instruction data is to be transmitted based on the information indicating the user 3B, and transmits the action instruction data to the identified user terminal 100 (i.e., the user terminal 100B).

In the example described with reference to Figures 22 and 23, as shown in Figure 23 (C), the distribution terminal 400 transmits the action instruction data for the video addressed to user 3B (user 3 whose user name is "BBBBB") only to user terminal 100B.

As described above, the content of the voice based on the voice data included in the action instruction data is based on the content of the support that the user 3 provided to the player 4 when participating in the most recent game. Since the content of the support differs for each user 3, the content of the voice will be different for each user 3. In other words, after the game ends, action instruction data including voices with different content is transmitted to the user terminals 100 of at least some of the users 3 who participated in the game.

Furthermore, the motion of the avatar object 610 in the example of FIG. 22 is different from the motion in the example of FIG. 20. That is, when generating the action instruction data for user 3B, the player 4 selects motion data that is different from the motion instruction data for user 3A. Specifically, in step S66, the player 4 inputs a tap operation on the UI image 745B, which selects the motion data corresponding to the record image 742B. In this way, the player 4 can make the motion data included in the action instruction data different for each user 3.

Then, the motion instruction data for each user 3, including the voice data with different content for each user 3 and the motion data selected for each user 3, is transmitted only to the user terminal 100 of each user 3. In other words, the motion instruction data that is unique for each user terminal 100 is transmitted to each of the user terminals 100 of the selected users 3.

Figure 24 is a diagram showing an overview of the transmission of game progress information from a game playing terminal 300 to a user terminal 100. While the action instruction data for video playback in the user terminal 100 is unique for each user terminal 100, as shown in Figure 24, the game progress information transmitted to the user terminals 100 of all users 3 participating in the game during game execution is common to each user terminal 100. In other words, the action instruction data included in the game progress information is also common to each user terminal 100. In this way, it can be said that the action instruction data for video playback and the action instruction data for progressing the game are different data in terms of similarities and differences between user terminals 100 and transmission destinations.

(Video Playback Processing in User Terminal 100)
FIG. 25 is a flowchart showing an example of the flow of a video playback process executed by the user terminal 100.

In step S71, the processor 10, functioning as the video playback unit 117, receives the action instruction data. In step S72, the processor 10, functioning as the video playback unit 117, notifies the user 3 of the reception of the action instruction data. As an example, the processor 10 notifies the user 3 of the reception of the action instruction data by at least one of displaying a notification image on the display unit 152, playing a notification sound from a speaker (not shown), and lighting or blinking a lighting unit (not shown) constituted by an LED (light-emitting diode) or the like.

In step S73, the processor 10, as the operation receiving unit 111, receives a first playback operation for playing the video. The first playback operation may be, for example, an operation of tapping on a notification image. In step S74, the processor 10, as the video playback unit 117, renders the action instruction data and plays the video. For example, the processor 10 may launch an application for playing the game and play the video, or may launch a video playback application separate from the application and play the video. Hereinafter, the video will be referred to as the "thank you video."

FIG. 26 is a diagram showing a specific example of playback of a thank you video. Specifically, it is a diagram showing an example of playback of a thank you video on the user terminal 100 of user 3A. In the thank you video 910A played on the user terminal 100, the avatar object 610 speaks audio 920A while performing a certain motion. In other words, the processor 10 outputs audio 920A from a speaker (not shown) while playing back the thank you video 910A that includes the avatar object 610 performing a certain motion.

The motion in the thank you video 910A is based on the motion data selected by the player 4 when generating the action instruction data addressed to the user 3A, and the audio 920A is based on the audio data generated from the speech audio 820A input by the player 4 when generating the action instruction data. In other words, the audio 920A is audio that includes the content of the assistance provided by the user 3A in the game and gratitude for that assistance. In this way, by inputting the first playback operation, the user 3A can watch a thank you video in which the avatar object 610 speaks the content of the assistance he or she provided in the game and gratitude for that assistance.

As an example, after playback of the thank you video 910A has ended, the user terminal 100 may display at least one UI image on the touch screen 15. The UI image may be, for example, a UI image that accepts an operation to play the thank you video 910A again, a UI image that accepts an operation to transition to another screen, or a UI image that accepts an operation to end the application.

Also, as an example, the user terminal 100 may display at least one UI image on the touch screen 15 while the thank you video 910A is being played. The UI image may be, for example, a plurality of UI images each of which accepts an operation to temporarily stop or end the thank you video 910A being played, or to change the scene being played.

Note that these UI images displayed during playback of thank you video 910A and after playback of thank you video 910A has stopped do not include UI images for replying to avatar object 610. In other words, thank you video 910A according to this embodiment does not include a means for replying to avatar object 610.

Figure 27 is a diagram showing another specific example of playback of a thank you video. Specifically, it is a diagram showing an example of playback of a thank you video on the user terminal 100 of user 3B. In the thank you video 910B played on the user terminal 100, the avatar object 610 utters audio 920B while performing a certain motion. In other words, the processor 10 outputs audio 920B from a speaker (not shown) while playing back the thank you video 910B that includes the avatar object 610 performing a certain motion.

The motion in the thank you video 910B is based on the motion data selected by the player 4 when generating the action instruction data addressed to the user 3B, and the audio 920B is based on the audio data generated from the speech audio 820B input by the player 4 when generating the action instruction data. Therefore, the motion performed by the avatar object 610 in the example of FIG. 27 is different from the motion in the example of FIG. 26. Also, the audio 920B is audio that includes the content of the assistance provided by the user 3B in the game and gratitude for the assistance. Therefore, in the example of FIG. 27, the content of the audio 920B is different from the content of the audio 920A in the example of FIG. 26.

In this way, after the game ends, the thank you video received by at least some of the user terminals 100 of the users 3 who participated in the game is a video in which the speech content of the avatar object 610 differs for each user 3.

The processor 10 may display a UI image 930 including content encouraging participation in the next game, superimposed on the video 910. The UI image 930 may be distributed together with the action instruction data, or may be stored in the user terminal 100 as game information 132.

In this game, the game play terminal 300 distributes game progress information, which is distribution content, to multiple user terminals 100 via the server 200. Each of the multiple user terminals 100 displays the distribution content on the touch screen 15 based on the game progress information. Specifically, multiple types of object data for displaying objects including an avatar object 610 are stored in advance in the memory 11 of the user terminal 100. When the user terminal 100 receives game progress information from the game play terminal 300, it identifies action instruction data from the game progress information and analyzes (renders) the action instruction data to identify the type, position, posture, etc. of the object. In addition, based on the analysis result, the user terminal 100 uses the object data stored in the memory 11 to place objects including the avatar object 610 in the virtual space 600B. The touch screen 15 displays a field of view image 660 of the field of view area 640B according to the current position and orientation of the virtual camera 620B. Note that objects including the avatar object 610 are also placed in the virtual space 600A defined by the game play device 300, and the positions, orientations, etc. of the objects are consistent between the virtual spaces 600A and 600B.

The distributed content includes multiple types of content in which different characters are displayed as avatar objects 610. The multiple types of content include, for example, first distributed content that is distributed once or irregularly, and second distributed content that is distributed regularly every day or at a specified time on a specified day of the week.

The first and second distribution contents each include a plurality of types of content that differ, for example, in the characters that appear as the main characters or facilitators, the characters that appear in the content, the progress, or other aspects, and the types of games that are played during the content. In addition, the second distribution content has the same character that corresponds to the type (for example, character a for content a) appearing each time, and the character background (for example, a wreath object, etc.) is displayed, but the content, etc. differs each time even for the same type of content (for example, content a).

Characters displayed as avatar objects 610 are associated with decorative objects (texture images) such as clothing and accessories for each character type. Below, we will provide an example of a specific method for specifying the display mode of the decorative objects.

The distributed content includes first character content in which a first character appears in virtual space 600B as avatar object 610, and second character content in which a second character appears in virtual space 600B as avatar object 610. When distributing first character content, the action instruction data is data for enabling display of an image of virtual space 600B in which objects including the first character are placed (first display data). On the other hand, when distributing second character content, the action instruction data is data for enabling display of an image of virtual space 600B in which objects including the second character are placed (second display data).

For both the first and second characters, decorative objects equivalent to clothing and accessories (necklaces, earrings, glasses, etc.) are pre-associated with each body part, such as the upper body, lower body, hands, feet, neck, earlobes, and eyes. The type and body parts of the decorative objects to be associated are determined to differ depending on the type of character.

In this embodiment, a first character is associated with decorative objects including a first object corresponding to a shirt and a third object (blouse) that is worn over the first object (shirt) to cover a part of the first object. On the other hand, a second character is associated with decorative objects including a second object of the same type as the first object (a shirt with a different design, etc.).

Here, the display mode of the first object associated with the first character is determined on the game play terminal 300 side according to a predetermined rule (e.g., random number drawing) for each distribution, such as at the start of distribution. The game play terminal 300 distributes game progress information including display mode information capable of identifying the display mode to the user terminal 100, and the user terminal 100 identifies and sets the display mode of the first object based on the display mode information. On the other hand, the display modes of the decorative objects other than the first object associated with the first character, including the display mode of the third object, are fixed in advance. The user terminal 100 sets the display modes of the decorative objects other than the first object to a fixed display mode. On the other hand, the display modes of the decorative objects associated with the second character, including the display mode of the second object of the same type as the first object, are fixed in advance. The user terminal 100 sets the display modes of the decorative objects to a fixed display mode.

For this reason, the display mode of the first object among the decorative objects associated with the first character is changed for each distribution. Here, changing the display mode includes changing the texture image, and the texture image of the first object is changed for each distribution. On the other hand, the display modes of the decorative objects other than the first object associated with the first character and the decorative objects associated with the second character are fixed regardless of the number of distributions (distribution period).

The display mode is the mode in which a decorative object is displayed on the display unit 152. In this embodiment, an example is shown in which the display mode determined for a first object (shirt) is the display color of the first object (i.e., the color of the shirt). Therefore, among decorative objects such as shirts and blouses associated with the first character, the color of the shirt is specified from among multiple colors (white, black, red, green, blue, purple, etc.).

In this embodiment, the second distribution content that is distributed periodically includes, for example, a first character content in which a first character appears, in which the display mode of the first object is changed each time. In addition, in this embodiment, an example of the first character is a character that looks like an office lady. However, the first character is not limited to this, and may be, for example, a character whose age is set to less than a specified age (18 years old) (e.g., a high school student character), a character that looks like a cheering squad member, or a character that looks like a female university student.

Content is distributed from the game play terminal 300 during a predetermined distribution period. When content distribution begins, the processor 30 of the game play terminal 300 executes the display mode control process shown in FIG. 28 (A). Furthermore, content viewing on the user terminal 100 side is started by a touch operation by the user on a predetermined content icon. When content viewing begins by a touch operation, the processor 10 on the user terminal 100 side executes the display mode setting process shown in FIG. 28 (B).

Referring to FIG. 28(A), in step S81, the game play terminal 300 determines whether the current timing coincides with the start timing of content distribution that is predetermined for each piece of content, based on a distribution schedule stored in memory 31. That is, information such as the time period during which the content is distributed and the type of content (characters appearing, i.e., whether it is first character content or second character content, etc.) is stored in memory 31 as a distribution schedule. In step S81, it is determined whether there is a match based on the distribution schedule.

If it is determined that the start timing coincides, the process proceeds to step S82. In step S82, it is determined whether the content to be distributed this time is the first character content based on the distribution schedule. If it is determined that the content is the first character content, the process proceeds to step S83.

In step S83, the display mode of the first object (i.e., the shirt color) of the decorative objects associated with the first character during the current distribution period is determined (randomly) from among a plurality of display modes (white, black, red, green, blue, purple, etc.) by random number lottery. More specifically, the first character content is the second distribution content that is distributed periodically as described above, and in step S83, a display mode different from the display mode determined when the first character content was previously distributed is determined by random number lottery. As a result, the display mode of the first object (i.e., the shirt color) is changed to a display mode different from that at the time of the previous distribution.

In step S84, display mode information (first data) capable of identifying the display mode determined in step S83 is set as information included in the game progress information. When the first character content is distributed, game progress information that corresponds to the motions of the performer, etc. and includes the display mode information according to the determination in step S83 is distributed every predetermined time (e.g., 1/60th of a second).

When the processing of step S84 is completed, when it is not determined in step S81 that the current timing coincides with the start timing of the content distribution, or when it is not determined in step S82 that the content to be distributed this time is the first character content, the process returns. When content other than the first character content is distributed, game progress information according to the actor's motions, etc. is distributed every predetermined time (e.g., 1/60th of a second).

With reference to FIG. 28(B), in step S91, the user terminal 100 determines whether or not an operation to start viewing content has been performed based on an input operation on the touch screen 15. If it is determined that the operation to start viewing has not been performed, the process returns, and if it is determined that the operation to start viewing has been performed, the process proceeds to step S92. In step S92, the game progress information received from the game play terminal 300 is analyzed to determine whether or not the game progress information includes display mode information.

If it is determined that the display mode information is included, the process proceeds to step S93, where the display mode of the first object for the current distribution period is set based on the display mode information. That is, among the display modes of the decorative objects for the current distribution period, the display mode of the first object (i.e., the color of the shirt) is set to the display mode specified by the display mode information (one of white, black, red, green, blue, purple, etc.). When the process of step S93 is completed, or if it is not determined in step S92 that game progress information including the display mode information has been received, the process proceeds to step S94.

In step S94, a fixed display mode is set as the display mode of the decorative objects other than the first object among the decorative objects associated with the first character. Also, when viewing content in which a character other than the first character appears, in step S94, a fixed display mode is set as the display mode of the decorative objects (including the shirt) associated with a character other than the first character (e.g., the second character, etc.). That is, when the first character content is delivered, the decorative objects (blouse, trousers, shoes, etc.) other than the first object (shirt) among the decorative objects associated with the first character are set to a fixed display mode (fixed color). On the other hand, when content other than the first character content (e.g., the second character content) is delivered, all of the costume objects associated with the character other than the first character (e.g., the second character, etc.) are set to a fixed display mode.

The processing of steps S93 and S94 generates model data for the avatar object 610 to be operated within the virtual space 600B. While the content is being viewed, the avatar object 610 based on the model data moves based on the action instruction data included in the game progress information distributed from the game play terminal 300. When the processing of step S94 is completed, the process returns.

FIG. 29 shows an example of the display of objects including the first character while the first character content is being viewed. In FIG. 29, a wreath object reminiscent of a wreath is placed behind the first character. The first character moves based on the action instruction data.

The decorative objects associated with the first character include a shirt worn on the upper body, a blouse worn over the shirt, pants worn on the lower body, and shoes worn on the feet. In addition, the colors of the display modes of all decorative objects other than the first object, the shirt, are uniformly determined. For example, the color white is determined for the blouse and shoes, and the color gray is determined for the pants.

Meanwhile, for the shirt, which is the first object, a different color is set for each content distribution from the previous distribution, as shown in step S83 in FIG. 28. As the display mode of the shirt, in the Nth live distribution for the first character content, white is set as shown in FIG. 29(A). In addition, in the N+1th live distribution for the first character content, gray is set as shown in FIG. 29(B), and in the N+2th live distribution for the first character content, blue is set as shown in FIG. 29(C) (the stripes in the figure indicate blue). In this way, for the first character content, the same first character appears wearing a shirt of a different color for each content distribution.

<Effects of this embodiment>
According to this embodiment, an image of the virtual space 600B including the first character moving in conjunction with the movement of the performer is displayed based on the movement instruction data (first display data) distributed from the game play terminal 300. Here, the first character is associated with a predetermined first object (shirt), and the display mode of the first object (shirt color) is changed according to a predetermined rule. This allows the display mode of the first object to be changed spontaneously, and as a result, the user can feel the reality of the first character.

Furthermore, according to this embodiment, an image of the virtual space 600B including a second character that moves in conjunction with the movements of the performer is displayed based on the movement instruction data (second display data) distributed from the game play terminal 300. Here, the second character is associated with a second object (shirt) of the same type as the first object, and the display mode of the second object (shirt color) is fixed. This makes it possible to emphasize the realism of the first character.

Furthermore, according to this embodiment, the display mode of the first object is changed for each distribution period to a display mode that differs from the display mode in the previous distribution period. This gives the impression that the display mode of the first object is being intentionally changed for each distribution, and provides the user with the excitement of wondering in what display mode the object will appear in the current distribution period, thereby improving the viewer ratings of the distribution.

Furthermore, according to this embodiment, the first display data is data distributed from the game play terminal 300, and includes motion data corresponding to the motion input by the performer. A video of the first character moving within the virtual space 600B is displayed by moving the first character in real time within the virtual space 600B based on the motion data. This makes it possible to reduce the amount of data distributed.

Furthermore, according to this embodiment, the display mode of the first object changed according to a predetermined rule is specified by display mode information (first data) distributed from the game play terminal 300. The display mode of the first object is set based on the first data. This allows the display mode of the first object in multiple user terminals 100 to be uniformly controlled on the game play terminal 300 side.

Furthermore, according to this embodiment, a predetermined third object (blouse) is associated with the first character. The third object is an object that covers at least a portion of the first object. This makes it possible to control the manner in which the first object is displayed by the third object.

<Modification>
Modifications of the above-described embodiment are listed below.

(1) In the above embodiment, a normal viewing mode was described in which an avatar object 610, etc. is operated in the same virtual space 600B as the virtual space 600A, and a field of view image 660 of a field of view area 640B corresponding to the position, orientation, and tilt of a virtual camera 620B disposed in the virtual space 600B, which changes in response to a swipe operation on the touch screen 15 of the user terminal 100, is displayed on the touch screen 15 of the user terminal 100.

However, the space in which the avatar object 610 and the like are operated is not limited to a virtual space, and may be a real space. The real space may be, for example, a space specified by an acquired image photographed and acquired by the camera (imaging unit) 17. Specifically, for example, when the AR viewing mode is switched to, in the user terminal 100, a planar portion such as a floor surface is specified by analyzing the acquired image photographed and acquired by the camera 17, and the AR virtual space in which at least the avatar object 610 is located is placed at a predetermined position (for example, the center position of the planar portion) of the planar portion. In addition, the virtual camera 620B in this case is placed at a position that captures the AR virtual space from the same direction as the shooting direction of the camera 17 when the acquired image was photographed. As a result, an image in the augmented reality space obtained by superimposing the field of view image from the virtual camera 620B on the acquired image from the camera 17 is displayed on the display unit 152. Furthermore, the user terminal 100 detects the actual position, orientation, inclination, etc. of the camera 17, and changes the position, orientation, inclination, etc. of the virtual camera 620B according to the position, orientation, inclination, etc. This allows the captured image to change in response to changes in the position, orientation, tilt, etc. of the camera 17.

In a user terminal 100 in which the AR viewing mode is selected, the avatar object 610 and other objects move in an augmented reality space in which objects including the avatar object 610 are arranged in relation to an image acquired from the camera 17. This gives the user the impression (sense of realism) that the avatar object 610 and other objects are actually moving in the real space in front of the user.

Figure 30 shows an example in which a character modeled after a woman wearing a hat (hereinafter referred to as the first' character) appears in virtual space 600B. The first' character is associated with decorative objects such as a dress worn over the entire body of the first' character, a hat worn on the head, and earrings in the ears. In Figure 30, the earrings among the decorative objects associated with the first' character correspond to the first object. For this reason, the display mode of the earrings is set to a different display mode for each distribution by step S83 in Figure 28. Note that the second distribution content that is distributed periodically includes, for example, character content in which the first' character appears and in which the display mode of the first object is changed each time.

First, the normal viewing mode will be described with reference to FIG. 30(A). FIG. 30(A-1) is a display example when the first' character faces forward and holds up the index finger of the right hand in front. FIG. 30(A-2) is a display example when the first' character faces diagonally forward to the right and points forward with the index finger of the right hand. In the normal viewing mode, the position, direction, and inclination of the virtual camera 620B are changed in response to a swipe operation. However, the piercing is covered by a hat corresponding to the third object, and no matter how the swipe operation is performed, the virtual camera 320B cannot be moved to a position where the piercing is visible (for example, a position where the piercing can be displayed in FIG. 30(B-3) or FIG. 30(B-4) described later). In the normal viewing mode, as shown in FIG. 30(A), an AR mode transition icon 182 for switching to the AR viewing mode is normally displayed. In the normal viewing mode, a touch operation or the like on the AR mode transition icon 182 is detected, and the AR viewing mode can be switched to.

The AR viewing mode will be described with reference to FIG. 30(B). FIG. 30(B-1) and FIG. 30(B-2) respectively show display examples during the AR viewing mode at the same timing as FIG. 30(A-1) and FIG. 30(A-2). In the AR viewing mode, an image from the virtual camera 620B is displayed on the display unit 152 among images of the augmented reality space in which the first' character is positioned relative to the acquired image photographed and acquired by the camera 17. The first' character moves based on the action instruction data distributed from the game play terminal 300. The position, direction, inclination, etc. of the virtual camera 620B are changed according to the actual position, direction, inclination, etc. of the camera 17. In the AR viewing mode, the AR mode transition icon 182 is displayed in an emphasized manner (black and white inverted display in FIG. 30(B)) to notify that the AR viewing mode is in progress.

For this reason, if the position and orientation of the camera 17 is adjusted so that the camera is looking up at the left ear from diagonally below to the left of the 1' character, the virtual camera 320B moves to a position where the piercing is visible, and an image of the piercing is displayed on the display unit 152 (see (B-3) in FIG. 30(B)). That is, the display state of the piercing can be confirmed by selecting the AR viewing mode and adjusting the position and orientation of the camera 17. This can motivate the user to view in the AR viewing mode, and can improve the user's taste. Note that if the position and orientation of the camera 17 is adjusted so that the camera is looking up at the left half of the 1' character's body from the feet of the character, an image of the left half of the body including the feet is displayed on the display unit 152, as shown in (B-4) in FIG. 30(B).

In this way, in the AR viewing mode, by adjusting the position and orientation of the camera 17, the virtual camera 620B can be moved to a position where the earrings, which are the first object associated with the first' character, can be displayed (for example, a position where the first' character's neck is viewed from, or a position where the first' character's feet are viewed from, etc.). As a result, the first object, the earrings, can be displayed, and the user can confirm the display state of the earrings. On the other hand, in the normal viewing mode, as described above, no matter how the operation is performed, the virtual camera 620B cannot be moved to a position where the earrings, which are the first object associated with the first' character, can be displayed (for example, a position where the first' character's neck is viewed from, or a position where the first' character's feet are viewed from, etc.). As a result, the first object, the earrings, cannot be displayed, and the user cannot confirm the display state of the earrings.

In the AR viewing mode, the normal viewing mode can be switched to by detecting a touch operation or the like on the AR mode transition icon 182, which is displayed in an highlighted manner. Note that in the AR viewing mode, the AR mode transition icon 182 may be displayed in an highlighted manner and the AR mode reset icon may be displayed in a normal manner, and the AR mode may be reset (restarted) in response to a touch on the AR mode reset icon. When the AR mode is reset, a flat surface such as a floor surface is identified again, and a field of view image in the AR virtual space 600C is placed at a predetermined position on the flat surface.

When a tap operation is performed on the AR mode transition icon 182 displayed in an emphasized manner while the first object is displayed in the AR viewing mode, the viewpoint of the virtual camera 620B is returned to a predetermined viewpoint (for example, a viewpoint capturing the virtual space 600B from the front), or is changed to a viewpoint that minimizes the amount of movement of the virtual camera 620B within the range in which the virtual camera 620B can move in the normal mode. This makes it possible to minimize the shift in the viewpoint before and after the mode switch. Also, when the first object is displayed in the AR viewing mode (such as FIG. 30(B-3)), the AR mode transition icon 182 may be temporarily hidden.

In the AR viewing mode, the virtual camera 620B is controlled in synchronization with changes in the position, orientation, tilt, etc. of the camera 17. However, the position, orientation, tilt, etc. of the virtual camera 620B may be controlled in response to an input operation from the user (for example, a swipe operation, a touch operation, etc.). This allows the viewing area of the virtual camera 620B to be changed in response to an input operation from the user, and as a result, the viewing image arranged on the acquired image can be changed. Specifically, the avatar object, etc. arranged on the acquired image (for example, only the avatar object 610, the avatar object 610 and other objects, the entire virtual space) can be rotated or moved by a swipe operation. This makes it possible to rotate and move the avatar object 610, etc. by an input operation on the touch screen 15, even when an augmented reality space is generated and the avatar object 610, etc. is operated in a small space where the position, orientation, etc. of the user terminal 100 cannot be sufficiently adjusted.

Furthermore, as a viewing mode, a TV mode may be provided in which the position, direction, tilt, etc. of the virtual camera 620B is changed based on the operation of a switch on the system 1 operator side, and it may be possible to switch to this TV mode in response to a user's operation. However, in this TV mode, the virtual camera 320B cannot be moved to a position where the first object is visible, i.e., the first object cannot be displayed on the display unit 152.

(2) In the above embodiment, a video of the avatar object 610 in motion is played back using a real-time rendering method. However, the video may also be played back using a video distribution method. In this case, for the first character content, a first object whose display mode has been changed according to a predetermined rule is associated with a first character in the game play terminal 300, and video data including a video of the first character in motion within a virtual space is distributed from the game play terminal 300 as first display data. A video of the first character in motion within a virtual space based on the video data is displayed on the touch screen 15 of the user terminal 100.

(3) In the above embodiment, the display mode of the clothes of the avatar object 610 is uniformly determined according to the type of the avatar object 610. However, the display mode of the clothes of the avatar object 610 may be different for each user according to the progress of the game (for example, each time the game is won or an item is inserted). For example, even if the game progress information transmitted from the game play terminal 300 is the same, the clothes of the avatar object 610 may be clothes according to the progress of the game. In this case, multiple types of clothes data of the avatar object 610 may be stored in advance on the side of each user terminal 100. Note that the subject of the display mode of the avatar object 610 is not limited to the clothes, but may be hairstyle, skin color, degree of makeup, etc. As a result, the variation of the avatar object 610 can be increased without increasing the processing load on the game play terminal 300 that outputs the game progress information, which makes it possible to attract users.

Alternatively or in addition, the display mode of the avatar object 610 may be made different from the display mode displayed on the user terminals 100 of other users depending on the charge made by the user during the game. Also, the display mode of the avatar object 610 may be made different depending on the user ranking, which is updated depending on the game results. Furthermore, the user terminals 100 that can receive the first character content may be limited to some user terminals 100, such as the user terminals 100 of users who charge a large amount during the game.

(4) In the above embodiment, the color of the display mode of the first object (shirt) is changed by the processing of step S83. However, in step S83, instead of or in addition to the color, the shape of the first object may be changed. For example, the shirt may be changed from a V-neck to a crew neck, or the earrings may be changed from star-shaped to ring-shaped. Also, in step S83, instead of or in addition to determining the color, shape, type, etc., it may be determined whether the first object itself is displayed (for example, worn) or not (not worn).

(5) In the above embodiment, one of the decorative objects associated with the first character (e.g., a shirt, earrings) is set as the first object, and the display mode of the first object is changed. However, the first object is not limited to a shirt or earrings, and may be an object such as socks, pants, a hair clip, gloves, a hat, or underwear, as long as it is an object associated with the first character. In addition, the first object is not limited to a decorative object, and may be, for example, an object related to the first character itself (e.g., hairstyle, nails, etc.). Furthermore, in addition to changing the display mode of the first object associated with the first character, the display mode of an object in the background of the first character (e.g., a wreath object) may also be changed.

(6) In the above embodiment, the display mode of decorative objects other than the first object among those associated with the first character is fixed in advance. However, the display mode of decorative objects other than the first object may be changed, for example, every time a predetermined number of times (five times) is distributed, every season, every time period, etc., rather than for each distribution. Similarly, in the above embodiment, the display mode of decorative objects associated with the second character is fixed in advance, but the display mode may also be changed every time a predetermined number of times (five times) is distributed, every season, every time period, etc.

(7) In the above embodiment, an example was described in which the display mode of the first object associated with a character (such as the first character or the first' character) appearing in the second distribution content that is distributed periodically is changed for each distribution. However, the display mode of the first object associated with a character appearing in the first distribution content that is distributed irregularly may also be changed to a display mode different from that at the time of the previous distribution.

<Additional Notes>
The matters described in the above embodiments will be supplemented below.

(Note 1):
According to one aspect of an embodiment disclosed herein, a program executed in a terminal device (user terminal 100) having a processor causes the processor to execute the steps of: receiving (S26) first display data (movement instruction data) for enabling display of an image of a virtual space including a first character (avatar object 610) that moves in conjunction with the movements of an actor; and displaying (S29) an image of the first character moving within the virtual space based on the first display data, wherein the first character is associated with a predetermined first object, and the display manner (color) of the first object is changed according to a predetermined rule.

(Note 2):
In (Supplementary Note 1), the processor executes a step (S26) of receiving second display data for enabling display of an image of a virtual space including a second character (another avatar object 610) that moves in conjunction with the movements of an actor, and a step (S29) of displaying an image of the second character moving within the virtual space based on the second display data, wherein the second character is associated with a second object (a shirt) of the same type as the first object, and the display manner (color) of the second object is fixed.

(Appendix 3):
In (Supplementary Note 1) or (Supplementary Note 2), the display mode of the first object is changed for each distribution period to a display mode different from the display mode in the previous distribution period.

(Note 4):
In any of (Supplementary Note 1) to (Supplementary Note 3), the first display data is data distributed from outside and includes motion data corresponding to a motion input by an actor, and the displaying step causes the first character to move in real time within the virtual space based on the motion data, and displays an image of the first character moving within the virtual space.

(Note 5):
In (Supplementary Note 4), the processor is caused to execute a step (S92) of receiving first data (display mode information) for specifying the display mode of the first object changed according to the predetermined rule, and a step (S93) of setting the display mode of the first object based on the first data.

(Note 6):
In (Supplementary Note 4) or (Supplementary Note 5), the processor is caused to execute a step of setting one of a plurality of types of viewing modes (normal viewing mode, AR viewing mode) in which at least a portion of the aspect for identifying a viewpoint differs, and the displaying step displays an image of the virtual space as seen from a identified viewpoint, and when the viewing mode is a first viewing mode (AR viewing mode) of the plurality of types of viewing modes, the viewpoint can be moved to a position at which the first object is displayed, and when the viewing mode is a mode other than the first viewing mode, the viewpoint cannot be moved to the position at which the first object is displayed.

(Appendix 7):
In (Supplementary Note 6), the terminal device includes a detection unit that detects a position and orientation of the terminal device and a touch screen, the first viewing mode is a viewing mode in which a viewpoint can be identified based on the position and orientation detected by the detection unit, and the multiple types of viewing modes include a second viewing mode (normal viewing mode) that identifies a viewpoint based on a touch operation on the touch screen, and when in the second viewing mode of the multiple types of viewing modes, it is not possible to move the viewpoint to a position where the first object is displayed.

(Appendix 8):
In any one of (Supplementary Note 1) to (Supplementary Note 3), the first display data is data distributed from outside, and is video data including a video of a first character associated with a first object whose display mode has been changed according to the predetermined rule, moving within the virtual space, and the displaying step displays a video of the first character moving within the virtual space based on the video data.

(Appendix 9):
In any one of (Supplementary Note 1) to (Supplementary Note 8), a predetermined third object is associated with the first character, and the third object is an object that covers at least a portion of the first object.

(Appendix 10):
According to one aspect of an embodiment disclosed herein, a method executed in a terminal device having a processor includes the steps of receiving first display data for enabling display of an image of a virtual space including a first character that moves in conjunction with an actor's movements, and displaying an image of the first character moving within the virtual space based on the first display data, wherein the first character is associated with a predetermined first object, and the display manner of the first object is changed according to a predetermined rule.

(Appendix 11):
According to one aspect of an embodiment disclosed herein, a terminal device having a processor receives first display data for enabling display of an image of a virtual space including a first character that moves in conjunction with an actor's movements, and displays an image of the first character moving within the virtual space based on the first display data, wherein the first character is associated with a predetermined first object, and the display manner of the first object is changed according to a predetermined rule.

[Software implementation example]
The control blocks (particularly control units 110, 210, 310, 410) of the user terminal 100, server 200, game playing terminal 300 (HMD set 1000), and distribution terminal 400 may be realized by logic circuits (hardware) formed in an integrated circuit (IC chip) or the like, or may be realized by software.

In the latter case, the user terminal 100, the server 200, the game play terminal 300 (HMD set 1000), and the distribution terminal 400 are equipped with a computer that executes instructions of a program, which is software that realizes each function. This computer is equipped with, for example, one or more processors, and a computer-readable recording medium that stores the above program. The object of the present invention is achieved by the processor reading the above program from the recording medium and executing it in the above computer. The above processor can be, for example, a CPU (Central Processing Unit). The above recording medium can be a "non-transient tangible medium", such as a ROM (Read Only Memory), as well as a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, etc. In addition, a RAM (Random Access Memory) that expands the above program may also be included. The above program may be supplied to the above computer via any transmission medium (such as a communication network or broadcast waves) that can transmit the program. Note that one aspect of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the above program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention.

1 System, 2 Network, 3, 3A, 3B User (first user), 4 Player (performer), 10, 20, 30, 40 Processor, 11, 21, 31, 41 Memory, 12, 22, 32, 42 Storage, 13, 23, 33, 43 Communication IF, 14, 24, 34, 44 Input/output IF, 15, 45 Touch screen, 17 Camera, 18 Distance measurement sensor, 51 Monitor, 52 Gaze sensor, 53 First camera, 54 Second camera, 55 Microphone, 56 Speaker, 100, 100A, 100B, 100C User terminal (computer, first computer, first information processing device), 110, 210, 310, 410 Control unit (first control unit, second control unit), 111, 311, 413 Operation reception unit, 112, 312, 412 Display control unit, 113, 313 UI control unit, 114, 314 Animation generation unit, 115, 315 Game progress unit, 116, 316 Virtual space control unit, 117 Video playback unit, 120, 220, 320, 420 Memory unit (first memory unit, second memory unit), 131, 231, 331 Game program (program, first program), 132, 232, 332 Game information, 133, 233, 333 User information, 151, 451 Input unit, 152, 452 Display unit (display), 200 Server, 211 Communication mediation unit, 212 Log generation unit, 213 List generation unit, 234, 421 User list, 300 Game play terminal (external device, second external device), 317 Reaction processing unit, 400 Distribution terminal (external, first external device, computer, second information processing device), 411 Communication control unit, 414 Voice reception unit, 415 Motion identification unit, 416 Operation instruction data generation unit, 422 Motion list, 423 Distribution program (program, second program), 540, 1020, 1021 Controller, 500 HMD, 510 HMD sensor, 520 Motion sensor, 530 Display, 600A, 600B Virtual space, 610 Avatar object (character), 620A, 620B Virtual camera, 631, 632, 633, 634 Object, 640A, 640B View area, 650, 660 View image, 671 Enemy object, 672, 673 Obstacle object, 674 Performance object, 691, 692 Speech, 701, 702, 703A, 70B, 704A, 704B, 705, 706, 711, 711A, 711B, 711C, 711D, 722, 723, 745, 745A, 745B, 745C, 752, 762, 763, 930, 2011, 2022, 2031, 2032, 2033, 2034, 2037, 2038, 2051, 2063, 2072, 2073, 2075 UI image (message UI, UI), 721 Download screen, 731 User list screen (list), 732, 732A, 732B, 732C, 742, 742A, 742B, 742C Record image, 733, 733A, 733B, 733C User name, 734, 734A, 734B, 734C Tag information, 735, 735A, 735B, 735C Icon, 741 Motion list screen (options), 743, 743A, 743B, 743C Motion name, 744, 744A, 744B, 744C, 753 Motion image, 751 Distribution screen, 761 Distribution completion screen, 810A, 810B Motion video, 820A, 820B Speech voice, 910A, 910B Video, 920A, 920B Voice, 1000 HMD set, 1010 Object, 1030 Storage medium, 2010 Home screen, 2020 Ranking screen, 2021 Title image, 2026, 2026A, 2026B Ranking, 2027, 2027A, 2027B Charge amount, 2028, 2028A, 2028B Number of transmissions, 2029 Sent notification, 2030 Last transmission date, 2035 Detail display area, 2036 Scroll bar, 2040 Details screen, 2050, 2060 Preparation screen, 2052 Text, 2053, 2061 Selection image, 2054A, 2054B, 2054C, 2062A, 2062B, 2062C, 2062D, 2062E, 2062F Choices, 2070 Voice input screen, 2074 Tag image

Claims (2)

Computer,
a means for receiving first display data for enabling display of an image of a virtual space including a first character;
means for displaying an image of the first character moving within the virtual space based on the first display data;
a setting unit for setting one of a plurality of types of viewing modes including at least a first viewing mode and a second viewing mode, the modes being different in at least a part of the manner for identifying a viewpoint;
the display means displays an image of the virtual space seen from a specified viewpoint,
a predetermined first object is associated with the first character;
A program, wherein in the first viewing mode, the viewpoint can be moved to a position where the first object, which is covered by an object different from the first object and therefore hidden, is displayed, and in the second viewing mode, the viewpoint cannot be moved to a position where the first object is displayed, and the first viewing mode and the second viewing mode are switched between by accepting a user input operation. a display manner of the first object is changed regardless of the presence or absence of a request from a user, and is determined according to a type of the first object;
The program according to claim 1 , wherein the first object is identifiable by the user even if the first object is changed to any display mode in the first viewing mode.

Images