WO2024080135A1

WO2024080135A1 - Display control device, display control method, and display control program

Info

Publication number: WO2024080135A1
Application number: PCT/JP2023/034925
Authority: WO
Inventors: 悟郎高木; 智裕石井; 河野　温子
Original assignee: ソニーグループ株式会社
Priority date: 2022-10-13
Filing date: 2023-09-26
Publication date: 2024-04-18

Abstract

A display control device according to one aspect of the present disclosure comprises: an acquisition unit that acquires, as input information, a motion of a user in a real space; a determination unit that determines whether or not an input corresponding to a preregistered specific action is included in the input information; and a display control unit that reflects, in an avatar in a virtual space, the motion of the user corresponding to the input information, and that, if the input corresponding to the specific action is determined to be included in the input information, reflects the specific action in the avatar in the virtual space and displays the avatar in the virtual space.

Description

DISPLAY CONTROL DEVICE, DISPLAY CONTROL METHOD, AND DISPLAY CONTROL PROGRAM

The present disclosure relates to a display control device, a display control method, and a display control program for representing an avatar in virtual reality.

Various technologies have been put into practical use that present users with a three-dimensional virtual space constructed within a computer or computer network, known as the Metaverse.

For example, technology has been proposed that applies VR (Virtual Reality) and AR (Augmented Reality) technologies to expressions in virtual space, providing users with avatars that are easy to operate and enabling backgrounds to be displayed that are in line with reality (e.g., Patent Document 1).

International Publication No. 2014/119098

However, there is still room for improvement in the methods used to represent avatars in virtual spaces.

For example, in a virtual space, users can communicate intimately with each other using avatars. For this reason, it is desirable for the service provider of the virtual space to not only have the avatar reproduce realistic movements that correspond to reality, but also to provide avatars that can express a variety of emotions and movements that users can feel close to.

This disclosure therefore proposes a display control device, a display control method, and a display control program that can represent avatars in a virtual space in a variety of ways.

In order to solve the above problems, a display control device according to one embodiment of the present disclosure includes an acquisition unit that acquires user behavior in real space as input information, a determination unit that determines whether the input information includes an input corresponding to a specific action registered in advance, and a display control unit that reflects the user behavior corresponding to the input information in an avatar in a virtual space, and, if it is determined that the input information includes an input corresponding to the specific action, further reflects the specific action in an avatar in the virtual space and displays the avatar in the virtual space.

FIG. 2 is a diagram showing an overview of a display control process according to the embodiment. FIG. 11 is a diagram illustrating an example of a determination process according to the embodiment. FIG. 1 is a diagram illustrating an example of the configuration of a display control device according to an embodiment. FIG. 4 is a diagram illustrating an example of a gesture storage unit according to the embodiment. 5 is a flowchart showing a flow of a display control process according to the embodiment. FIG. 11 is a diagram (1) for explaining a first variation of the display control process according to the embodiment. FIG. 11 is a diagram (2) for explaining a first variation of the display control process according to the embodiment. FIG. 11 is a diagram (1) for explaining a second variation of the display control process according to the embodiment. FIG. 11 is a diagram (2) for explaining a second variation of the display control process according to the embodiment. FIG. 11 is a diagram (3) for explaining a second variation of the display control process according to the embodiment. FIG. 11 is a diagram for explaining a third variation of the display control process according to the embodiment; FIG. 11 is a diagram (2) for explaining a third variation of the display control process according to the embodiment. FIG. 11 is a diagram (3) for explaining a third variation of the display control process according to the embodiment. FIG. 4 is a diagram for explaining a third variation of the display control process according to the embodiment. FIG. 5 is a diagram for explaining a third variation of the display control process according to the embodiment; FIG. 6 is a diagram for explaining a third variation of the display control process according to the embodiment. FIG. 11 is a diagram for explaining a fourth variation of the display control process according to the embodiment; FIG. 13 is a diagram (2) for explaining a fourth variation of the display control process according to the embodiment. FIG. 11 is a diagram (3) for explaining a fourth variation of the display control process according to the embodiment. FIG. 11 is a diagram for explaining a fifth variation of the display control process according to the embodiment; FIG. 13 is a diagram (2) for explaining a fifth variation of the display control process according to the embodiment. FIG. 11 is a diagram (3) for explaining a fifth variation of the display control process according to the embodiment. FIG. 4 is a diagram for explaining a fifth variation of the display control process according to the embodiment. FIG. 5 is a diagram (5) for explaining a fifth variation of the display control process according to the embodiment. FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the display control device.

The following describes the embodiments in detail with reference to the drawings. Note that in each of the following embodiments, the same parts are designated by the same reference numerals, and duplicate descriptions will be omitted.

The present disclosure will be described in the following order.
1. Embodiment 1-1. Overview of display control processing according to embodiment 1-2. Configuration of display control device according to embodiment 1-3. Processing procedure according to embodiment 1-4. Variations of processing according to embodiment 1-4-1. Expression of excitement between avatars 1-4-2. Expression of avatar emotions 1-4-3. Visualization of conversation 1-4-4. Proximity effect between avatars 1-4-5. Expression related to display and interaction of avatars 1-5. Modifications 2. Other embodiments 3. Effects of a display control device according to the present disclosure 4. Hardware configuration

1. EMBODIMENTS
(1-1. Overview of Display Control Process According to the Embodiment)
An example of a display control process according to the embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an overview of the display control process according to the embodiment.

The display control process according to the embodiment is executed by a display control device 100, not shown in FIG. 1. The display control device 100 is an information processing terminal for implementing VR and AR technology. That is, in the embodiment, the display control device 100 displays a three-dimensional virtual space (hereinafter simply referred to as "virtual space") constructed in a computer or computer network, such as a metaverse, on a display unit (display, etc.). The display control device 100 also displays an avatar, which is a character that resembles a user, in the virtual space. For example, the display control device 100 is implemented by any information processing device, such as a PC (Personal Computer), a tablet terminal, or a smartphone. The display control device 100 may be, for example, a head mounted display (HMD) or AR glasses.

For example, the display control device 100 captures an image of a user in real space with an image sensor (such as a camera) and reflects the user's gestures and facial expressions in an avatar in the virtual space. This allows the user to experience the sensation of having an alter-ego of himself or herself present in the virtual space. Furthermore, since multiple avatars (in other words, multiple users) can exist simultaneously in the virtual space, users can communicate via the avatars in the same way as in the real space. For example, users can deepen their interactions with other users by greeting them and conversing (voice chat or text chat) via their avatars.

Because virtual spaces can be expressed in a variety of ways, they can be constructed not only as realistic spaces based on real space, but also as spaces with cartoon-like expressions that users can relate to. In this case, it is also desirable for the provider of the virtual space to not only be able to make the avatar reproduce realistic movements based on reality, but also to be able to provide a variety of emotional expressions and diverse movements that users can relate to.

The display control device 100 according to the present disclosure realizes the above processing by the configuration shown below. For example, the display control device 100 acquires user behavior in real space as input information, and determines whether the input information includes an input corresponding to a specific action registered in advance. The display control device 100 then reflects the user behavior corresponding to the input information in an avatar in a virtual space, and if it is determined that the input information includes an input corresponding to a specific action, it further reflects the specific action in the avatar in the virtual space.

Specifically, the display control device 100 registers in advance comical movements including cartoon expressions, such as a thumbs-up, a signature pose of each avatar, and a high-five, as specific movements (hereinafter referred to as "gestures") to be performed by the avatar. When the display control device 100 receives an input from the user of an action for activating the gesture, the display control device 100 reflects the gesture, rather than the user's movement itself, in the avatar. Note that when the display control device 100 does not receive a gesture from the user, the display control device 100 may reflect the user's gesture itself in the avatar. In this way, the display control device 100 reflects both gestures associated with the user's conscious movements and the user's unconscious movements in the avatar. This allows the display control device 100 to display an avatar in a virtual space that not only imitates the user's movements, but also has expressive and comical movements. Through such expressions, the display control device 100 can build a virtual space that makes it easier for users to communicate with each other and to feel familiar with each other.

The above display control process will be explained below with reference to FIG. 1. As shown in the left part of FIG. 1, the display control device 100 displays an avatar 10A that exists in a virtual space on an example display screen 20. At this time, the display control device 100 uses a camera to constantly capture images of the user corresponding to the avatar 10A (i.e., the user whose movements and expressions are reflected in the avatar 10A).

Then, the display control device 100 acquires input information input by the user based on image data obtained from the camera. Note that input information is information acquired from various sensors including the camera, and is a general term for information used for displaying avatars, etc.

As an example, the display control device 100 acquires the user's facial expression 21 as input information using an image-based facial expression extraction technique. The display control device 100 also acquires the user's skeleton 22 using an image-based skeleton estimation technique. The display control device 100 also acquires gesture information 23 indicating the movement of the user's arms and hands using hand tracking technique. Note that the display control device 100 may acquire not only image data, but also voice or text information input by the user as input information. The display control device 100 then combines this information to accept input information from the user to be reflected in the avatar 10A.

For example, based on the user's input information, the display control device 100 acquires the avatar 10A nodding, shaking its head to indicate refusal, tilting its head in distress, etc. as unconscious actions of the user that are not specific actions. The display control device 100 may also acquire various emotional expressions by the user, such as a laughing expression or expression, or a surprised expression or expression, as unconscious actions. In this disclosure, these are collectively referred to as "unconscious actions." The example on the left side of Figure 1 shows an example in which the unconscious action of the user raising his hand is reflected in the avatar 10A and displayed in the virtual space.

On the other hand, the display control device 100 can also determine, based on the acquired image data, whether or not the user is performing a specific action to trigger a gesture. That is, the display control device 100 determines whether or not the input information includes an input corresponding to a specific action that has been registered in advance. For example, as shown in the right part of FIG. 1, the display control device 100 displays an avatar 10B that exists in a virtual space in a screen display example 25. At this time, the display control device 100 uses a camera to constantly capture images of the user corresponding to the avatar 10B (i.e., the user whose movements and expressions are reflected in the avatar 10B).

In the example of Figure 1, avatar 10A and avatar 10B are drawn separately to make the comparison easier to understand, but in reality, unconscious actions and conscious actions are expressed by the same avatar. Hereinafter, when there is no need to distinguish between avatar 10A, avatar 10B, etc., they will be collectively referred to simply as "avatar 10."

When the user performs a specific action, the display control device 100 accepts a request to transition to an input mode for determining the specific action. For example, when the user desires that the avatar 10B make a distinctive gesture based on a specific action by the user, the user requests the display control device 100 to transition to an input mode for consciously making the gesture.

Then, the user makes a characteristic movement, such as "taking a pre-registered pose with his index finger and thumb in front of the camera." Similar to the left part of FIG. 1, the display control device 100 acquires the user's behavior, such as the user's facial expression 26, the user's skeletal structure 27, and gesture information 28, based on the captured image. The display control device 100 then combines the acquired information to determine whether or not the characteristics of the user's behavior match the characteristics for activating a pre-registered gesture. Note that various known technologies may be used for acquiring the user's behavior and determining the gesture.

When the display control device 100 determines that the behavior input by the user is a movement that has enough characteristics to trigger a gesture, it reflects the gesture in the avatar 10B. For example, as shown in the right part of FIG. 1, the avatar 10B makes a pose with its index finger and thumb. At this time, the display control device 100 may perform a comical expression that includes a dramatic effect such as displaying radial lines around the avatar 10B along with the pose. This expression allows the user to distinguish whether the movement of the avatar 10B is a specific gesture registered in advance or the user's movement itself. In other words, such specific gestures are consciously made to be performed by the avatar 10B by the user. For this reason, in this disclosure, these are collectively referred to as "conscious movements."

Then, the display control device 100 can reflect both these unconscious and conscious actions in the avatar 10 and express them as reactions of the avatar 10. This allows the display control device 100 to display the avatar 10 in the virtual space in a variety of expressions, including not only a direct reflection of the user's movements, but also cartoon-like expressions, etc.

As described above, the user can switch between the input modes of unconscious actions and conscious actions, and reflect input information from unconscious actions or conscious actions in the avatar 10. Such switching may be performed, for example, based on an explicit designation by the user (manual operation such as pressing a button, voice input, etc.). Furthermore, the display control device 100 can set in advance a spatial range within which the user will perform conscious actions, and switch the input mode based on such a setting. This point will be explained using FIG. 2. FIG. 2 is a diagram showing an example of the determination process according to the embodiment.

FIG. 2 shows an example in which a user inputs his/her own behavior by being imaged by the camera 141 of the display control device 100. At this time, the user makes hand signs 30 in a pre-set space 31 (e.g., a space within a predetermined distance from the camera 141). The display control device 100 determines whether the user's behavior was performed in the space 31, for example, using image analysis or a depth sensor. If the user's behavior was performed in the space 31, the display control device 100 considers the behavior to be one that ignites a conscious action and determines that a gesture has been ignited. On the other hand, the display control device 100 reflects the user's behavior observed in space 32, which is a range outside the space 31, in the avatar 10 as an unconscious action.

In other words, when the display control device 100 is located at a predetermined distance from the camera 141, it does not determine that the input from the user is an action for igniting a gesture, but only determines whether the user's actions performed in the space 31 are actions for igniting a gesture. This allows the user to reflect unconscious and conscious actions in the avatar 10 at will, without having to explicitly specify the switching of input modes.

Note that the switching of input modes is not limited to the example shown in FIG. 2. For example, when a user is engaged in voice chat (i.e., voice input), the display control device 100 may automatically switch to an input mode for extracting unconscious actions (bone detection). Alternatively, when a user is engaged in text chat (i.e., manual character input), the display control device 100 may automatically switch to an input mode for extracting conscious actions (gesture detection).

In addition, the display control device 100 may perform motion blending that reflects both movements while switching modes. In addition, the display control device 100 may turn off the input mode itself while the user is performing a UI (User Interface) operation, so that movements of different intentions are not reflected in the avatar 10. Alternatively, the display control device 100 may activate the gesture input mode only within a predetermined time after performing an arbitrary key operation. In addition, the display control device 100 may prevent erroneous gesture input by requesting the user to input a predetermined word when performing a gesture. In addition, the display control device 100 may activate the gesture input mode when a mute button that turns off voice input from the user is pressed, and may set the display control device 100 to the unconscious action input mode in other cases. In addition, when the unconscious action input mode is turned off, the display control device 100 may simultaneously turn off the voice input and transition to a privacy mode (wherein input from the user is not accepted). In this way, the display control device 100 may use different input modes in response to various aspects.

As described above with reference to Figures 1 and 2, the display control process according to the embodiment can reflect unconscious and conscious actions in the avatar 10 as desired, allowing the avatar 10 to be displayed in a variety of ways.

(1-2. Configuration of the display control device according to the embodiment)
Next, a configuration of the display control device 100 will be described. Fig. 3 is a diagram showing an example of the configuration of the display control device 100 according to an embodiment. The display control device 100 is one of the components of a display control system 1 according to the embodiment shown in Fig. 3. The display control system 1 includes the display control device 100 and an analysis server 50.

The analysis server 50 performs various analysis processes described below and provides the analysis results to the display control device 100. The analysis server 50 performs information processing with a relatively high processing load, such as text-based sentiment analysis and machine learning processing for image recognition, which are not performed by the display control device 100. For example, the analysis server 50 is a cloud server managed by a business operator that operates a service related to virtual space.

As shown in FIG. 3, the display control device 100 has a communication unit 110, a memory unit 120, a control unit 130, a sensor unit 140, and a display unit 150. The display control device 100 may also have an input unit (such as a keyboard or a touch panel) that accepts various operations from a user who operates the display control device 100.

The communication unit 110 is realized, for example, by a NIC (Network Interface Card) or a network interface controller. The communication unit 110 is connected to the network N by wired or wireless means, and transmits and receives information to and from the analysis server 50, etc., via the network N. The network N is realized, for example, by a wireless communication standard or method such as Bluetooth (registered trademark), the Internet, Wi-Fi (registered trademark), UWB (Ultra Wide Band), or LPWA (Low Power Wide Area).

The storage unit 120 is realized, for example, by a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk.

The storage unit 120 stores various information related to the display control process according to the embodiment. In the embodiment, the storage unit 120 includes a gesture storage unit 121.

The gesture storage unit 121 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the gesture storage unit 121 according to an embodiment. As shown in FIG. 4, the gesture storage unit 121 has items such as "gesture ID," "gesture," and "determination item."

"Gesture ID" is identification information for identifying a gesture. "Gesture" indicates the specific content of the gesture. "Determination item" indicates the item for extracting a feature amount used when determining whether or not the user's behavior will trigger the gesture. Although not shown in FIG. 4, the information on each gesture may include motion data (such as animation of the skeleton and facial expressions, dramatic effects, etc.) for moving the 3D model of the avatar 10 in accordance with the gesture.

For example, the example shown in Figure 4 indicates that the gesture with gesture ID "A01" corresponds to a "thumbs up," and whether or not to fire that gesture is determined by the user's "hand position" and "arm movement."

Referring back to FIG. 3, the explanation will continue. The sensor unit 140 is a sensor that detects various environmental information. For example, the sensor unit 140 includes a camera 141, which is an imaging device that is composed of an outward-facing camera that captures images of the outside of the display control device 100, an inward-facing camera that captures images of the user, etc.

For example, the camera 141 recognizes a subject (e.g., a user located in real space) located in front of the display control device 100. In this case, the camera 141 acquires an image of the user who is the subject, and can calculate the distance from the display control device 100 (in other words, the camera 141) to the subject based on the parallax between images captured by the stereo camera and an analysis of the captured image data. Alternatively, the sensor unit 140 may detect the distance in real space using a depth sensor capable of detecting the distance to any subject, such as a user.

The sensor unit 140 also includes a microphone 142 that can measure the user's voice and surrounding sounds. For example, when engaging in voice chat, the user can input voice by using the microphone 142 provided in the display control device 100.

Furthermore, for example, when the display control device 100 is a wearable device such as an HMD, the sensor unit 140 may have a function of detecting various information related to the user's motion, such as the orientation, inclination, motion, and moving speed of the user's body, in addition to the camera 141 and microphone 142. Specifically, the sensor unit 140 detects information related to the user's motion, such as information related to the user's head and posture, the motion of the user's head and body (acceleration and angular velocity), the direction of the field of view, and the speed of the viewpoint movement. For example, the sensor unit 140 functions as various motion sensors such as a three-axis acceleration sensor, a gyro sensor, and a speed sensor, and detects information related to the user's motion. More specifically, the sensor unit 140 detects at least one change in the position and posture of the user's head by detecting the components of the yaw direction, pitch direction, and roll direction as the motion of the user's head.

The sensor unit 140 does not necessarily have to be provided in the display control device 100, and may be, for example, an external sensor connected to the display control device 100 by wire or wirelessly. For example, the camera 141 does not necessarily have to be provided inside the display control device 100, and may be an imaging device installed outside the display control device 100.

The display unit 150 displays various information output from the control unit 130. For example, the display unit 150 is a display that outputs video to the user. The display unit 150 may also include an audio output unit (such as a speaker) that outputs audio.

The control unit 130 is realized, for example, by a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU, etc., executing a program stored inside the display control device 100 (for example, a display control program related to the present disclosure) using a RAM or the like as a working area. The control unit 130 is also a controller, and may be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

As shown in FIG. 3, the control unit 130 has an acquisition unit 131, a determination unit 132, and a display control unit 133.

The acquisition unit 131 acquires various types of information. For example, the acquisition unit 131 acquires input information to be reflected in the avatar 10 by sensing the user using the sensor unit 140. Specifically, the acquisition unit 131 acquires the user's behavior in real space as input information.

The acquisition unit 131 may also acquire various operations and setting requests from the user. For example, the acquisition unit 131 accepts designation from the user as to whether or not to reflect a specific action in the avatar 10. In other words, the acquisition unit 131 accepts designation from the user as to whether to set an input mode for inputting an unconscious action or an input mode for inputting a conscious action (gesture).

Furthermore, when the input information is acquired, the acquisition unit 131 may acquire surrounding information, etc., using the sensor unit 140, etc. Specifically, as shown in FIG. 2, the acquisition unit 131 acquires the spatial position from the camera 141 when the user's behavior is captured. This allows the determination unit 132, which will be described later, to determine at what distance from the camera 141 the hand sign performed by the user is captured, and therefore the mode can be automatically switched to the gesture mode.

The acquisition unit 131 can acquire information not only about the user currently captured by the camera 141, but also about multiple users connected to the virtual space. In other words, the acquisition unit 131 can acquire the behavior of multiple users as input information. By using such input information from multiple users, the determination unit 132 can determine the level of interest in a conversation between multiple users, as described below.

The acquisition unit 131 may also receive various requests sent from the user via the UI. For example, the acquisition unit 131 receives requests to change the appearance of the avatar, and various operations for services related to the virtual space, such as purchasing, selecting, or exchanging an object (such as an item in the virtual space) linked to the avatar.

The determination unit 132 performs various determinations for controlling the display of the avatar 10. For example, the determination unit 132 determines whether or not the input information from the user includes an input corresponding to a specific action that has been registered in advance. The input corresponding to a specific action that has been registered in advance is, in other words, a registered behavior that the user uses to trigger a gesture.

For example, when the user specifies an input mode (gesture mode), the determination unit 132 determines whether or not the input information includes an input corresponding to a specific action. In other words, the determination unit 132 determines whether or not the behavior performed by the user in front of the camera 141 is an intentional action that triggers a gesture.

Furthermore, when the mode is changed according to the distance, the determination unit 132 does not determine whether or not the input information includes an input corresponding to a specific action if the position from the camera 141 is farther than a predetermined distance, but determines whether or not the input information includes an input corresponding to a specific action if the position from the camera 141 is within the predetermined distance. In other words, the determination unit 132 performs gesture determination for behaviors such as hand signs made by the user in a position relatively close to the camera 141, and does not perform gesture determination for behaviors such as hand signs made by the user outside the set spatial range, determining them to be unconscious actions. This allows the determination unit 132 to freely switch input modes to express the avatar 10.

The display control unit 133 controls the display unit 150 to display information output from the control unit 130. In other words, the display control unit 133 outputs the virtual space image rendered as video content to the display. Note that the output destination device is not limited to the built-in display, but may be other information devices such as a smartphone, or a video output device such as a television.

For example, based on input information acquired from the user, such as an image of the user, a voice uttered by the user, or text entered by the user, the display control unit 133 reflects the user's behavior corresponding to the input information in the avatar 10 in the virtual space. Furthermore, when the display control unit 133 determines that the input information includes an input corresponding to a specific action, it reflects a gesture, which is a specific action, in the avatar 10 in the virtual space and displays the avatar 10 in the virtual space.

For example, when the display control unit 133 is in a mode for inputting unconscious actions, it reflects the user's skeletal data, etc., acquired based on an image of the user, in the avatar 10. On the other hand, when the display control unit 133 is in a gesture mode, if the determination unit 132 determines that the user's behavior will trigger a gesture, it reflects a gesture, including a specific animation or special effects, that has been registered in advance, in the avatar 10. That is, gestures made by the avatar 10 are expressed by animations that involve special movements or special effects, such as a thumbs-up, high-five, or clapping. Note that the display control unit 133 may control the sound output by the avatar (pre-registered laughter) in addition to the gestures.

(1-3. Processing Procedure According to the Embodiment)
Next, a process procedure according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the flow of a display control process according to the embodiment.

As shown in FIG. 5, the display control device 100 uses the sensor unit 140 to acquire an image of the user (step S11). As described above, the display control device 100 may also acquire the user's voice, input text, and the like along with the image.

Then, the display control device 100 converts the user's image into skeletal data based on known posture estimation techniques, etc. (step S12).

Then, the display control device 100 determines whether the current mode is the gesture mode (step S13). If the current mode is the gesture mode (step S13; Yes), the display control device 100 determines the feature point positions and movements in the skeletal data (step S14).

Then, the display control device 100 determines whether or not the feature point position and movement correspond to a registered gesture (step S15). If they correspond to a registered gesture (step S15; Yes), the display control device 100 reads out the motion data of the gesture from the storage unit 120 (step S16). The display control device 100 then reflects the read out motion data in the avatar 10 (step S18).

If the mode is not the gesture mode (step S13; No), the display control device 100 does not perform gesture determination, and converts the skeletal data obtained from the user into skeletal data for the avatar 10 (step S17). Then, the display control device 100 reflects the skeletal data in the avatar 10 (step S18).

Also, in step S15, if the feature point positions and movements do not correspond to a gesture (step S15; No), the display control device 100 also converts the skeletal data obtained from the user into skeletal data for the avatar 10 (step S17). Then, the display control device 100 reflects the skeletal data in the avatar 10 (step S18).

Then, the display control device 100 displays the avatar 10 in the virtual space based on the information reflected in the avatar 10 (step S19). The display control device 100 repeats this display control process for each predetermined frame acquired by the sensor unit 140.

(1-4. Variations of Processing According to the Embodiment)
(1-4-1. Expressing excitement between avatars)
The above-described expressions of the avatar 10 based on unconscious and conscious actions may be combined with various other expressions. Variations of the display control process according to the embodiment will be described below.

First, the first variation will be described. In the first variation, the display control device 100 displays various visual effects according to the excitement between multiple avatars. Note that the excitement between avatars means the excitement between users. In this way, the display control device 100 can express in a way that other users can understand that active communication is taking place between multiple users in the virtual space.

That is, in the first variation, the acquisition unit 131 acquires the behavior of multiple users as input information. Then, the determination unit 132 determines the level of excitement of the interaction between the multiple users in the virtual space based on the behavior of the multiple users. Furthermore, the display control unit 133 displays a presentation effect in the vicinity of the multiple users according to the level of excitement of the interaction.

Specifically, the determination unit 132 determines the level of excitement in an exchange based on the number of users participating in the exchange, the frequency of conversations between users, and the degree of overlap in the voices of multiple users.

The display control unit 133 may also determine the expression or frequency of occurrence of the dramatic effect based on the level of excitement in the interaction, the body orientation of the avatar 10, and the content of the conversation between multiple users.

The first variation will be specifically described below with reference to FIG. 6. FIG. 6 is a diagram (1) for explaining the first variation of the display control process according to the embodiment.

The example shown in FIG. 6 shows that multiple avatars, avatar 10A, avatar 10B, and avatar 10C, are actively interacting by exchanging greetings and conversations. At this time, the display control device 100 determines the level of excitement in the interaction between the multiple users, and displays an arc-shaped effect 200 and radial effect 201 around the avatars based on the determined level of excitement. In this way, the display control device 100 can perform a display that allows multiple avatars who are actively interacting to be distinguished simply by looking at the situation from a distance, even if they are not actually participating in the conversation.

The level of excitement between avatars may be derived from various parameters. For example, if the excitement between avatars is due to voice communication such as voice chat, parameters that may be used include the volume, speed, pitch, emotion, frequency of overlap, amount of filler, and the presence or absence of specific language expressions of the users' voices. In other words, the level of excitement in communication in a virtual space is not necessarily measured only by the volume of the voices or the amount of conversation.

For example, in a stereophonic environment such as a virtual three-dimensional space, simultaneous speech is possible just as in a real space. For this reason, the display control device 100 can use, for example, "overlap (simultaneous speech by multiple people)" as a parameter with high importance (weighted) for determining excitement.

The display control device 100 may also analyze the emotion of each voice by having the analysis server 50 or the like analyze the voices, and may calculate a high excitement level when a more emotional voice is included. Specifically, the display control device 100 may calculate a high excitement level for an exchange that includes shouts expressing admiration or strong emotion, such as "Yay (Japanese)" or "Wow! Yeah! (English)." Alternatively, the display control device 100 may calculate a high excitement level based on the frequency of occurrence of greetings or words (positive words based on emotion analysis) that strongly imply an intention of celebration or joy, such as "Cheers!".

The display control device 100 can also change the presentation (visual) effects displayed depending on the level of excitement.

For example, the display control device 100 may change the size or frequency of the dramatic effect represented by the arc depending on the level of excitement. Specifically, when the level of excitement is relatively high, the display control device 100 may display a large arc so as to cover all of the multiple users participating in the conversation. This allows the display control device 100 to accurately convey the excitement of the multiple users even to a user who is viewing the multiple users from a distance.

The display control device 100 may also display a large effect even if conversation is not necessarily active. For example, if multiple users are paying attention to some kind of event in the virtual space, conversation will be reduced, but it is determined that the users are actively interacting with each other. In this case, the display control device 100 may determine the level of excitement based on, for example, the intersection of the avatars' gazes or interactions other than conversation (such as the closeness of the users to each other). Then, depending on the level of excitement, the display control device 100 displays the arc-shaped or radial effect larger or brighter than usual. In this way, the display control device 100 can show other users that multiple users are actively interacting with each other, even without conversation.

Next, an example of a process in which the display control device 100 determines the level of excitement will be shown with reference to FIG. 7. FIG. 7 is a diagram (2) for explaining a first variation of the display control process according to the embodiment.

FIG. 7 shows a composite waveform 205 obtained from a conversation between multiple users. In FIG. 7, range 206 shows a waveform in which the voices of multiple users are measured to be high, indicating that the conversation is taking place at a relatively loud volume. Range 207 shows a waveform in which multiple voices from multiple users are synthesized, indicating that a lively exchange of conversation is taking place.

The display control device 100 determines the level of excitement among multiple users based on such waveform analysis. For example, the display control device 100 determines the level of excitement based on the characteristics of volume changes over a specified period of time. Alternatively, the display control device 100 may estimate the number of participants in a conversation based on waveform analysis, and determine the level of excitement to be higher the greater the estimated number of participants.

The display control device 100 may also change the representation of the dramatic effect based on the waveform. For example, the display control device 100 may change the color or density (frequency of occurrence) of an arc-shaped or radial display, the speed at which the arc-shaped or radial light spreads, etc., in accordance with the amplitude of the waveform.

In addition, the display control device 100 may not necessarily display an arc-shaped effect, but may display a speech bubble with a series of acute angles according to the voice, conversation content, or emotion. This allows the display control device 100 to express, for example, that multiple users are arguing with each other. In this case, the display control device 100 may display speech bubbles or other dramatic effects using a three-dimensional display (application of a rim shader) that is valid regardless of the viewing angle.

The display control device 100 may also change the speed at which the special effects are generated depending on the distance from the observer to the multiple users. Specifically, the display control device 100 may display the special effects slower the farther the distance is, and may display the special effects faster the closer the distance is. This allows the observing user to know at a glance how far away the actively interacting users are.

(1-4-2. Avatar Emotion Expression)
Next, a second variation will be described. In the second variation, when the display control device 100 acquires an input that strongly indicates an emotion, such as a laughing voice of the user, and determines that the user is laughing, the display control device 100 causes the avatar 10 to make a laughing gesture.

In other words, in the second variation, the determination unit 132 determines whether or not the input information from the user includes an input corresponding to a specific action linked to a specific emotion, such as laughing. If it is determined that the input information includes a specific action linked to a specific emotion, the display control unit 133 reflects the specific action linked to the specific emotion in the avatar 10.

At this time, the determination unit 132 may compare the input information with voiceprint information that is registered in advance by the user and is associated with a specific emotion, to determine whether the input information includes an input corresponding to a specific action associated with the specific emotion.

The display control unit 133 may also display an avatar 10 that reflects a specific action linked to a specific emotion, and may also display a dramatic effect linked to the specific emotion near the avatar 10.

The second variation will be described in detail with reference to FIG. 8 and subsequent figures. FIG. 8 is a diagram (1) for explaining the second variation of the display control process according to the embodiment.

In the example shown in FIG. 8, the avatar 10 is making a gesture associated with "laughing," which is an example of a specific emotion. At this time, the display control device 100 may display a cartoon-like effect 210 associated with laughter around the avatar 10. In other words, the display control device 100 can display an avatar 10 that is rich in emotional expression by having the avatar 10 perform a laughing gesture in response to the user laughing.

The display control device 100 can determine that the user has laughed using several techniques. This will be explained with reference to FIG. 9. FIG. 9 is a diagram (2) for explaining a second variation of the display control process according to the embodiment.

FIG. 9 shows an example of an audio waveform 215 obtained from a user. In the audio waveform 215, the measurement value obtained when the environmental sound (background noise) is measured while the user is engaged in voice chat or the like is set as measurement result 216. In addition, in the audio waveform 215, the audio waveform obtained when the user is engaged in voice chat or the like is set as measurement result 217. At this time, the peak of measurement result 217 is indicated by the line of result 218. And the difference between result 218 and measurement result 216 is indicated by difference 219.

In other words, the display control device 100 uses the microphone 142 to measure the environmental sound (background noise) when the user is engaged in voice chat, etc. Then, the display control device 100 measures the user's voice, estimates the peak volume as laughter, and measures the difference 219 with the environmental sound. In this way, the display control device 100 registers the peak value of the voice and the difference value with the environmental sound for each user.

Then, when the user next makes a sound and a sound exceeding (or equal to) the difference 219 is observed, the display control device 100 determines that the user has laughed. According to this method, the display control device 100 can determine whether or not the user has laughed based on the sound volume.

As another method, the display control device 100 can learn voiceprint information when the user laughs and make a determination based on the learning results. This will be explained using FIG. 10. FIG. 10 is a diagram (3) for explaining a second variation of the display control process according to the embodiment.

FIG. 10 illustrates an example of a voice waveform 220 acquired from a user. Within the voice waveform 220, range 221 is the waveform when the user's laughter is measured. The display control device 100 (or the analysis server 50) accumulates the waveforms when laughter is measured for each user, and learns the characteristics of the accumulated waveforms using a known voice learning method (e.g., deep learning that extracts characteristic patterns). This allows the display control device 100 to generate a learned voiceprint pattern that indicates the "laughing" state for each user.

Then, the next time the user speaks and the voice waveform is acquired, the display control device 100 can determine whether the voice contains laughter by comparing it with the learned voiceprint pattern. This allows the display control device 100 to determine with high accuracy whether the user has laughed.

It should be noted that human laughter is diverse, and the emotions contained in laughter include not only joy but also various other emotions. For this reason, in a machine learning model that determines laughter through learning, the display control device 100 may generate a model that categorizes emotions, etc., rather than a binary determination of whether or not someone is laughing.

Also, there are several possible methods for acquiring laughter as training data. For example, the display control device 100 acquires a waveform indicating laughter based on the user's explicit designation. Specifically, the display control device 100 acquires the user's voice when the user presses an arbitrary button to cause the display control device 100 to acquire (record) laughter. At this time, the user laughs at will. This allows the display control device 100 to acquire a voiceprint pattern indicating the user's laughter (i.e., training data labeled "laughter"). Alternatively, the display control device 100 may acquire the user's voiceprint pattern so as to accumulate only the user's laughter using a general-purpose machine learning model for voice recognition that determines laughter. In general, the characteristics of laughter vary from person to person, so it is desirable for the display control device 100 to acquire and learn a voiceprint pattern for each individual user.

The display control device 100 may acquire voice based on various emotions of the user, not limited to laughter. Then, the display control device 100 can cause the avatar 10 to express the user's unique emotional expression by reflecting the expression based on the acquired voice in the avatar 10.

As an example, the display control device 100 may acquire the filler characteristics of each user and reflect the characteristics of the user who utters the filler in the avatar 10. Fillers such as "umm" and catchphrases such as "that's nice" uttered by a user can be said to indicate the characteristics of that user. For this reason, the display control device 100 learns the fillers and catchphrases of each user, and when a filler or catchphrase is observed, reflects it in the avatar 10, thereby reproducing the characteristics of that user in a virtual space. In this case, the display control device 100 can better reflect the characteristics of the user in the avatar 10 by emphasizing and cartoonish expressions, such as by expressing the sounds uttered as fillers or catchphrases in 3D using speech bubbles.

(1-4-3. Visualization of conversation)
Next, a third variation will be described. In the third variation, a method will be described in which, when the display control device 100 displays a conversation between avatars in a virtual space, not only is text indicating the conversation content displayed in a typical speech bubble, but the conversation content is expressed in a variety of ways.

For example, in a third variation, the display control unit 133 may display words uttered by the avatar 10 based on the input information, and may also display a history of words that transitions upward from the avatar.

In addition, the display control unit 133 may display the words uttered by the avatar 10 as icons that pictorially represent the meanings contained in the input information.

The determination unit 132 may also determine the distance between the position in the virtual space where the display of the words is observed (i.e., the position where the observer avatar is located) and the position of the avatar 10 that utters the words. In this case, the display control unit 133 displays the word history or icons as a dramatic effect that does not include the meaning of the words and icons, based on the determined distance. Note that, as will be described later, an example of a dramatic effect that does not include meaning is when no text or the like indicating the content of a conversation is displayed, and a history of lively conversation is simply displayed as a smoke signal or balloon.

The determination unit 132 may also determine the positional relationship between the position in the virtual space where the display of the words is observed and the position of the avatar 10 that speaks the words. In this case, the display control unit 133 may determine the angle and three-dimensional display of the display of the words observed in the virtual space based on the determined positional relationship.

As described above, in the third variation, the display control device 100 displays an icon or a conversation that includes visual effects, rather than a typical conversation displayed as text, etc. This allows the display control device 100 to display, in an easily observable manner, an active conversation taking place between avatars, etc.

The third variation will be described in detail using FIG. 11 and subsequent figures. FIG. 11 is a diagram (1) for explaining the third variation of the display control process according to the embodiment.

The example shown in FIG. 11 shows that avatar 10A is having a conversation with the observer avatar, saying, "There's a movie I want to see!" Since avatar 10A is having the conversation in the vicinity of the observer, the conversation is displayed as text in speech bubble 230.

On the other hand, the content of a conversation that avatar 10B is having, which is located farther away than avatar 10A, is shown by icon 231. Specifically, the display control device 100 performs a content and emotion analysis of the conversation of avatar 10B using the analysis server 50, and displays the analysis results as icon 231. In the example of FIG. 11, icon 231 indicates that avatar 10B had a conversation about going to the movies, then getting in a car, and then going out for dinner.

In this way, for conversations between avatars 10B that are farther away from the observer than a predetermined distance, the display control device 100 displays the conversation content as icons (images, etc.) rather than displaying the text as is. This allows the observer to grasp the general content of the conversation and how lively it is, without having to decipher the detailed text information of distant conversations.

Furthermore, the display control device 100 can display conversations between avatars in an animation in which past history scrolls upward, as in icon 231. This allows the display control device 100 to communicate the flow of the conversation to other users in a visually easy-to-understand manner, and also allows users who are far away to understand how lively the conversation is.

This point will be described in detail with reference to FIG. 12. FIG. 12 is a diagram (2) for explaining a third variation of the display control process according to the embodiment.

Figure 12 shows multiple avatars 235 having a conversation at a great distance from the observer. At this time, the conversation between the multiple avatars 235 is displayed without displaying any semantic content such as text or icons, and only the conversation history 236 in the form of speech bubbles or circles (such displays are sometimes called "text balloons").

As described above, the conversation history 236 is animated and displayed upwards in the order of the most recent conversations. Therefore, in the example of FIG. 12, the conversations between multiple avatars 235 are expressed like smoke signals in the virtual space. This allows even a distant observer to know where in the virtual space an active conversation is taking place.

The display control device 100 can also change the expression method in various ways depending on the distance between the observer and the avatar 10. This will be explained using FIG. 13. FIG. 13 is a diagram (3) for explaining a third variation of the display control process according to the embodiment.

Figure 13 shows how a conversation taking place near the observer (240), a conversation taking place at a medium distance from the observer (241), and a conversation taking place very far from the observer (242) are each expressed differently.

For example, in conversation 240, the content of the avatars' conversation is displayed in text, and the details of the conversation are displayed. Note that conversation 240 may also display image information shared between users, not just text, such as images sent by users during text chat. This allows an observer to get a rough idea of what the communication is about, without having to read the text.

For example, in conversation 241, the text display of the avatars' conversation is omitted, and icons and the like are inserted between the conversation so that the observer can grasp the general content of the conversation. For example, in conversation 241, when two avatars are conversing, the observer can grasp the conversation content to the extent of which avatar is speaking. Note that older history in conversation 241 may be displayed in such a way that the lines of the avatars merge with each other as they move upward.

For example, in conversation 242, the display of the meaning of the avatar's conversation, such as text and icons, is omitted, and only the conversation history is displayed like a smoke signal. At this time, the display control device 100 may display the history more transparently the earlier it is (i.e., the conversation history further up), like a smoke signal in real space. The display control device 100 may also display the display showing each line of dialogue smaller than

conversations

240 and 241. In other words, the display control device 100 can change the display area, such as a speech bubble for displaying dialogue, depending on the distance.

In this way, by changing the way conversation is expressed depending on the distance, the display control device 100 can provide the user with a UI that does not appear cluttered even when lively communication is taking place in the virtual space.

The display control device 100 can also change the expression method in various ways depending on the positional relationship between the observer and the avatar 10. This will be explained using FIG. 14. FIG. 14 is a diagram (4) for explaining a third variation of the display control process according to the embodiment.

The left diagram in Figure 14 shows a conversation 250 taking place near the observer. Specifically, in conversation 250, avatar 10A and avatar 10B are having a conversation. Furthermore, conversation 250 includes a conversation history 251 of avatar 10A and a conversation history 252 of avatar 10B.

The right diagram in FIG. 14 shows an overhead view 255 of the conversation 250 as seen from above. As shown in the overhead view 255, avatar 10A and avatar 10B are not facing each other as seen by an observer 256, but are standing at a slight angle.

At this time, as shown in the left diagram of FIG. 14, the display control device 100 displays the conversation history 251 at an angle based on the positional relationship between the observer 256 and the avatar 10A (here, the angle between the line of sight of the observer 256 and the direction of the avatar 10A). Similarly, the display control device 100 displays the conversation history 252 at an angle based on the positional relationship between the observer 256 and the avatar 10B.

This representation allows the observer 256 to recognize the positional relationship with avatar 10A and avatar 10B simply by looking at the conversation history.

The display control device 100 may also perform display including dramatic effects not only on the angle but also on the content of the conversation itself, such as text. This point will be explained using FIG. 15. FIG. 15 is a diagram (5) for explaining a third variation of the display control process according to the embodiment.

In FIG. 15, avatar 10A and avatar 10B are having a conversation, and text 260 indicating the content of the conversation, production effect 261, and text 262 are displayed.

For example, the text 260 may include highlighting, such as being displayed in bold compared to other text. For example, the display control device 100 automatically highlights when the user utters a pre-registered word that strongly indicates an emotion, such as "yabai (crazy)."

The effect 261 is a character icon including a three-dimensional display. For example, the effect 261 is automatically displayed when the user utters a pre-registered word that strongly indicates an emotion, such as "yabai (crazy)."

The text 262 is also text that includes a three-dimensional display. For example, the text 262 is displayed based on a predetermined criterion, such as when the user utters a pre-registered word that strongly indicates an emotion, such as "surprise," or when the user's voice volume is higher than normal.

These special text displays and special effects may be displayed based on, for example, sentiment analysis of user conversations. For example, the display control device 100 cooperates with the analysis server 50 to determine whether the content entered in the text chat is positive or negative using sentiment analysis. The display control device 100 then automatically generates appropriate visual effects for the speech bubbles in which the text is displayed. Specifically, if the conversation is positive, the display control device 100 may display speech bubbles or text in bright colors or with animations of bouncing letters, or the like. Alternatively, if the conversation is negative, the display control device 100 may display speech bubbles in cool colors or with animations of crying text, or the like.

These speech bubbles and text may also be displayed in a history display (known as a word cloud, for example) in which the size of the text varies depending on the frequency of the words that appeared in the conversation.

These text displays and the like may also have three-dimensional physical characteristics. This will be explained using FIG. 16. FIG. 16 is a diagram (6) for explaining a third variation of the display control process according to the embodiment.

FIG. 16 shows examples of text 270 and text 275 with three-dimensional display. Text 270 is configured so that the characters appear to be floating from the speech bubble. Therefore, in display 271 when text 270 is viewed from the side, the characters appear to be floating from the speech bubble.

The text 275 is also configured in such a way that the characters appear to be contained within a speech bubble. For this reason, when the text 275 is viewed from the side, the display 276 shows the characters as if they were contained within a cloud-like speech bubble. For this reason, to the observer, the characters appear to be displayed as two-dimensional plates, and the speech bubble appears to be displayed in a three-dimensional cloud or balloon-like shape.

The 3D display shown in Figure 16 etc. can also be displayed backwards. In this case, the images and text in the 3D display are displayed with the left and right reversed.

In this way, the display control device 100 can express text in conversations in a variety of ways, providing visual entertainment to the user.

(1-4-4. Proximity effect between avatars)
Next, a fourth variation will be described. In the fourth variation, an example will be shown in which the display control device 100 performs an expression that encourages interaction when avatars come close to each other.

For example, in a fourth variation, the determination unit 132 may determine whether the first avatar is included within a range recognized by the second avatar. Then, when it is determined that the first avatar is included within a range recognized by the second avatar, the display control unit 133 may adjust the movement path of the first avatar so that the first avatar approaches the second avatar.

The determination unit 132 may also determine whether or not the first avatar has approached the second avatar in the virtual space. When it is determined that the first avatar has approached the second avatar, the display control unit 133 reflects a specific action corresponding to the approach in at least one of the first avatar and the second avatar.

In other words, in the fourth variation, the display control device 100 encourages interaction between users by using special displays and actions so that avatars can recognize each other's presence and communicate with each other in close proximity.

The fourth variation will be described in detail using FIG. 17 and subsequent figures. FIG. 17 is a diagram (1) for explaining the fourth variation of the display control process according to the embodiment.

The first diagram in FIG. 17 shows avatar 10D approaching a location where avatar 10A, avatar 10B, and avatar 10C are having a conversation. At this time, avatar 10A recognizes that avatar 10D has entered a range where proximity is determined, either through initially set viewpoint recognition or by other means.

When avatar 10A recognizes the proximity of avatar 10D as shown in FIG. 17 (FIG. 2), it initiates a specific action. For example, avatar 10A performs a gesture of greeting by raising its hand. At this time, display control device 100 may automatically display a friendly message or the like to encourage avatar 10A to join the conversation with avatar 10D.

When avatar 10D responds to the greeting and indicates its intention to join the conversation with avatar 10A, etc., avatar 10D joins the conversation, as shown in FIG. 3 in FIG. 17.

In these operations, the display control device 100 performs corrections so that the gaze of the avatar 10A and the avatar 10D meets, and causes the avatar 10A and the avatar 10D to perform a greeting gesture. In this way, the display control device 100 can allow the avatars to actively interact with each other.

For such automatic conversation participation, the display control device 100 performs control processing based on several conditions. For example, the display control device 100 determines that one or more avatars are already in a conversation (chat) state. The display control device 100 also determines that another avatar has approached the area where the conversation is taking place. When these conditions are met, the display control device 100 corrects the display so that the avatars' viewpoints are more easily aligned, or causes them to perform gestures.

This point will be explained using FIG. 18. FIG. 18 is a diagram (2) for explaining a fourth variation of the display control process according to the embodiment.

As shown in FIG. 18, the display control device 100 recognizes a proximity area 280 of avatar 10A and a proximity area 281 with avatar 10D. The display control device 100 also recognizes a distance 282 between avatar 10A and avatar 10D. When the distance 282 becomes equal to or smaller than a predetermined distance, the display control device 100 corrects the positions of avatar 10A and avatar 10D so that their lines of sight meet, for example by making a correction so that their bodies face each other.

Then, the display control device 100 recognizes that the proximity area 280 of the avatar 10A and the proximity area 281 of the avatar 10D have come into contact. In this case, the display control device 100 causes the avatar 10A and the avatar 10D to make gestures to greet each other.

Furthermore, when avatar 10A and avatar 10D enter within each other's proximity area, display control device 100 starts a conversation such as a voice chat between the two. At this time, display control device 100 newly recognizes conversation area 283 in which the conversation between avatar 10A and avatar 10D takes place.

In addition, when users make eye contact or make a greeting gesture and indicate their intention to refuse to have a conversation, the display control device 100 may correct the positions of their avatars so that they are not close to each other.

The display control device 100 may also perform control to prioritize voice chat when avatars are physically close to each other. This allows the display control device 100 to create an environment in which a newly joining avatar can easily call out to the unspecified number of avatars already participating in the conversation. The display control device 100 may also prioritize text chat when the user has muted voice input.

Furthermore, the display control device 100 may pre-set areas where conversation is encouraged and areas where conversation is prohibited for control that encourages interaction by bringing avatars close to each other. This point will be explained using FIG. 19. FIG. 19 is a diagram (3) for explaining a fourth variation of the display control process according to the embodiment.

FIG. 19 shows a situation in which some kind of event is taking place in a virtual space. For example, the event involves a specific avatar appearing on stage 290. In this case, the display control device 100 sets conversation-prohibited areas 292 and conversation-prohibited areas 293 in the passageways leading to stage 290 or in locations away from stage 290, so that the avatar does not remain in the passageways leading to stage 290 or in locations away from stage 290. On the other hand, the display control device 100 sets conversation areas 294 around stage 290, etc., to actively encourage conversation around stage 290 and around monument 291.

In this case, the display control device 100 does not make any corrections to allow avatars to start a conversation in the conversation prohibited area 292 or the conversation prohibited area 293. On the other hand, the display control device 100 makes corrections to allow avatars to start a conversation in the conversation area 294. This allows the display control device 100 to actively guide conversations between avatars in areas in the virtual space where conversation is encouraged.

The display control device 100 may correct the movement path of the avatar 10D so that the avatar 10D can easily participate in the conversation. For example, the display control device 100 may correct the walking path or walking speed so that the path along which the avatar 10D is walking automatically approaches an area close to the avatar 10A or an area where the avatars 10A and the like are having a conversation. The display control device 100 may also correct the body orientation so that the avatar 10D can easily make eye contact with the avatar 10A while walking.

(1-4-5. Expressions related to avatar display and interaction)
Next, a fifth variation will be described. In the fifth variation, an example is shown in which the display control device 100 changes the appearance of the avatar 10 and displays a UI for exchanging items for changing the appearance between avatars.

For example, in the fifth variation, the acquisition unit 131 acquires a request from the user to change the appearance of the avatar 10. The display control unit 133 then reflects the change based on the request in the avatar 10, changing the appearance of the avatar 10 in the virtual space along with a dramatic effect.

The acquisition unit 131 may also acquire from the user a request to exchange any one of a plurality of objects linked to the first avatar with a second avatar. In this case, when the request is acquired, the display control unit 133 displays a list of the plurality of objects linked to the first avatar near the first avatar.

The acquisition unit 131 may also acquire an instruction from the user to select one of the objects displayed in a list to be exchanged for the second avatar. In this case, when the display control unit 133 acquires the instruction, it displays one of the objects so that it floats from the list display, and moves the floating object in the virtual space so that it overlaps with the second avatar in accordance with the operation received from the user.

In this way, in the fifth variation, the display control device 100 performs display control such as changing the appearance of the avatar 10 and actively encouraging the exchange of items between avatars 10. This allows the display control device 100 to provide visual entertainment for the user and actively encourage interaction between users.

The fifth variation will be described in detail using FIG. 20 and subsequent figures. FIG. 20 is a diagram (1) for explaining the fifth variation of the display control process according to the embodiment.

In the first diagram of FIG. 20, an example is shown in which a UI displays candidate options 300 for changing the appearance of avatar 10A. In this example, it is assumed that the user has selected candidate 301.

The second diagram in FIG. 20 shows the display control device 100 changing the appearance of the avatar 10A. When changing the appearance of the avatar 10A, the display control device 100 may display a dramatic effect 302 in which the avatar 10A appears to be covered in smoke.

FIG. 3 in FIG. 20 shows the state in which the display control device 100 has changed the appearance of the avatar 10A. The display control device 100 displays the changed appearance of the avatar 10A in the virtual space. In this way, the user can arbitrarily change the appearance of the avatar 10A.

In the virtual space, a user can exchange items with other users to change the appearance of the avatar 10. This will be explained using FIG. 21. FIG. 21 is a diagram (2) for explaining a fifth variation of the display control process according to the embodiment.

FIG. 21 shows the UI display when avatar 10A and an observer exchange items. Avatar 10A presents exchange candidate items 310 to the observer. The observer also presents exchange candidate items 311. When the observer selects exchange button 312, the items are exchanged. Note that in virtual space, there may be a concept of ownership that indicates which user (avatar) owns each item. The display control device 100 may perform internal processing so that such ownership is also exchanged when the items are handed over.

Such item exchanges between avatars may be displayed in a different UI. This will be explained with reference to FIG. 22. FIG. 22 is a diagram (3) for explaining a fifth variation of the display control process according to the embodiment.

FIG. 22 shows a state in which avatar 10A and avatar 10B are about to exchange item 320. In this case, the user corresponding to avatar 10A selects item 320 on the screen display using a pointing device such as a mouse.

Then, the display control device 100 enlarges the display around the item 320. In the enlarged display, the user can move the item 320 closer to the avatar 10B by, for example, moving their arm in real space. In this case, the display control device 100 displays a bone display 321 of the avatar 10A that corresponds to the user's arm, and moves the item 320 displayed near the arm of the avatar 10A toward the avatar 10B.

When the item 320 is brought closer than a predetermined distance to the bone display 322 of the avatar 10B, the display control device 100 determines that the item 320 has been transferred. The display control device 100 then displays the item 320 near the bone display 322 and transfers ownership of the item 320 to the avatar 10B.

In this way, the display control device 100 can complete the item exchange in a way that makes it appear as if avatars are actually handing over items to each other, rather than displaying a UI that indicates the item exchange.

Furthermore, as described above, when an item is used for the purpose of changing the appearance of the avatar 10, the display control device 100 may display a UI corresponding to the part to be changed. This point will be explained using FIG. 23. FIG. 23 is a diagram (4) for explaining a fifth variation of the display control process according to the embodiment.

As shown in FIG. 23, the avatar 10A has a head 330, a face 332, a body 334, and feet 338 set as parts of the appearance that can be changed. In this case, the user can display a list of items that can be changed in each part. For example, the display control device 100 displays an item candidate 331 that corresponds to the head 330. Similarly, the display control device 100 displays an item candidate 333 that corresponds to the face 332, an item candidate 335 that corresponds to the body 334, and an item candidate 337 that corresponds to the feet 338. The display control device 100 can also display an item candidate 339 for changing the overall appearance of the avatar 10A.

In other words, the display control device 100 displays a pop-up display of target item candidates for each part of the avatar 10A where a change in appearance is requested. Then, the display control device 100 selects an item for changing the appearance of the avatar 10A in accordance with the user's operation.

Note that, when displaying the UI of an item, the display control device 100 may, for example, superimpose items present in a lower hierarchical level on a selected item. For example, when glasses are selected as an item for changing the face portion 332 of the avatar 10A, the display control device 100 may further display multiple candidate items included in the glasses category. In this case, the user can select one item from the candidate items 333, for example, in a single stroke, and then select an item in the lower hierarchical level.

The display control device 100 may also set an appropriate shortcut depending on the type of device used by the user when selecting an item. For example, when selecting an item, the display control device 100 may accept input by key operation on a keyboard or the like, in addition to input by a pointing device such as a mouse or by the user's gestures.

The display control device 100 may also control the delivery of items using a UI display. This will be described with reference to FIG. 24. FIG. 24 is a diagram (5) for explaining a fifth variation of the display control process according to the embodiment.

The example shown in FIG. 24 shows a state in which the user selects item 341 from among item candidates 340 held by avatar 10A. For example, the user selects item 341 by pressing and holding a finger or a pointing device on the screen. In this case, the display control device 100 transitions item 341 to a display in which it appears to be floating above item candidates 340 (floating display).

The user drags the floating item 341 to the avatar 10B. With this operation, the display control device 100 determines that the item 341 has been handed over to the avatar 10B. In this way, the display control device 100 can perform processes such as item exchange between users using various UIs and screen displays. Note that in the embodiment, the things exchanged between avatars in the virtual space are described as items, but the form that the items take can be arbitrarily set by the service side providing the virtual space. For example, the items may be data such as NFT (Non-Fungible Token).

(1-5. Modified Examples)
The processing according to the above embodiment may involve various modifications. For example, each device in Fig. 3 conceptually shows a function in the display control system 1, and may take various forms depending on the embodiment.

For example, in the above embodiment, an example was shown in which the display control device 100 performs processing in cooperation with the analysis server 50. However, the display control device 100 may also perform the processing performed by the analysis server 50 on its own device.

The display control device 100 may also be composed of multiple devices. For example, the display control device 100 may be divided into a first information processing device that acquires the user's behavior as an image and processes the acquired image, and a second information processing device that processes input of the user's voice, text, etc. In this case, the first information processing device acquires the user's behavior as an image, performs posture estimation based on the image, and performs hand-raising determination, head direction determination, hand sign determination, etc. The second information processing device may also perform laughter determination based on voice and emotion analysis based on text.

The display control device 100 may also be a device in which the display unit and the information processing unit are configured separately. In this case, the information processing unit of the display control device 100 may be any information processing device such as a server or a PC.

2. Other Embodiments
The processing according to each of the above-described embodiments may be implemented in various different forms other than the above-described embodiments.

Furthermore, among the processes described in each of the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

Furthermore, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

Furthermore, the above-mentioned embodiments and variations can be combined as appropriate to the extent that they do not cause any contradictions in the processing content.

Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be present.

(3. Effects of the display control device according to the present disclosure)
As described above, the display control device according to the present disclosure (the display control device 100 in the embodiment) includes an acquisition unit (the acquisition unit 131 in the embodiment), a determination unit (the determination unit 132 in the embodiment), and a display control unit (the display control unit 133 in the embodiment). The acquisition unit acquires a user's behavior in the real space as input information. The determination unit determines whether or not the input information includes an input corresponding to a specific action registered in advance. The display control unit reflects the user's behavior corresponding to the input information in an avatar in a virtual space, and when it is determined that the input information includes an input corresponding to a specific action, the display control unit further reflects the specific action in the avatar in the virtual space and displays the avatar in the virtual space.

In this way, the display control device according to the present disclosure displays an avatar by combining unconscious actions that accompany the user's natural movements with conscious actions to activate gestures. This allows the display control device to display an avatar in a virtual space not only in a way that directly reflects the user's movements, but also in a variety of expressions, including cartoon-like expressions, etc.

The acquisition unit also receives a designation from the user as to whether or not a specific action should be reflected in the avatar. If there is a designation from the user, the determination unit determines whether or not the input information includes an input corresponding to the specific action.

The acquisition unit also acquires the position from the imaging device when the user's behavior is captured. If the position from the imaging device is farther than a predetermined distance, the determination unit does not determine whether the input information includes an input corresponding to a specific action, and if the position from the imaging device is within the predetermined distance, the determination unit determines whether the input information includes an input corresponding to the specific action.

In this way, the display control device according to the present disclosure reflects the conscious action in the avatar when the user specifies that a conscious action should be performed or when the user makes a gesture in a specific area. This allows the display control device to display the avatar in accordance with the user's intention.

The acquisition unit also acquires the behavior of the multiple users as input information. The determination unit determines the level of excitement of the interaction between the multiple users in the virtual space based on the behavior of the multiple users. The display control unit displays a presentation effect in the vicinity of the multiple users according to the level of excitement of the interaction.

The determination unit also determines the level of excitement in the exchange based on the number of users participating in the exchange, the frequency of conversation between users, and the degree of overlap in the voices of multiple users.

The display control unit also determines the expression or frequency of occurrence of the dramatic effect based on the level of excitement in the interaction, the orientation of the avatar's body, and the content of the conversation between multiple users.

In this way, the display control device according to the present disclosure can visually display the excitement of a conversation between avatars, effectively showing even users who are not participating in the conversation how the avatars are actively interacting with each other.

The determination unit also determines whether or not the input information includes an input corresponding to a specific action linked to a specific emotion. If it is determined that the input information includes a specific action linked to a specific emotion, the display control unit reflects the specific action linked to the specific emotion in the avatar and displays the avatar in the virtual space.

The determination unit also compares the voiceprint information registered in advance by the user, which is associated with a specific emotion, with the input information to determine whether the input information includes an input corresponding to a specific action associated with the specific emotion.

The display control unit also displays an avatar that reflects a specific action linked to a specific emotion, and displays a dramatic effect linked to the specific emotion near the avatar.

In this way, the display control device according to the present disclosure can reflect gestures associated with specific emotions, such as laughter, in an avatar. This allows the display control device to express a wide variety of emotions in virtual space, just as it does in real space.

The display control unit also displays the words spoken by the avatar based on the input information, and displays the history of those words in an upward transition starting from the avatar.

The display control unit also displays the words uttered by the avatar as icons that pictorially represent the meaning contained in the input information.

The determination unit also determines the distance between the position in the virtual space where the display of the words is observed and the position of the avatar uttering the words. Based on the determined distance, the display control unit displays the word history or an icon as a dramatic effect that does not include the meaning of the words and icons.

The determination unit also determines the positional relationship between the position in the virtual space where the display of the words is observed and the position of the avatar uttering the words. The display control unit determines the angle and three-dimensional display of the display of the words observed in the virtual space based on the determined positional relationship.

In this way, the display control device according to the present disclosure can present the conversation history in various ways, without cluttering the screen display, and allows the user to see at a glance where active communication is taking place.

The determination unit also determines whether the first avatar is included within a range recognized by the second avatar in the virtual space. If it is determined that the first avatar is included within a range recognized by the second avatar, the display control unit adjusts the movement path of the first avatar so that the first avatar approaches the second avatar.

The determination unit also determines whether or not the first avatar has approached the second avatar in the virtual space. When it is determined that the first avatar has approached the second avatar, the display control unit displays the first avatar and the second avatar while reflecting a specific action corresponding to the approach of at least one of the first avatar and the second avatar.

In this way, the display control device according to the present disclosure can correct the behavior of avatars, thereby encouraging the avatars to actively interact with each other. This allows the display control device to increase opportunities for conversations between users in the virtual space, and stimulate interaction in the virtual space.

The acquisition unit also acquires a request from the user to change the appearance of the avatar. The display control unit reflects the changes based on the request in the avatar, and changes the appearance of the avatar in the virtual space along with the dramatic effects.

The acquisition unit also acquires a request from the user to exchange any one of the multiple objects linked to the first avatar for the second avatar. When the request is acquired, the display control unit displays a list of the multiple objects linked to the first avatar near the first avatar.

The acquisition unit also acquires an instruction from the user to select one of the objects displayed in the list to be exchanged for the second avatar. When the display control unit acquires the instruction, it displays one of the objects in question so that it floats from the list display, and moves the floating object in the virtual space so that it overlaps with the second avatar in accordance with the operation received from the user.

In this way, the display control device according to the present disclosure provides a UI with excellent operability, thereby reducing stress for users when interacting with each other, such as exchanging items. This allows the display control device to support active interaction between users.

(4. Hardware Configuration)
Information devices such as the display control device 100 according to each embodiment described above are realized by a computer 1000 having a configuration as shown in Fig. 25, for example. The display control device 100 will be described below as an example. Fig. 25 is a hardware configuration diagram showing an example of a computer 1000 that realizes the functions of the display control device 100. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input/output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

The CPU 1100 operates based on the programs stored in the ROM 1300 or the HDD 1400 and controls each component. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

The ROM 1300 stores boot programs such as the Basic Input Output System (BIOS) that is executed by the CPU 1100 when the computer 1000 starts up, as well as programs that depend on the hardware of the computer 1000.

HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a display control program related to the present disclosure, which is an example of program data 1450.

The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

The input/output interface 1600 is an interface for connecting the input/output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input/output interface 1600. The CPU 1100 also transmits data to an output device such as a display, edger, or printer via the input/output interface 1600. The input/output interface 1600 may also function as a media interface that reads programs and the like recorded on a specific recording medium. Examples of media include optical recording media such as DVDs (Digital Versatile Discs) and PDs (Phase change rewritable Disks), magneto-optical recording media such as MOs (Magneto-Optical Disks), tape media, magnetic recording media, and semiconductor memories.

For example, when the computer 1000 functions as the display control device 100 according to the embodiment, the CPU 1100 of the computer 1000 executes a display control program loaded onto the RAM 1200 to realize the functions of the control unit 130, etc. Also, the display control program according to the present disclosure and data in the storage unit 120 are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via the external network 1550.

The present technology can also be configured as follows.
(1)
an acquisition unit that acquires user behavior in a real space as input information;
a determination unit that determines whether or not the input information includes an input corresponding to a specific action that has been registered in advance;
a display control unit that reflects a behavior of the user corresponding to the input information in an avatar in a virtual space, and, when it is determined that the input information includes an input corresponding to the specific action, further reflects the specific action in an avatar in the virtual space and displays the avatar in the virtual space;
A display control device comprising:
(2)
The acquisition unit is
accepting a designation from the user as to whether or not the specific action is to be reflected in the avatar;
The determination unit is
determining whether or not an input corresponding to the specific action is included in the input information when the specific action is designated by the user;
The display control device according to (1).
(3)
The acquisition unit is
Acquire a position from an imaging device when the behavior of the user is imaged;
The determination unit is
When the position from the imaging device is farther than a predetermined distance, it is not determined whether or not the input corresponding to the specific action is included in the input information, and when the position from the imaging device is within a predetermined distance, it is determined whether or not the input corresponding to the specific action is included in the input information.
The display control device according to (1) or (2).
(4)
The acquisition unit is
Obtaining the behavior of multiple users as input information,
The determination unit is
determining a degree of excitement of an interaction between the plurality of users in the virtual space based on the behavior of the plurality of users;
The display control unit is
displaying a performance effect in the vicinity of the plurality of users according to the level of excitement of the exchange;
The display control device according to any one of (1) to (3).
(5)
The determination unit is
determining a level of excitement in the exchange based on the number of users participating in the exchange, the frequency of conversations between the users, and the degree of overlap of voices between the multiple users;
The display control device according to (4).
(6)
The display control unit is
determining an expression or occurrence frequency of the performance effect based on the level of excitement of the exchange, the body orientation of the avatar, and the content of the conversation between the plurality of users;
The display control device according to (5) above.
(7)
The determination unit is
determining whether an input corresponding to the specific action associated with a specific emotion is included in the input information;
The display control unit is
when it is determined that the input information includes a specific action associated with the specific emotion, reflecting the specific action associated with the specific emotion in the avatar and displaying the avatar in the virtual space;
A display control device according to any one of (1) to (6).
(8)
The determination unit is
comparing the input information with voiceprint information registered in advance by the user, the voiceprint information being associated with the specific emotion, to determine whether or not the input information includes an input corresponding to a specific action associated with the specific emotion;
The display control device according to (7) above.
(9)
The display control unit is
displaying the avatar reflecting a specific action associated with the specific emotion, and displaying a performance effect associated with the specific emotion in the vicinity of the avatar;
The display control device according to (7) or (8).
(10)
The display control unit is
displaying words uttered by the avatar based on the input information, and displaying a history of the words in an upward transition manner starting from the avatar;
A display control device according to any one of (1) to (9).
(11)
The display control unit is
displaying the words uttered by the avatar as icons pictorially representing the meanings contained in the input information;
The display control device according to (10).
(12)
The determination unit is
determining a distance between a position in a virtual space where the display of the words is observed and a position of an avatar uttering the words;
The display control unit is
displaying the history of the words or the icons as a dramatic effect not including the meaning of the words and the icons based on the determined distance;
The display control device according to (11).
(13)
The determination unit is
determining a positional relationship between a position in a virtual space where the display of the words is observed and a position of an avatar uttering the words;
The display control unit is
determining an angle and a three-dimensional representation of the representation of the words observed in a virtual space based on the determined positional relationship;
The display control device according to (11) or (12).
(14)
The determination unit is
determining whether a first avatar is included within a range recognized by a second avatar in the virtual space;
The display control unit is
When it is determined that the first avatar is included in a range recognized by the second avatar, a movement path of the first avatar is adjusted so that the first avatar approaches the second avatar;
The display control device according to any one of (1) to (13).
(15)
The determination unit is
determining whether the first avatar has come close to the second avatar in the virtual space;
The display control unit is
when it is determined that the first avatar is in proximity to the second avatar, displaying the first avatar and the second avatar by reflecting a specific action corresponding to the proximity in at least one of the first avatar and the second avatar;
The display control device according to (14).
(16)
The acquisition unit is
receiving a request from the user to change an appearance of the avatar;
The display control unit is
reflecting the change based on the request in the avatar, thereby changing the appearance of the avatar in the virtual space together with a dramatic effect;
The display control device according to any one of (1) to (15).
(17)
The acquisition unit is
receiving from the user a request to exchange any one of a plurality of objects associated with a first avatar with a second avatar;
The display control unit is
When the request is received, a list of a plurality of objects associated with the first avatar is displayed in the vicinity of the first avatar.
The display control device according to any one of (1) to (16).
(18)
The acquisition unit is
obtaining, from the user, an instruction to select one of the plurality of objects displayed in the list to be exchanged for the second avatar;
The display control unit is
When the instruction is acquired, the display device displays any one of the objects in a floating manner from the list display, and moves the floating object in a virtual space so as to overlap the second avatar in accordance with an operation received from the user.
The display control device according to (17).
(19)
The computer
The user's behavior in the real world is acquired as input information,
determining whether the input information includes an input corresponding to a specific action registered in advance;
reflecting the behavior of the user corresponding to the input information in an avatar in a virtual space, and when it is determined that the input information includes an input corresponding to the specific action, further reflecting the specific action in an avatar in the virtual space and displaying the avatar in the virtual space;
A display control method comprising:
(20)
Computer,
an acquisition unit that acquires user behavior in a real space as input information;
a determination unit that determines whether or not the input information includes an input corresponding to a specific action that has been registered in advance;
a display control unit that reflects a behavior of the user corresponding to the input information in an avatar in a virtual space, and, when it is determined that the input information includes an input corresponding to the specific action, further reflects the specific action in an avatar in the virtual space and displays the avatar in the virtual space;
A display control program for causing the display control device to function as a display control device having the display control program.

REFERENCE SIGNS LIST 10 Avatar 50 Analysis server 100 Display control device 110 Communication unit 120 Storage unit 130 Control unit 131 Acquisition unit 132 Determination unit 133 Display control unit 140 Sensor unit 141 Camera 142 Microphone 150 Display unit

Claims

an acquisition unit that acquires user behavior in a real space as input information;
a determination unit that determines whether or not the input information includes an input corresponding to a specific action that has been registered in advance;
a display control unit that reflects a behavior of the user corresponding to the input information in an avatar in a virtual space, and, when it is determined that the input information includes an input corresponding to the specific action, further reflects the specific action in an avatar in the virtual space and displays the avatar in the virtual space;
A display control device comprising:
The acquisition unit is
accepting a designation from the user as to whether or not the specific action is to be reflected in the avatar;
The determination unit is
determining whether or not an input corresponding to the specific action is included in the input information when the specific action is designated by the user;
The display control device according to claim 1 .
The acquisition unit is
Acquire a position from an imaging device when the behavior of the user is imaged;
The determination unit is
When the position from the imaging device is farther than a predetermined distance, it is not determined whether or not the input corresponding to the specific action is included in the input information, and when the position from the imaging device is within a predetermined distance, it is determined whether or not the input corresponding to the specific action is included in the input information.
The display control device according to claim 1 .
The acquisition unit is
Obtaining the behavior of multiple users as input information,
The determination unit is
determining a level of excitement of an interaction between the plurality of users in the virtual space based on the behavior of the plurality of users;
The display control unit is
displaying a performance effect in the vicinity of the plurality of users according to the level of excitement of the exchange;
The display control device according to claim 1 .
The determination unit is
determining a level of excitement in the exchange based on the number of users participating in the exchange, the frequency of conversation between the users, and the degree of overlap of voices between the multiple users;
The display control device according to claim 4.
The display control unit is
determining an expression or occurrence frequency of the performance effect based on the level of excitement of the exchange, the body orientation of the avatar, and the content of the conversation between the plurality of users;
The display control device according to claim 5 .
The determination unit is
determining whether an input corresponding to the specific action associated with a specific emotion is included in the input information;
The display control unit is
when it is determined that the input information includes a specific action associated with the specific emotion, the specific action associated with the specific emotion is reflected in the avatar, and the avatar is displayed in the virtual space.
The display control device according to claim 1 .
The determination unit is
comparing the input information with voiceprint information registered in advance by the user, the voiceprint information being associated with the specific emotion, to determine whether or not the input information includes an input corresponding to a specific action associated with the specific emotion;
The display control device according to claim 7.
The display control unit is
displaying the avatar reflecting a specific action associated with the specific emotion, and displaying a performance effect associated with the specific emotion in the vicinity of the avatar;
The display control device according to claim 7.
The display control unit is
displaying words uttered by the avatar based on the input information, and displaying a history of the words in an upward transition manner starting from the avatar;
The display control device according to claim 1 .
The display control unit is
displaying the words uttered by the avatar as an icon that pictorially represents the meaning based on the meaning included in the input information;
The display control device according to claim 10.
The determination unit is
determining a distance between a position in a virtual space where the display of the words is observed and a position of an avatar uttering the words;
The display control unit is
displaying the history of the words or the icons as a dramatic effect not including the meaning of the words and the icons based on the determined distance;
The display control device according to claim 11.
The determination unit is
determining a positional relationship between a position in a virtual space where the display of the words is observed and a position of an avatar uttering the words;
The display control unit is
determining an angle and a three-dimensional representation of the representation of the words observed in a virtual space based on the determined positional relationship;
The display control device according to claim 11.
The determination unit is
determining whether a first avatar is within a range recognized by a second avatar in the virtual space;
The display control unit is
When it is determined that the first avatar is included in a range recognized by the second avatar, a movement path of the first avatar is adjusted so that the first avatar approaches the second avatar;
The display control device according to claim 1 .
The determination unit is
determining whether the first avatar has come close to the second avatar in the virtual space;
The display control unit is
when it is determined that the first avatar is in proximity to the second avatar, displaying the first avatar and the second avatar by reflecting a specific action corresponding to the proximity in at least one of the first avatar and the second avatar;
The display control device according to claim 14.
The acquisition unit is
receiving a request from the user to change an appearance of the avatar;
The display control unit is
reflecting the change based on the request in the avatar, thereby changing the appearance of the avatar in the virtual space together with a dramatic effect;
The display control device according to claim 1 .
The acquisition unit is
receiving a request from the user to exchange any one of a plurality of objects associated with a first avatar with a second avatar;
The display control unit is
When the request is received, a list of a plurality of objects associated with the first avatar is displayed in the vicinity of the first avatar.
The display control device according to claim 1 .
The acquisition unit is
obtaining, from the user, an instruction to select one of the plurality of objects displayed in the list to be exchanged for the second avatar;
The display control unit is
When the instruction is acquired, the display device displays any one of the objects in a floating manner from the list display, and moves the floating object in a virtual space so as to overlap the second avatar in accordance with an operation received from the user.
The display control device according to claim 17.
The computer
The user's behavior in the real world is acquired as input information,
determining whether the input information includes an input corresponding to a specific action registered in advance;
reflecting the behavior of the user corresponding to the input information in an avatar in a virtual space, and when it is determined that the input information includes an input corresponding to the specific action, further reflecting the specific action in an avatar in the virtual space and displaying the avatar in the virtual space;
A display control method comprising:
Computer,
an acquisition unit that acquires user behavior in a real space as input information;
a determination unit that determines whether or not the input information includes an input corresponding to a specific action that has been registered in advance;
a display control unit that reflects a behavior of the user corresponding to the input information in an avatar in a virtual space, and, when it is determined that the input information includes an input corresponding to the specific action, further reflects the specific action in an avatar in the virtual space and displays the avatar in the virtual space;
A display control program for causing the display control device to function as a display control device having the display control program.