WO2023032264A1

WO2023032264A1 - Information processing device, information processing method, and program

Info

Publication number: WO2023032264A1
Application number: PCT/JP2022/006581
Authority: WO
Inventors: 隆太郎峯; 巨成高橋; 仕豪温; 真幹堀川
Original assignee: ソニーグループ株式会社
Priority date: 2021-09-03
Filing date: 2022-02-18
Publication date: 2023-03-09
Also published as: EP4386687A1; JPWO2023032264A1; CN117859154A

Abstract

[Problem] To provide an information processing device, an information processing method, and a program which are capable of making a virtual object which is associated with a user in a virtual space act more naturally even when not operated by the user. [Solution] An information processing device comprising a control unit that controls the behavior of a virtual object which is associated with a user in a virtual space, in accordance with operations by the user, wherein during a period of no operation by the user, the control unit generates behavior of the virtual object on the basis of sensing data pertaining to the user in real space and reflects the behavior in the virtual object.

Description

Information processing device, information processing method, and program

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, the use of virtual spaces, which are virtual worlds connected via the Internet, has become widespread. A user can use a character (also called an avatar) that acts as an alter ego of the user in a virtual space and communicate with users all over the world via a network. Avatars are represented by, for example, two-dimensional or three-dimensional CG (Computer Graphics).

Regarding the use of virtual space, Patent Document 1 below, for example, discloses a technique for reflecting the actions and objects held in the real world by a participant who participates in communication in the avatar of the participant in the virtual space.

JP 2009-140492 A

However, during the non-operation period in which the user does not operate, unnatural phenomena occur, such as the avatar (virtual object) disappearing from the virtual space, or the avatar not moving at all in the virtual space. may be passed on to other users.

Therefore, the present disclosure proposes an information processing device, an information processing method, and a program capable of causing a virtual object associated with a user in a virtual space to behave more naturally even when the user does not operate. .

According to the present disclosure, the controller includes a control unit that controls the behavior of the virtual object associated with the user in the virtual space according to the user's operation, and the control unit controls the user during a non-operation period of the user. The present invention proposes an information processing apparatus that generates behavior of the virtual object based on sensing data in a real space and performs control to reflect the behavior on the virtual object.

According to the present disclosure, the processor includes controlling the behavior of the virtual object associated with the user in the virtual space according to the user's operation, and further, during a period in which the user is not operating, An information processing method is proposed, including generating behavior of the virtual object based on sensing data in real space and performing control to reflect the behavior on the virtual object.

According to the present disclosure, a computer is caused to function as a control unit that controls the behavior of a virtual object associated with the user in a virtual space in accordance with a user's operation, and the control unit operates during a period in which the user is not operating (2) proposes a program that generates behavior of the virtual object based on sensing data of the user in the real space and performs control to reflect the behavior on the virtual object.

1 is a diagram describing an overview of an information processing system according to an embodiment of the present disclosure; FIG. FIG. 4 is a diagram illustrating how the user's sensing data in the real space is reflected in the behavior of the user's avatar in the virtual space according to the present embodiment; It is a block diagram which shows an example of a structure of the user terminal by this embodiment. It is a block diagram which shows an example of a structure of the management server by this embodiment. It is a figure which shows an example of the database of the avatar action by this embodiment. 4 is a sequence diagram showing an example of the flow of operation processing according to the embodiment; FIG. FIG. 10 is a diagram illustrating an example of expression of an avatar's autonomous action restricted according to the privacy level according to the present embodiment; It is a block diagram explaining generation|occurrence|production of general-purpose action by the modification of this embodiment.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and drawings, constituent elements having substantially the same functional configuration are denoted by the same reference numerals, thereby omitting redundant description.

Also, the description shall be given in the following order.
1. Overview of an information processing system according to an embodiment of the present disclosure2. Configuration example 2-1. Configuration example of user terminal 10 2-2. Configuration example of virtual space server 20 3 . Operation processing 4. Modification 5. supplement

<<1. Outline of information processing system according to an embodiment of the present disclosure>>
An information processing system according to an embodiment of the present disclosure relates to control of a virtual object that is associated with a user in a virtual space and serves as an alter ego of the user. A virtual object that serves as an alter ego of a user is, for example, a humanoid or non-human character represented by two-dimensional or three-dimensional CG, and is also called an avatar. In recent years, communication in virtual space has become widespread, and not only simple communication such as games and conversations, but also various communication such as live distribution of artists and trading of in-game contents such as 3D models. ing. In addition, there is a trend to hold various events such as exhibitions that have been held in the real world using avatars in virtual space without visiting the actual site. It is attracting attention as a living space for A virtual world on the Internet, which is a virtualized real space, is commonly called a Metaverse.

FIG. 1 is a diagram explaining an overview of an information processing system according to an embodiment of the present disclosure. As shown in FIG. 1, the information processing system according to the present embodiment includes one or more user terminals 10 (10A, 10B, 10C, . and including.

The user terminal 10 is an information processing terminal used by the user. The user terminal 10 transmits information of operation input by the user and sensing data to the virtual space server 20 . Also, the user terminal 10 performs control for displaying the video of the user's viewpoint in the virtual space received from the virtual space server 20 . The user's point of view may be the point of view of the user's avatar in the virtual space, or the point of view of the view including the appearance of the avatar.

Also, the user terminal 10 can be realized by a smartphone, a tablet terminal, a PC (personal computer), an HMD (Head Mounted Display) worn on the head, a projector, a television device, a game machine, or the like. The HMD may have a non-transmissive display that covers the entire field of view, or may have a transmissive display. Examples of HMDs having a non-transmissive display unit include glasses-type devices having a so-called AR (Augmented Reality) display function that superimposes and displays a virtual object in real space. Also, the HMD may be a device capable of arbitrarily switching the display unit between a non-transmissive type and a transmissive type.

For example, by using a non-transmissive HMD that covers the entire field of view as the user terminal 10, the user can experience virtual space through VR (Virtual Reality). The display unit of the HMD includes a left-eye display and a right-eye display, allowing the user to stereoscopically view an image from the user's viewpoint in the virtual space, thereby providing a more realistic sense of immersion in the virtual space.

The virtual space server 20 is an information processing device that generates and controls virtual space, and generates and distributes video from arbitrary viewpoints in virtual space. The virtual space server 20 may be realized by a single device or by a system composed of a plurality of servers. Various 2D or 3D virtual objects are placed in the virtual space. An example of a virtual object is each user's avatar. The virtual space server 20 can control each user's avatar in real time based on information received from each user terminal 10 . Each user can view video from a user's point of view (for example, a user's avatar's point of view) in virtual space using the user terminal 10 and communicate with other users via the avatar. The virtual space server 20 can also control the transmission of the voice received from the user terminal 10 (user's uttered voice) to the other user terminal 10 corresponding to another user avatar near the user avatar. This enables voice conversations between avatars in the virtual space. Conversation between avatars is not limited to voice, and may be conducted in text.

(Organization of issues)
The avatar (virtual object) placed in the virtual space is operated by the user in real time. You may end up in a state where you can't move at all. If such a phenomenon that can be said to be unnatural in the real space occurs, there is a risk that other users will feel uncomfortable with the virtual space. In particular, in the case of the Metaverse, which is used as a second living space, it is not desirable for the avatar to suddenly disappear or become completely motionless.

Therefore, in the information processing system according to the present disclosure, it is possible to cause the avatar, which is a virtual object associated with the user in the virtual space, to behave more naturally even when the user does not operate.

That is, the virtual space server 20 avoids an unnatural state by causing the avatar to act autonomously while the user is not operating. However, uniform behavior control by a simple autopilot is not sufficient for a more natural expression of avatar behavior. On the other hand, in the present embodiment, during a non-operation period, avatar behavior associated with the user is generated based on the user's sensing data in the real space, and control is performed to reflect the avatar behavior. As a result, more natural autonomous behavior of the avatar is realized, and the user's behavior in the real space is reflected in the user's avatar, so that the user's own state in the virtual space is compared with the state in the real space. It is possible to reduce the user's sense of incongruity with the autonomous action of the avatar.

FIG. 2 is a diagram explaining how the user's sensing data in the real space is reflected in the behavior of the user's avatar 4 in the virtual space according to this embodiment. As shown in FIG. 2, various sensors sense the user's state while shopping in the real space. reflected in For example, the virtual space server 20 generates “shopping behavior” from sensing data and reflects it on the behavior of the avatar 4 . More specifically, the virtual space server 20 controls the behavior of the avatar 4 to purchase a specified product at a specified store in the virtual space. The product to be purchased may be a product that the user has previously added to the planned purchase/favorite list, or may be appropriately determined based on the user's tastes and preferences, action history in the virtual space, tasks, and the like. Alternatively, as will be described later, a predetermined item may be purchased free of charge as a reward from the service side for the autonomous action. Shopping behaviors that do not involve actual purchases, such as avatars entering and exiting stores in virtual space and window shopping around a number of stores, may also be used.

In this way, even while the user is not operating, the avatar in the virtual space autonomously performs natural actions, which reduces the discomfort felt by other users. In addition, since the user's behavior in the real space is reflected in the user's avatar, the user's sense of incongruity with respect to the autonomous behavior of the avatar is reduced.

The outline of the information processing system according to one embodiment of the present disclosure has been described above. Next, the configuration of each device included in the information processing system according to this embodiment will be described with reference to the drawings.

<<2. Configuration example >>
<2-1. Configuration example of user terminal 10>
FIG. 3 is a block diagram showing an example of the configuration of the user terminal 10 according to this embodiment. As shown in FIG. 3 , the user terminal 10 has a communication section 110 , a control section 120 , an operation input section 130 , a motion sensor 140 , a positioning section 150 , a display section 160 , an audio output section 170 and a storage section 180 . The user terminal 10 may be implemented by, for example, a wearable device such as a transparent or non-transparent HMD, smart phone, tablet terminal, smart watch, or smart band.

(Communication unit 110)
The communication unit 110 communicates with the virtual space server 20 by wire or wirelessly to transmit and receive data. The communication unit 110 is, for example, a wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), infrared communication, or a mobile communication network (4G (fourth generation mobile communication system), Communication using 5G (fifth generation mobile communication system) or the like can be performed.

(control unit 120)
The control unit 120 functions as an arithmetic processing device and a control device, and controls overall operations within the user terminal 10 according to various programs. The control unit 120 is realized by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 120 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 120 according to the present embodiment performs control to display on the display unit 160 the video from the user's viewpoint in the virtual space, which is transmitted (for example, streamed) from the virtual space server 20 . The control unit 120 also controls the reproduction of the audio signal transmitted from the virtual space server 20 together with the video of the user's viewpoint from the audio output unit 170 . Further, the control unit 120 controls transmission of information acquired by the operation input unit 130 , the motion sensor 140 , and the positioning unit 150 from the communication unit 110 to the virtual space server 20 . For example, various operation information is input from the operation input unit 130 and transmitted to the virtual space server 20 as input information for user operations on the virtual space. Also, motion data acquired by the motion sensor 140 can be transmitted to the virtual space server 20 as information for controlling the position and posture (orientation of the face, etc.) of the avatar. Here, as an example, a device (device used to operate the avatar) for the user to view the video in the virtual space and a device for the user's real space used to generate the autonomous behavior of the avatar during the non-operating period of the avatar. Although the description is based on the assumption that the device that transmits the sensing data to the virtual space server 20 is the same device, these may be different devices.

The control unit 120 according to this embodiment also functions as a state recognition unit 121. The state recognition unit 121 recognizes the user's state based on the user's sensing data acquired by the motion sensor 140 . The state of the user is, for example, walking, running, standing, sitting, or sleeping. The state of the user recognized by the state recognition unit 121 is transmitted to the virtual space server 20 by the control unit 120, and used in the virtual space server 20 when generating the autonomous action of the avatar. The location information acquired by the location positioning unit 150 is also used in the virtual space server 20 when generating autonomous actions of the avatar. The control unit 120 may transmit the location information to the virtual space server 20 together with the state of the user, or may transmit the location information when a change (movement) in the location is detected. can be Also, the control unit 120 may transmit the user's state and location information to the virtual space server 20 during a non-operation period in which the user does not operate the avatar in the virtual space. Also, the control unit 120 may specify the name of the place where the user is by combining the positional information and the map information, and transmit the name to the virtual space server 20 . The place name may be a general name. For example, if it is specified that the user is in "XX park in XX city", the user may simply be notified of "park". This may protect the user's privacy. Map information may be stored in the storage unit 180 in advance. The map information is not limited to outdoor map information, and includes indoor map information such as inside a school, inside a company, inside a department store, inside one's home, and the like. Control unit 120 can also identify which room the user is in, such as a bedroom or a living room, from the position information.

In addition, the control unit 120 is obtained by an external sensor (for example, a camera installed around the user, a motion sensor attached to the user separately from the user terminal 10, etc.) obtained from the communication unit 110 The user's sensing data (captured image, motion data) may be used for recognizing the user's state and specifying the location, or may be transmitted to the virtual space server 20 as it is. Also, the control unit 120 may transmit the operation information received from the controller held by the user to the virtual space server 20 .

(Operation input unit 130)
Operation input unit 130 receives an operation instruction from the user and outputs the operation content to control unit 120 . The operation input unit 130 may be, for example, a touch sensor, a pressure sensor, or a proximity sensor. Alternatively, the operation input unit 130 may be a physical configuration such as buttons, switches, and levers.

(motion sensor 140)
The motion sensor 140 has a function of sensing the motion of the user. More specifically, motion sensor 140 may have an acceleration sensor, an angular velocity sensor, and a geomagnetic sensor. Furthermore, the motion sensor 140 may be a sensor capable of detecting a total of 9 axes, including a 3-axis gyro sensor, a 3-axis acceleration sensor, and a 3-axis geomagnetic sensor. The motion of the user includes motion of the user's body and motion of the head. More specifically, the motion sensor 140 senses the movement of the user terminal 10 worn by the user as the movement of the user. For example, when the user terminal 10 is configured by an HMD and worn on the head, the motion sensor 140 can sense the movement of the user's head. Further, for example, when the user terminal 10 is configured by a smart phone and the user goes out while the user terminal 10 is in a pocket or bag, the motion sensor 140 can sense the movement of the user's body. Also, the motion sensor 140 may be a wearable device configured separately from the user terminal 10 and worn by the user.

(Position positioning unit 150)
The positioning unit 150 has a function of acquiring the current position of the user. In this embodiment, it is assumed that the user possesses the user terminal 10, and the position of the user terminal 10 is regarded as the current position of the user.

Specifically, the positioning unit 150 calculates the absolute or relative position of the user terminal 10 . For example, the position positioning unit 150 may position the current position based on an acquired signal from the outside. For example, a GNSS (Global Navigation Satellite System) that detects the position of the user terminal 10 by receiving radio waves from an artificial satellite may be used. In addition to GNSS, a method of detecting a position by transmitting/receiving with Wi-Fi (registered trademark), Bluetooth (registered trademark), a mobile phone/PHS/smartphone, or the like, short-distance communication, or the like may be used. Also, the positioning unit 150 may estimate information indicating relative changes based on the detection results of an acceleration sensor, an angular velocity sensor, or the like. The positioning unit 150 can perform outdoor positioning and indoor positioning using the various methods described above. Note that the position may include an altitude. Position positioning unit 150 may include an altimeter.

(Display unit 160)
The display unit 160 has a function of displaying a video (image) of the user's viewpoint in the virtual space. For example, the display unit 160 may be a display panel such as a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display.

(Audio output unit 170)
Audio output section 170 outputs an audio signal under the control of control section 120 . The audio output unit 170 may be configured as headphones, earphones, or bone conduction speakers, for example.

(storage unit 180)
The storage unit 180 is implemented by a ROM (Read Only Memory) that stores programs, calculation parameters, and the like used in the processing of the control unit 120, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate. The storage unit 180 according to this embodiment may store, for example, an algorithm for state recognition.

Although the configuration of the user terminal 10 has been specifically described above, the configuration of the user terminal 10 according to the present disclosure is not limited to the example shown in FIG.

For example, the user terminal 10 may be realized by multiple devices. Specifically, the motion sensor 140 or the positioning unit 150 and the control unit 120 may be configured separately.

In addition, the user terminal 10 may further have various sensors. For example, the user terminal 10 has a camera, a microphone, a biosensor (detection unit for pulse, heart rate, perspiration, blood pressure, body temperature, respiration, myoelectric value, electroencephalogram, etc.), a gaze detection sensor, a distance measurement sensor, etc., and obtains The information obtained may be transmitted to the virtual space server 20 . Also, the state recognition unit 121 may recognize the user's state (running, walking, sleeping, etc.) in consideration of not only motion data but also biometric data acquired by a biosensor, for example. In addition, the control unit 120 may analyze the captured image around the user acquired by the camera, and specify the user's position (the name of the place where the user is).

<2-2. Configuration example of virtual space server 20>
FIG. 4 is a block diagram showing an example of the configuration of the virtual space server 20 according to this embodiment. As shown in FIG. 4, the virtual space server 20 has a communication section 210, a control section 220, and a storage section 230. FIG.

(Communication unit 210)
The communication unit 210 transmits and receives data to and from an external device by wire or wirelessly. The communication unit 210 is, for example, wired/wireless LAN (Local Area Network), Wi-Fi (registered trademark), Bluetooth (registered trademark), mobile communication network (LTE (Long Term Evolution), 4G (4th generation mobile communication method), 5G (fifth generation mobile communication method)), etc., to connect to the user terminal 10 for communication.

(control unit 220)
The control unit 220 functions as an arithmetic processing device and a control device, and controls overall operations within the virtual space server 20 according to various programs. The control unit 220 is implemented by an electronic circuit such as a CPU (Central Processing Unit), a microprocessor, or the like. The control unit 220 may also include a ROM (Read Only Memory) that stores programs to be used, calculation parameters, and the like, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate.

The control unit 220 according to this embodiment also functions as an avatar behavior generation unit 221 and an avatar control unit 222.

The avatar behavior generation unit 221 has a function of generating the avatar's autonomous behavior based on the user's sensing data in the real space during the non-operation period. Also, the avatar control unit 222 has a function of controlling the user's avatar according to the autonomous behavior generated by the avatar behavior generation unit 221 . As a result, it is possible to avoid an unnatural situation in which the avatar suddenly disappears from the virtual space or does not move at all even during a period of non-operation. Also, the user's sensing data is, for example, at least one of the user's state and location information. Sensing data may be transmitted from the user terminal 10 . The information detected by the motion sensor 140 or the positioning unit 150 may be transmitted directly from the user terminal 10, or the recognition result recognized based on the information may be transmitted. A user's state can be recognized from the motion data, as described above. Such recognition may be performed by the user terminal 10 or by the control unit 220 of the virtual space server 20 . The location information may be the name of the place. In this embodiment, by reflecting the user's state and position information in the real space in the avatar's autonomous action, it is possible to reduce the user's sense of incongruity with respect to the avatar's autonomous action during the non-operation period. A user's sense of incongruity may occur in a familiar virtual space called the Metaverse when, for example, the user's avatar behaves arbitrarily and has nothing to do with his or her actual behavior. It is a sense of incongruity.

The generation of avatar actions according to this embodiment will be described in detail below.

The avatar action generation unit 221 may refer to a database of avatar actions based on the user state and position information obtained by sensing the user, for example, to generate autonomous actions of the avatar. An example of an avatar behavior database is shown in FIG. As shown in FIG. 5, a database is used in which avatar actions are associated in advance with user states and positions. It can be said that states and positions (locations) are factors that constitute avatar behavior. For example, in the example shown in FIG. 5, the action "eating" is composed of the state of "sitting" and the position (place) factors of "living room at home" and "restaurant". The factors that make up each avatar action are desirably factors in the user action that match the avatar action. For example, as shown in FIG. 5, factors of the avatar behavior "sleeping" include "state: sleep" and "position: bedroom" that constitute the matching user behavior "sleeping". Note that the avatar action defined here is an action that the avatar can perform in the virtual space. The correspondence relationship between the avatar behavior shown in FIG. 5 and each factor such as state and position is an example, and the present embodiment is not limited to this.

First, the avatar action generation unit 221 matches the user state and position information acquired from the user terminal 10 with the avatar action database, and calculates the matching rate. For example, if the information obtained from the user terminal 10 is "state: walking" and "position: shop", the matching rate for "shopping" among the avatar actions listed in the database of FIG. The precision rate of "defeating" is calculated as 50%. This is to satisfy the condition factor of the condition factor and position factor of "defeat the enemy". In this case, the avatar action generation unit 221 may determine the avatar action (here, "shopping") that completely corresponds (with a matching rate of 100%) as the autonomous action of the avatar. Further, if there is no completely corresponding avatar action, the avatar action generation unit 221 may stochastically determine the avatar action candidate including at least one of the corresponding state factor and position factor. At this time, the avatar action generation unit 221 increases the selection probability of candidates that include more suitable factors, thereby determining natural autonomous actions that are less uncomfortable with respect to user actions in the real space. can be done. Note that even when there is a completely corresponding avatar action (with a matching rate of 100%), the avatar action generation unit 221 generates a probabilistic You may make it decide to.

With the above-described method that uses a database that associates state factors and position factors with avatar actions, it is possible to generate autonomous actions even for actions that are not performed in real space but are performed in virtual space. One example is the avatar action of "defeating the enemy" listed in the database of FIG. This behavior occurs frequently in the virtual space with the purpose of defeating the enemy, but there is no direct corresponding behavior in the real space. Therefore, in the present embodiment, for example, factors such as "state: walking/standing" and "position: road" configure "avatar action: defeating enemy". As a result, even an action that is not performed in the real space but is performed only in the virtual space can be selected as the autonomous action of the avatar based on the user's sensing data in the real space.

Also, in the example shown in FIG. 5, there are no candidates for avatar actions composed of exactly the same factors, but multiple avatar actions having exactly the same factors may be defined. At this time, the avatar action generation unit 221 selects probabilistically from a plurality of avatar action candidates that completely correspond (with a matching rate of 100%).

Also, the categories of states and categories of positions shown in FIG. 5 are examples, and the present embodiment is not limited to these. The more diverse these categories are, the more appropriately it is possible to reflect the user's behavior in the real space on the avatar's behavior. In addition, by preparing candidates for avatar actions that are more diverse and comprehensively composed of state and position factors, it is possible to generate more appropriate and natural autonomous actions.

The avatar behavior database according to this embodiment can be reused in multiple virtual spaces for different services. That is, a database such as that shown in FIG. 5 may be used in generating an autonomous action of an avatar during a period of non-user operation in another virtual space. Information in the database can be shared by cooperation between virtual spaces. In addition, if there is an inappropriate autonomous action or an insufficient autonomous action in each virtual space, the avatar action defined in each virtual space can be corrected, changed, or added as appropriate to create an appropriate behavior for each virtual space. It is possible to create a database. For example, an avatar action of "defeating an enemy" may be changed to "cultivating a field" without changing the constituent factors. It is also possible for the user's avatar to move to another virtual space by cooperation between the virtual spaces. It is also assumed that a user's avatar exists for each virtual space (for each service). As an example, the user terminal 10 may transmit user sensing data to multiple virtual space servers 20 . This makes it possible to refer to databases in each of a plurality of virtual spaces and reflect the user behavior in the real space on the autonomous behavior of the user's avatar.

As described above, in the present embodiment, a method for determining (generating) the autonomous behavior of an avatar using factors has been described as an example. If possible, the method is not particularly limited. For example, if there is a virtual space that can accurately reflect the state and position of the user in the real space, the detected information may be applied to the avatar in the virtual space as it is. Also, the "user's sensing data" to be reflected in the avatar is not limited to the state or position. Also, the "state" is not limited to states recognized based on motion data. For example, the state may be recognized based on the user's uttered voice (conversation) picked up by the microphone of the user terminal 10 or biological information (heart rate, blood pressure, body temperature, etc.).

(storage unit 230)
The storage unit 230 is implemented by a ROM (Read Only Memory) that stores programs and calculation parameters used in the processing of the control unit 220, and a RAM (Random Access Memory) that temporarily stores parameters that change as appropriate. According to this embodiment, the storage unit 230 stores information on the virtual space.

Although the configuration of the virtual space server 20 has been specifically described above, the configuration of the virtual space server 20 according to the present disclosure is not limited to the example shown in FIG. For example, the virtual space server 20 may be realized by multiple devices.

<<3. Operation processing >>
Next, the flow of virtual object processing according to this embodiment will be specifically described with reference to the drawings. FIG. 6 is a sequence diagram showing an example of the flow of operation processing according to this embodiment. Note that the processing shown in FIG. 6 is performed during a non-operation period in which the user does not operate the avatar (for example, when the user logs out or closes the screen displaying the image of the virtual space, and the operation is not performed for a certain period of time). case, etc.).

As shown in FIG. 6, the user terminal 10 first acquires the movement and position of the user from each sensor (step S103). Specifically, the motion sensor 140 acquires the motion of the user, and the position measurement unit 150 acquires position information.

Next, the state recognition unit 121 of the user terminal 10 recognizes the user's state based on the user's movement (motion data) (step S106).

Next, the user terminal 10 transmits the location information (which may be the general name of the location) and the recognition result of the state (step S109).

Next, the virtual space server 20 generates the user's avatar behavior based on the location information and the status recognition result received from the user terminal 10 (step S121). Note that the avatar behavior generator 221 of the virtual space server 20 may generate avatar behavior based on at least one of the position information and the state recognition result.

Then, the avatar control unit 222 of the virtual space server 20 applies the avatar behavior generated (selected) by the avatar behavior generation unit 221 to the user's avatar and controls it (step S124). This allows the user's avatar to act autonomously even during a period in which the user is not operating, thereby reducing unnaturalness. In addition, by reflecting the behavior of the user in the real space in the autonomous behavior of the avatar, it is possible to reduce the sense of incongruity and resistance of the user of his/her own avatar. In addition, the movement and position of the user are acquired by the user terminal 10 (or the wearable device worn by the user), thereby alleviating restrictions on the measurement range.

An example of operation processing according to the present embodiment has been described above. Note that the operation processing shown in FIG. 6 is an example, and part of the processing may be performed in a different order or in parallel.

<<4. Modification>>
<4-1. Generation of avatar behavior considering privacy>
In the embodiments described above, the user's state and position in the real space can be reflected in the avatar's behavior. However, since communication is performed with an unspecified number of other users in the virtual space, consideration of privacy is also important. Even in the above-described embodiment, the current position of the user is not specifically grasped, but depending on the situation, it is assumed that it is necessary to generate the autonomous behavior of the avatar with more strict consideration of privacy. be.

Therefore, the virtual space server 20 presets a privacy level for each action defined as, for example, an avatar's autonomous action, and shows (displays) the action up to a permitted level according to the familiarity between the user and other users. may Table 1 below is an example of the privacy level set for the autonomous action of the avatar.

In Table 1, a higher privacy level is set for "shopping" with high privacy, such as going out. The privacy level may be arbitrarily set by the user.

In addition, the user determines to what level the avatar's behavioral expression is permitted (whether to be shown) to other users in the virtual space. Such permission may be individually set for each other user, or may be set for each group by grouping other users in advance. For example, users with close relationships are allowed up to the highest privacy level (e.g., level 3), and other users with no close relationships are allowed up to the lowest privacy level (e.g., level 0). You may allow it. If there is no action that corresponds to the candidates for the autonomous action of the avatar due to restrictions by the privacy level, the avatar action generation unit 221 can select the general-purpose action. A general-purpose action is, for example, an action that is randomly selected from autonomous action candidates defined in an avatar action database. Alternatively, the action may be randomly selected from a large number of autonomous action candidates prepared as general-purpose actions.

FIG. 7 is a diagram explaining an example of expression of autonomous actions of avatars that are restricted according to the privacy level according to this embodiment. For example, it is assumed that user B (avatar 4b), who has a close relationship with user A, is allowed up to privacy level 3, and user C (avatar 4c), who is not close to user A, is allowed up to privacy level 0. do. The virtual space server 20 refers to the database shown in FIG. 5 based on the data sensed from the user in the real space (state: walking, place: shop), and performs the autonomous action of "shopping" for the user's avatar 4a. to decide. Next, as shown in Table 1, since the privacy level of "shopping" is "level 3", for user B who is permitted up to privacy level 3, the avatar 4a is "shopping". show On the other hand, for user C, who is permitted up to privacy level 0, the behavior of "shopping" is not permitted up to privacy level (level 3). at home, etc.). More specifically, the virtual space server 20 determines how the autonomous behavior of the avatar 4a is displayed when generating the video of each user's viewpoint to be transmitted to the user terminals of the user B and the user C. control accordingly.

(Regarding generic actions)
The above general-purpose action may be a selection method based on a learning base using the action history of each avatar in the virtual space, in addition to the method of selecting at random. FIG. 8 is a configuration diagram illustrating generation of a general-purpose action according to a modified example of this embodiment. The avatar action history DB 182 shown in FIG. 8 is a database that accumulates the action history (including time axis information) of all avatars in the virtual space. The information accumulated in the avatar action history DB 182 may be, for example, avatar autonomous actions that reflect the user's actions in real space. When generating a general-purpose action, the avatar action generation unit 221 refers to the current time axis information and the avatar action history DB 182, and acquires information on the percentage of autonomous actions of each avatar at the corresponding time. Then, the avatar action generation unit 221 determines (probabilistically selects based on the ratio information) the action with a higher percentage as the general-purpose action. As a result, the avatars can be made to behave in the same manner as most avatars do, and the user's privacy can be protected in a more natural and unobtrusive manner.

As another method of selecting general-purpose actions, it is also possible to use matching with the database of user actions described with reference to FIG. In the above example, autonomous behavior was determined stochastically by calculating the matching rate between the user's sensing data (state, position) in the real space and each avatar behavior candidate in the database. By adjusting the threshold of the relevance rate when calculating the relevance rate or adding noise when calculating the relevance rate, it is possible to select an appropriate autonomous action while protecting privacy. It is also possible to adjust the strength of privacy protection by adjusting the threshold value and adjusting the intensity of noise.

It is up to each user to decide whether to make the avatar perform autonomous actions similar to those of users in real space, whether to make the avatars perform general-purpose actions with complete consideration of privacy, or whether to make the avatars perform actions that take advantage of the characteristics of both. may be arbitrarily selected.

As explained above, in this modified example, it is possible to protect privacy in a space where communication with an unspecified number of users around the world is possible. Also, it is possible to set a permitted privacy level for each user.

<4-2. About Rewards in Virtual Space for Users Using Autonomous Behavior>
The virtual space server 20 may set a reward for the avatar's autonomous action. For example, in the case of shopping actions, items that can be used in the virtual space are acquired, in the case of work or defeating enemies, experience points and currency are acquired in the virtual space, and in the case of actions at home , includes rewards such as recovery of physical strength used in virtual space. In addition, when the avatar moves in the virtual space by autonomous action, the movement information and the image of the avatar's viewpoint are recorded, and when the user resumes the operation, the image etc. can be confirmed. may

Such rewards can promote an increase in the number of users who use autonomous behavior control.

<4-3. Generation of general-purpose actions considering time zone information>
Unlike real space, virtual space is less affected by the distance between users, and it is possible to easily communicate with users all over the world. However, when the behavior of the user in the real space is reflected in the autonomous behavior of the avatar, the avatar behavior of users in different time zones may coexist without consistency. Therefore, the avatar action generation unit 221 of the virtual space server 20 can control the expression of the avatar's autonomous action according to the time zone of the viewing user, and reduce the unnaturalness caused by the different time zones.

More specifically, first, the virtual space server 20 prepares an avatar action history DB (including time axis information) for each time zone. Next, for user A (viewing user) who is in a time zone that is, for example, the night time zone, when generating general-purpose actions of other user avatars (non-operating) visible to user A, A matching action (for example, "sleep") is extracted from the avatar action history DB.

<4-4. About how to reflect the information of the real space to the appearance of the avatar >
The virtual space server 20 may reflect the user's information in the real space on the appearance of the avatar. For example, each candidate for the avatar action as described with reference to FIG. 5 may be associated with the appearance of the avatar. For example, pajamas for "sleep" and suits for "work". The avatar control unit 222 of the virtual space server 20 can appropriately change the appearance of the avatar when controlling the autonomous action of the avatar. Since each action corresponds to the appearance, the appearance can be changed in the same way when the action is generated in the case of privacy protection as described above (generation of general-purpose action).

<<5. Supplement >>
Although the preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the present technology is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can conceive of various modifications or modifications within the scope of the technical idea described in the claims. are naturally within the technical scope of the present disclosure.

Note that in the present embodiment, as an example, a case will be described where video from the user's viewpoint is distributed from the virtual space server 20 to the user terminal 10 (more specifically, for example, streaming distribution). The system that realizes communication via is not limited to this. For example, each user terminal 10 may generate a virtual space and generate and display an image of the user's viewpoint in the virtual space. Information for generating the virtual space is obtained in advance from the virtual space server 20 . In this case, each user terminal 10 transmits the information of the operation input by the user, sensing data, etc. to the virtual space server 20 in real time. Next, the virtual space server 20 controls the transmission of the information regarding the movement of the user avatar received from the user terminal 10 to other user terminals 10 . The virtual space server 20 also transmits avatar autonomous control information as needed.

It is also possible to create one or more computer programs for causing hardware such as the CPU, ROM, and RAM built into the virtual space server 20 described above to exhibit the functions of the virtual space server 20. Also provided is a computer-readable storage medium storing the one or more computer programs.

Also, the effects described in this specification are merely descriptive or exemplary, and are not limiting. In other words, the technology according to the present disclosure can produce other effects that are obvious to those skilled in the art from the description of this specification, in addition to or instead of the above effects.

Note that the present technology can also take the following configuration.
(1)
A control unit that controls the behavior of the virtual object associated with the user in the virtual space according to the user's operation,
The control unit generates an action of the virtual object based on sensing data in a real space of the user during a period in which the user does not operate, and performs control to reflect the action on the virtual object.
Information processing equipment.
(2)
The information processing apparatus according to (1), wherein the sensing data includes information on at least one of a user's state and position.
(3)
The control unit generates the behavior of the virtual object by referring to a database that associates candidates for the behavior of the virtual object with at least one of one or more states or positions, and (1) or (2) above. ).
(4)
In the reference, the control unit calculates a matching rate between each action candidate defined in the database and the sensing data, and selects one action from the action candidates based on the matching rate. The information processing apparatus according to (3), wherein the action of the virtual object is generated by doing so.
(5)
The information processing apparatus according to (3) or (4), wherein the control unit generates an action of the virtual object according to a privacy level set for each action candidate.
(6)
When the privacy level of one action selected from the candidates for each action is a level that other users viewing the avatar, which is the virtual object, are not permitted to view, the control unit selects a general-purpose action as the action of the avatar. The information processing device according to (5), which performs control to generate an action.
(7)
The information processing device according to (6), wherein the control unit randomly selects the general-purpose action from the candidates for each action.
(8)
The information processing device according to (6), wherein the control unit generates the general-purpose action based on the action history of each avatar in the virtual space.
(9)
Any one of (1) to (8) above, wherein the control unit performs control to change the appearance of the virtual object to the appearance associated with the action to be generated when reflecting on the virtual object. The information processing device according to the item.
(10)
The information processing apparatus according to any one of (1) to (9), wherein the control unit generates an image of the user's viewpoint in the virtual space and controls transmission to the user terminal.
(11)
The information processing device further comprises a communication unit,
The information processing device according to any one of (1) to (10), wherein the communication unit receives the sensing data from a user terminal.
(12)
the processor
Controlling the behavior of a virtual object associated with the user in the virtual space according to the user's operation;
Furthermore, during a period in which the user is not operating, the action of the virtual object is generated based on sensing data in the user's real space, and controlled to be reflected in the virtual object.
Information processing methods.
(13)
the computer,
Functioning as a control unit that controls the behavior of the virtual object associated with the user in the virtual space according to the user's operation,
The control unit generates an action of the virtual object based on sensing data in a real space of the user during a period in which the user does not operate, and performs control to reflect the action on the virtual object.
program.

10 user terminal 110 communication unit 120 control unit 121 state recognition unit 130 operation input unit 140 motion sensor 150 positioning unit 160 display unit 170 voice output unit 180 storage unit 20 virtual space server 210 communication unit 220 control unit 221 avatar behavior generation unit 222 Avatar control unit 230 storage unit

Claims

A control unit that controls the behavior of the virtual object associated with the user in the virtual space according to the user's operation,
The control unit generates an action of the virtual object based on sensing data in a real space of the user during a period in which the user does not operate, and performs control to reflect the action on the virtual object.
Information processing equipment.
The information processing apparatus according to claim 1, wherein the sensing data includes information on at least one of a user's state and position.
2. The information according to claim 1, wherein the control unit generates the behavior of the virtual object by referring to a database that associates candidates for the behavior of the virtual object with at least one of one or more states or positions. processing equipment.
In the reference, the control unit calculates a matching rate between each action candidate defined in the database and the sensing data, and selects one action from the action candidates based on the matching rate. 4. The information processing apparatus according to claim 3, wherein the action of said virtual object is generated by doing so.
The information processing apparatus according to claim 3, wherein the control unit generates an action of the virtual object according to a privacy level set for each action candidate.
When the privacy level of one action selected from the candidates for each action is a level that other users viewing the avatar, which is the virtual object, are not permitted to view, the control unit selects a general-purpose action as the action of the avatar. 6. The information processing device according to claim 5, which performs control to generate an action.
The information processing apparatus according to claim 6, wherein the control unit randomly selects the general-purpose action from among the candidates for each action.
The information processing apparatus according to claim 6, wherein the control unit generates the general-purpose action based on the action history of each avatar in the virtual space.
The information processing apparatus according to claim 1, wherein the control unit performs control to change the appearance of the virtual object to the appearance associated with the action to be generated when reflecting on the virtual object.
The information processing apparatus according to claim 1, wherein the control unit generates an image of the user's viewpoint in the virtual space and controls transmission to the user terminal.
The information processing device further comprises a communication unit,
The information processing apparatus according to claim 1, wherein said communication unit receives said sensing data from a user terminal.
the processor
Controlling the behavior of a virtual object associated with the user in the virtual space according to the user's operation;
Furthermore, during a period in which the user is not operating, the action of the virtual object is generated based on sensing data in the user's real space, and controlled to be reflected in the virtual object.
Information processing methods.
the computer,
Functioning as a control unit that controls the behavior of the virtual object associated with the user in the virtual space according to the user's operation,
The control unit generates an action of the virtual object based on sensing data in a real space of the user during a period in which the user does not operate, and performs control to reflect the action on the virtual object.
program.