JP2024021028A - Information processing system, information processing method, and storage medium - Google Patents

Abstract

[Problem] To effectively promote avatar activities in virtual space. [Solution] An avatar processing unit that processes avatar actions in a virtual space; a first storage unit that stores a predetermined skill that is related to the avatar action and that can be associated with the avatar; a second storage unit that stores activation conditions that are set in the virtual space; and a determination unit that determines whether the activation conditions are met when a predetermined skill is associated with one target avatar; An information processing system is disclosed in which, in relation to an object, an avatar processing unit causes a target avatar to perform an action based on a predetermined skill when an activation condition is satisfied. [Selection diagram] Figure 15

Description

The present disclosure relates to an information processing system, an information processing method, and a storage medium.

Calculate the distance between the first part of the first avatar object and the second part of the second avatar object, and determine whether a predetermined condition for making it possible to execute a visual event in the virtual space is satisfied. If it is determined that a predetermined condition is met, a visual event is executed in the virtual space in response to the calculated interval becoming less than or equal to the predetermined interval. The technology is known.

JP2018-128966A

However, with the above-mentioned conventional techniques, it is difficult to effectively promote the activities of avatars in virtual space.

Therefore, in one aspect, the present disclosure aims to effectively promote avatar activities in virtual space.

In one aspect, an avatar processing unit that processes avatar motion in a virtual space;
a first storage unit that stores predetermined skills related to the avatar's actions and that can be associated with the avatar;
a second storage unit that stores activation conditions associated with the predetermined skill;
a determination unit that determines whether the activation condition is met when the predetermined skill is associated with one target avatar;
The activation condition is related to a predetermined object in the virtual space,
An information processing system is provided in which the avatar processing unit causes the target avatar to perform an action based on the predetermined skill when the activation condition is satisfied.

In one aspect, according to the present disclosure, it is possible to effectively promote the activities of an avatar in a virtual space.

FIG. 1 is a block diagram of a virtual reality generation system according to the present embodiment. FIG. 2 is an explanatory diagram of a terminal image that can be viewed through a head-mounted display. FIG. 2 is an explanatory diagram of a terminal image that can be viewed through a smartphone. It is an image diagram of an action based on a predetermined skill "lifting a soccer ball." It is an image diagram of an operation based on a predetermined skill "playing the piano." It is an image diagram of an operation based on a predetermined skill "performance in a band." It is an explanatory diagram of an example of a selection screen (user interface) of a predetermined skill. FIG. 2 is an explanatory diagram of a predetermined skill related to walking style (“stride”). FIG. 2 is an explanatory diagram of a predetermined skill related to walking style (“ladylike”). FIG. 3 is an explanatory diagram of an example of a joint model. FIG. 3 is a diagram showing the relationship between parts using a joint model. FIG. 3 is an explanatory diagram of changes in finger movement. FIG. 2 is an explanatory diagram of a function suitable for reflecting the characteristics of an avatar. FIG. 6 is an explanatory diagram of a period suitable for reflecting the characteristics of an avatar. 12 is a schematic flowchart showing an example of a specific drawing method when the predetermined skill is playing a musical instrument. FIG. 3 is an explanatory diagram of synchronous playback by a plurality of avatars. FIG. 2 is a schematic block diagram showing functions of a server device related to avatar actions based on predetermined skills. FIG. 2 is an explanatory diagram showing an example of data in an object information storage unit. It is an explanatory view showing an example of data in a skill related data storage part. It is an explanatory diagram showing an example of data in a user information storage part. It is an explanatory view showing an example of data in an avatar information storage part. 3 is a timing chart schematically showing an example of the operation of the virtual reality generation system related to the operation of an avatar based on a predetermined skill. 21 is a schematic flowchart illustrating an example of a drawing process regarding a target avatar that is executed in connection with the process shown in FIG. 20. 22 is a schematic flowchart showing an example of animation playback processing (step S234 in FIG. 21).

Hereinafter, each embodiment will be described in detail with reference to the accompanying drawings. In the accompanying drawings, for ease of viewing, only some of the parts with the same attribute may be labeled with reference numerals.

With reference to FIG. 1, an overview of a virtual reality generation system 1 according to an embodiment will be described. FIG. 1 is a block diagram of a virtual reality generation system 1 according to this embodiment. FIG. 2 is an explanatory diagram of a terminal image that can be viewed through a head-mounted display.

The virtual reality generation system 1 includes a server device 10 and one or more terminal devices 20. Although three terminal devices 20, 20A, and 20B are illustrated in FIG. 1 for simplicity, the number of terminal devices 20 may be two or more.

The server device 10 is, for example, an information processing system such as a server managed by an operator that provides one or more virtual realities. The terminal device 20 is a device used by a user, such as a mobile phone, a smartphone, a tablet terminal, a PC (Personal Computer), a head-mounted display, or a game device. A plurality of terminal devices 20 can be connected to the server device 10 via the network 3, typically in different ways for each user.

The terminal device 20 can execute the virtual reality application according to this embodiment. The virtual reality application may be received by the terminal device 20 from the server device 10 or a predetermined application distribution server via the network 3, or may be received by the terminal device 20 from a storage device provided in the terminal device 20 or a memory card readable by the terminal device 20. It may be stored in advance in a storage medium such as. The server device 10 and the terminal device 20 are communicably connected via the network 3. For example, the server device 10 and the terminal device 20 cooperate to execute various processes related to virtual reality.

The terminal devices 20 are communicably connected to each other via the server device 10. Note that in the following, "one terminal device 20 transmits information to another terminal device 20" means "one terminal device 20 transmits information to another terminal device 20 via the server device 10". It means that. Similarly, "one terminal device 20 receives information from another terminal device 20" means "one terminal device 20 receives information from another terminal device 20 via the server device 10". means. However, in a modified example, each terminal device 20 may be communicably connected without going through the server device 10.

Note that the network 3 may include a wireless communication network, the Internet, a VPN (Virtual Private Network), a WAN (Wide Area Network), a wired network, or any combination thereof.

In the example shown in FIG. 1, the virtual reality generation system 1 includes studio units 30A and 30B. The studio units 30A and 30B are host-side devices, similar to the host-side terminal device 20A. The studio units 30A and 30B may be placed in a content production studio, room, hall, or the like. Various equipment for motion capture may be installed in the studio units 30A and 30B.

Each studio unit 30A, 30B can have the same functions as the host's terminal device 20A and/or the server device 10. Hereinafter, when distinguishing between the host side and the participant side, in order to simplify the explanation, the host side terminal device 20A distributes various contents to each participant side terminal device 20B via the server device 10. Mainly explained. However, instead of or in addition to this, the studio units 30A and 30B facing the host user have the same functions as the host user's terminal device 20A, so that each participating user can receive information via the server device 10. Various contents may be distributed to the terminal device 20B. Note that in a modified example, the virtual reality generation system 1 may not include the studio units 30A and 30B.

In the following, the virtual reality generation system 1 realizes an example of an information processing system, but each element of a specific terminal device 20 (see terminal communication unit 21 to terminal control unit 25 in FIG. 1) An example of this may be realized, or a plurality of terminal devices 20 may cooperate to realize an example of an information processing system. Further, the server device 10 alone may realize an example of an information processing system, or the server device 10 and one or more terminal devices 20 may cooperate to realize an example of an information processing system. .

Here, an overview of the virtual reality according to this embodiment will be explained. The virtual reality according to the present embodiment is a virtual reality for any reality such as education, travel, role-playing, simulation, entertainment such as games and concerts, etc. A virtual reality medium is used. For example, the virtual reality according to this embodiment may be realized by a three-dimensional virtual space, various virtual reality media that appear in the virtual space, and various contents provided in the virtual space.

The virtual reality medium is electronic data used in virtual reality, such as cards, items, points, in-service currency (or virtual reality currency), tokens (e.g., non-fungible tokens (NFTs), etc.). )), including any media such as tickets, characters, avatars, parameters, etc. The virtual reality medium may also be virtual reality related information, such as level information, status information, parameter information (physical strength and attack power, etc.), or ability information (skills, abilities, spells, jobs, etc.). In addition, virtual reality media are electronic data that can be acquired, owned, used, managed, exchanged, synthesized, enhanced, sold, discarded, or gifted within virtual reality by users, but the usage mode of virtual reality media is It is not limited to what is specified in the specification.

Note that the avatar typically takes the form of a character facing forward, and may take the form of a person, an animal, or the like. Avatars can have various appearances (appearances when drawn) by being associated with various avatar items. Further, in the following explanation, the user and the avatar may be considered to be the same due to the nature of the avatar. Therefore, for example, "one avatar does XX" may be synonymous with "one user does XX".

A user may wear a wearable device on his head or part of his face, and view the virtual space through the wearable device. Note that the wearable device may be a head-mounted display or a glasses-type device. The glasses-type device may be so-called AR (Augmented Reality) glasses or MR (Mixed Reality) glasses. In either case, the wearable device may be separate from the terminal device 20 or may realize some or all of the functions of the terminal device 20. The terminal device 20 may be realized by a head mounted display.

(Configuration of server device)
The configuration of the server device 10 will be specifically explained. The server device 10 is configured by a server computer. The server device 10 may be implemented by a plurality of server computers working together. For example, the server device 10 may be implemented in cooperation with a server computer that provides various types of content, a server computer that implements various types of authentication servers, and the like. Further, the server device 10 may include a web server. In this case, some of the functions of the terminal device 20, which will be described later, may be realized by the browser processing an HTML document and various programs (JavaScript) associated with the HTML document received from the Web server.

As shown in FIG. 1, the server device 10 includes a server communication section 11, a server storage section 12, and a server control section 13.

The server communication unit 11 includes an interface that communicates with an external device wirelessly or by wire to send and receive information. The server communication unit 11 may include, for example, a wireless LAN (Local Area Network) communication module or a wired LAN communication module. The server communication unit 11 is capable of transmitting and receiving information to and from the terminal device 20 via the network 3.

The server storage unit 12 is, for example, a storage device, and stores various information and programs necessary for various processes related to virtual reality.

The server control unit 13 may include a dedicated microprocessor or a CPU (Central Processing Unit) that implements a specific function by reading a specific program, a GPU (Graphics Processing Unit), or the like. For example, the server control unit 13 cooperates with the terminal device 20 to execute a virtual reality application in response to user input.

(Configuration of terminal device)
The configuration of the terminal device 20 will be explained. As shown in FIG. 1, the terminal device 20 includes a terminal communication section 21, a terminal storage section 22, a display section 23, an input section 24, and a terminal control section 25.

The terminal communication unit 21 includes an interface that communicates with an external device wirelessly or by wire to send and receive information. The terminal communication unit 21 is a wireless communication device compatible with mobile communication standards such as LTE (Long Term Evolution) (registered trademark), LTE-A (LTE-Advanced), a fifth generation mobile communication system, and UMB (Ultra Mobile Broadband). It may also include a communication module, a wireless LAN communication module, a wired LAN communication module, or the like. The terminal communication unit 21 is capable of transmitting and receiving information to and from the server device 10 via the network 3.

The terminal storage unit 22 includes, for example, a primary storage device and a secondary storage device. For example, the terminal storage unit 22 may include a semiconductor memory, a magnetic memory, an optical memory, or the like. The terminal storage unit 22 stores various information and programs used for virtual reality processing, which are received from the server device 10. Information and programs used for virtual reality processing may be acquired from an external device via the terminal communication unit 21. For example, a virtual reality application program may be obtained from a predetermined application distribution server. Hereinafter, the application program will also be simply referred to as an application.

The terminal storage unit 22 may also store data for drawing a virtual space, such as images of indoor spaces such as buildings and outdoor spaces. Note that a plurality of types of data for drawing the virtual space may be prepared for each virtual space and used appropriately.

Furthermore, the terminal storage unit 22 may store various images (texture images) for projection (texture mapping) onto various objects arranged in a three-dimensional virtual space.

For example, the terminal storage unit 22 stores avatar drawing information related to an avatar as a virtual reality medium associated with each user. The avatar in the virtual space is drawn based on avatar drawing information related to the avatar.

The terminal storage unit 22 also stores drawing information related to various objects (virtual reality media) different from avatars, such as various gift objects, buildings, walls, NPCs (Non Player Characters), and the like. Various objects in the virtual space are drawn based on such drawing information. Note that the gift object is an object corresponding to a gift from one user to another user, and is a part of an item. Gift objects include items worn by the avatar (clothes and accessories), decorations (fireworks, flowers, etc.), backgrounds (wallpaper), and the like, tickets that can be used for gacha (lotteries), and the like. It's fine. Note that the term "gift" used in this application means the same concept as the term "token". Therefore, it is also possible to understand the technology described in the present application by replacing the term "gift" with the term "token."

The display unit 23 includes a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display. The display unit 23 can display various images. The display unit 23 is configured with a touch panel, for example, and functions as an interface for detecting various user operations. Note that, as described above, the display unit 23 may be built into a head-mounted display.

The input unit 24 may include physical keys, and may further include any input interface including a pointing device such as a mouse. Further, the input unit 24 may be capable of receiving non-contact user input such as voice input, gesture input, and gaze input. In addition, gesture input uses sensors (image sensors, acceleration sensors, distance sensors, etc.) to detect various states of the user, dedicated motion capture that integrates sensor technology and cameras, controllers such as joypads, etc. may be used. Further, the camera for detecting the line of sight may be placed within the head-mounted display. As mentioned above, the various states of the user include, for example, the user's orientation, position, movement, and the like; in this case, the user's orientation, position, and movement refer to the user's face, hands, and other body parts. This is a concept that includes not only the direction, position, and movement of parts or all parts, but also the direction, position, and movement of the user's line of sight, and the like.

Note that user input using gestures may be used to change the line of sight of the virtual camera. For example, as schematically shown in FIG. 3, when the user changes the orientation of the terminal device 20 while holding the terminal device 20 in his hand, the line of sight of the virtual camera may change depending on the direction. . In this case, even when using a terminal device 20 with a relatively small screen such as a smartphone, the width of the viewing area can be secured in the same manner as being able to look around the surroundings through a head-mounted display.

Terminal control unit 25 includes one or more processors. The terminal control unit 25 controls the overall operation of the terminal device 20.

The terminal control unit 25 transmits and receives information via the terminal communication unit 21. For example, the terminal control unit 25 receives various information and programs used for various processes related to virtual reality from at least one of the server device 10 and another external server. The terminal control unit 25 stores the received information and program in the terminal storage unit 22. For example, the terminal storage unit 22 may store a browser (Internet browser) for connecting to a web server.

The terminal control unit 25 starts a virtual reality application in response to a user's operation. The terminal control unit 25 cooperates with the server device 10 to execute various processes related to virtual reality. For example, the terminal control unit 25 causes the display unit 23 to display an image of the virtual space. For example, a GUI (Graphical User Interface) that detects user operations may be displayed on the screen. The terminal control unit 25 can detect user operations via the input unit 24 . For example, the terminal control unit 25 can detect various operations based on user gestures (operations corresponding to tap operations, long tap operations, flick operations, swipe operations, etc.). The terminal control unit 25 transmits operation information to the server device 10.

The terminal control unit 25 draws an avatar and the like together with the virtual space (image), and causes the display unit 23 to display the terminal image. In this case, for example, as shown in FIG. 2, a stereoscopic image for a head-mounted display may be generated by generating images G200 and G201 that are viewed by the left and right eyes, respectively. FIG. 2 schematically shows images G200 and G201 that are viewed by the left and right eyes, respectively. In addition, below, unless otherwise mentioned, the image in the virtual space refers to the entire image represented by the images G200 and G201. Further, the terminal control unit 25 realizes various movements of the avatar in the virtual space, for example, in response to various operations by the user.

The virtual space described below is an immersive space that can be viewed using a head-mounted display or the like, and is a continuous space in which the user can move freely (just like in real life) through an avatar. The concept includes not only a certain three-dimensional space but also a non-immersive space that can be viewed using a smartphone or the like as described above with reference to FIG. Note that the non-immersive space that can be viewed using a smartphone or the like may be a continuous three-dimensional space in which you can freely move around via an avatar, or a two-dimensional space that can be viewed using a smartphone or similar device. It may be a discontinuous space. Hereinafter, when making a distinction, a continuous three-dimensional space in which a user can freely move around using an avatar (e.g., a 3D avatar) is also referred to as a "metaverse space," and other virtual spaces (e.g., a metaverse space) are referred to as a "metaverse space." A continuous space) is also called a "non-metaverse space."

Here, in such various virtual spaces, a wide variety of users may exist. For example, a distribution user refers to a user who transmits information regarding video and/or audio. For example, a distribution user may broadcast a video individually, collaborate with multiple people to participate, video chat or voice chat where multiple people can participate and/or view, or use a virtual space where multiple people can participate and/or view. The user may be a user who hosts or holds an event (such as a party), that is, a user who primarily executes these events. Therefore, the distribution user in the present disclosure can also be referred to as a host user, an organizer user, an organizer user, or the like.

On the other hand, the viewing user refers to a user who receives information regarding video and/or audio. However, the viewing user can be a user who can not only receive the above information but also react. For example, the viewing user may be a user who views a video distribution, a collaboration distribution, or a user who participates in and/or views a video chat, voice chat, or event. Therefore, the viewing user in the present disclosure can also be referred to as a guest user, participating user, listener, viewing user, supporting user, etc.

Furthermore, the information processing system in the embodiment of the present disclosure is a digital world in which many people can participate at the same time and engage in free activities at a level similar to the real world, such as interacting, working, and playing through character objects (avatars). , can be used to provide the next Internet space (metaverse) where social activities can transcend the gap between reality and virtuality.

In such a metaverse space, user avatars can freely walk around the world and communicate with each other.

Then, one avatar (character object) among the plurality of avatars in the metaverse space may be configured to be able to distribute the video as the character object of the distribution user. That is, it may be possible to perform one-to-many video distribution in a many-to-many metaverse space.

In such a metaverse space, distribution users and viewing users may not be particularly distinguished.

Next, with reference to FIGS. 4 to 13, a characteristic configuration regarding the movement of the avatar in the virtual space will be described. Note that the technology described below is applicable not only to metaverse space but also to non-metaverse space (for example, a space where there are distribution users and viewing users).

FIG. 4 is an image diagram of an action based on the predetermined skill "Lifting a Soccer Ball", and shows the avatar X1 lifting. FIG. 5 is an image diagram of an action based on the predetermined skill "playing the piano", and shows the avatar X2 playing the piano. FIG. 6 is an image diagram of an operation based on the predetermined skill "performance in a band", and FIG. 7 is an explanatory diagram of an example of a selection screen (user interface) for the predetermined skill. FIGS. 8A and 8B are explanatory diagrams of predetermined skills related to walking. FIG. 9A is an explanatory diagram of an example of a joint model, and FIG. 9B is a diagram illustrating the relationship between parts using the joint model. FIG. 10 is an explanatory diagram of changes in finger movement, FIG. 11 is an explanatory diagram of a function suitable for reflecting the characteristics of an avatar, and FIG. 12 is an explanatory diagram of a function suitable for reflecting the characteristics of an avatar. It is an explanatory diagram of a period. In this embodiment, as will be described later, "characteristics of the avatar" refers to a "switching algorithm related to automation of detailed motion expressions among the actions of the avatar."

It should be noted that various objects appearing in the following description (for example, a soccer ball, a piano, etc.) are objects in a virtual space, and are different from the real thing, unless otherwise specified. Further, various events in the following description are various events in virtual space (for example, a concert, etc.) and are different from events in reality. In addition, in the following explanation, when an avatar acquires (obtains) a certain thing, it means a transition from a state in which the thing is not associated with the avatar to a state in which the thing is associated with the avatar. represents. Further, as described above, one avatar and the user associated with the one avatar are in the same relationship, so they may be described without distinguishing between them. Note that a specification in which one user is associated with a plurality of avatars is also possible, but in this case as well, the one user selects one avatar and acts in the virtual space. Therefore, in this case as well, at each point in time, one avatar and the user associated with the one avatar are in a relationship that can be regarded as the same.

In this embodiment, the avatar's actions in the virtual space include normal actions in response to user input from the user, as well as actions based on predetermined skills. Note that the term "action" is a concept that includes not only actions that involve movement of the avatar, but also actions that do not involve movement of the avatar (for example, an action that makes a sound).

The action based on the predetermined skill is an action that cannot be performed by an avatar to which the predetermined skill is not associated, and is an action that can only be performed by an avatar to which the predetermined skill is associated.

Although the content and attributes of the predetermined skill are arbitrary, the predetermined skill may include, for example, sports skills such as lifting, shooting, and overhead shooting of a soccer ball, as schematically shown in FIG. As shown schematically, there are artistic skills such as piano performance, cooking skills, acting skills, singing skills and vocal volume skills, entertainment skills such as magic, transformation, spells, and magical skills. This may include game-related skills such as games, skills related to driving vehicles such as cars, etc. FIG. 6 schematically shows a performance scene by four avatars A1 to A4 who are associated with skills for playing various musical instruments.

In this embodiment, there are a plurality of types of predetermined skills, and the predetermined skills are associated with avatars that satisfy predetermined acquisition conditions. The predetermined acquisition condition may be set for each predetermined skill, or may be set for each attribute of the predetermined skill. The predetermined acquisition conditions are arbitrary, and include conditions based on the consumption or exchange of virtual reality media (for example, virtual currency), conditions based on the activities of the avatar in the virtual space, conditions based on predetermined items associated with the avatar, etc. Alternatively, a combination of these may be used. For example, if the predetermined skill is lifting a soccer ball as shown in FIG. 4, the predetermined acquisition condition for the predetermined skill may be the acquisition (possession) of a specific soccer ball, or a certificate of completion at a soccer class, etc. It may also be acquisition (possession) of (predetermined item). Furthermore, in the case where the predetermined skill is piano performance as shown in FIG. It may also be the acquisition (possession) of an item). Further, the predetermined specific conditions may be linked to training for acquiring actual knowledge and skills by watching e-learning videos, or may be provided by watching advertisements such as advertising videos (for example, for sports goods and musical instruments). Good too. Further, the predetermined specific condition may be interpreted as not only acquiring an item within the system but also importing a certificate from outside the system using an NFT marketplace. Furthermore, the predetermined specific conditions may be linked to the history of a block chain or the like created by an actual educational institution, and may be provided with a certificate of a course taken or a certification exam.

Further, the granularity of the predetermined skill is arbitrary, and in the case of lifting a soccer ball shown in FIG. 4, the predetermined skill may have a finer granularity and be different for each type of skill in lifting (for example, skills with different difficulty levels). . Alternatively, as a coarser granularity, the predetermined skill may be a skill for handling a soccer ball (for example, a skill including dribbling, etc.). Further, in the case of piano performance shown in FIG. 5, the predetermined skill may be different for each song (for example, songs with different difficulty levels). In this case, the predetermined acquisition condition related to the predetermined skill may be the acquisition (possession) of a musical score or a cassette tape (predetermined item).

Note that although the predetermined skills are associated with individual avatars, one predetermined skill may be associated with multiple avatars. The predetermined skill that can be associated with a plurality of avatars may be related to performance collaboration (performance in a band) as shown in FIG. 6, for example. In the case of sports, the predetermined skills that can be associated with a plurality of avatars may include, for example, joint skills performed by a plurality of avatars, such as passing a soccer ball, catching a baseball, batting, and the like. In addition, other activities such as tug-of-war, three-legged competition, and long rope jumping may be used. Note that the collaboration may be performed using the studio units 30A, 30B, etc., or may occur spontaneously. Furthermore, the skills for collaboration may have a level of proficiency set through collaborative cooperative play games with different difficulty levels, such as "rhythm game," "rock, paper, scissors," and "acchi maite hoi." Among these, in addition to the proficiency level of each individual user, skill improvement may be set based on cooperative play time with an individual partner. This has the effect of increasing game play time and in-game currency consumption, substantially improving communication between users, and eliminating cheating.

In this embodiment, the actions based on the predetermined skill include actions that can be drawn with animation. Here, the motion that can be drawn by animation is the motion that can be drawn regardless of user input from the user. As a result, it is not necessary to track all of the user's movements, so the processing load can be reduced, and the avatar's movements that reflect the user's characteristics can be expressed efficiently. In particular, even if the motion based on the predetermined skill involves a complicated motion, the processing load related to drawing can be reduced. That is, the processing load related to drawing can be reduced while accurately drawing the motion based on the predetermined skill. Furthermore, since the type of action based on the predetermined skill differs depending on the predetermined skill acquired by the user, it is possible to express the user's characteristics.

For example, if we take "Rock, Paper, Scissors" as an example of the aforementioned game for improving collaboration skills, "Rock, Paper, Scissors" allows drawing and network processing to be performed while focusing on only two people. Although it is unreasonable to continue transmitting motion data that requires high-quality, high-speed physical expression in real time to a large number of people in a space via the Internet, by acquiring this skill, we can Rather than tracking and sharing all movements, we use differences that can express characteristics within the motion data, and symbolic expressions necessary for the game system (symbol goo, symbol scissors, symbol par/rock, scissors, paper). It becomes possible to write with less load.

The operation based on the predetermined skill may be activated (automatically) regardless of subsequent user input from the user when a predetermined activation condition is met. Alternatively, the action based on the predetermined skill may be activated in response to a subsequent predetermined input from the user (input instructing activation) when a predetermined activation condition is met. Such differences may occur depending on the attributes of a given skill.

The predetermined activation condition is associated with a predetermined object in the virtual space. The predetermined object may optionally be associated with a corresponding predetermined skill. For example, when the predetermined skill is soccer ball lifting shown in FIG. 4, the predetermined activation condition may be related to a soccer ball (object) as the predetermined object. For example, the predetermined activation condition may be satisfied when the positional relationship between the predetermined object and the predetermined part of the avatar becomes a predetermined relationship. Although the predetermined relationship is arbitrary, it may include that the distance between the predetermined object and the predetermined part of the avatar becomes short. For example, the predetermined relationship is that when the distance is less than a threshold, when the predetermined part of the avatar is located within a predetermined range around the predetermined object, the coordinates in the virtual space where the predetermined part of the avatar is located and the predetermined object are This may be the case where the distance difference from the location coordinates is a predetermined condition. Furthermore, the predetermined relationship may be adapted for each predetermined skill so that natural movement of the avatar relative to the predetermined object is achieved. For example, the predetermined triggering condition may be met if the avatar drops a soccer ball near the avatar's feet. Further, in the case of playing the piano shown in FIG. 5, the predetermined skill may be satisfied when the avatar places a musical score on the piano, takes a seat, and places his/her hands on the keyboard.

The predetermined object to which a predetermined activation condition is related may be different for each predetermined activation condition, or may be common (the same) among a plurality of predetermined activation conditions. That is, one common (same) predetermined activation condition to which a plurality of predetermined skills are associated may be common (same) between different predetermined objects. In this case, for example, multiple types of musical scores (ie, predetermined skills) that can be played on a predetermined object such as a piano may be associated. Further, a plurality of predetermined skills such as lifting, dribbling, and shooting may be associated with a soccer ball (object). In this case, when the avatar and the object (in the above example, the piano or the soccer ball) are located within a predetermined distance, a common predetermined activation condition is satisfied.

Here, if multiple types of predetermined skills associated with one avatar are associated with the same predetermined activation condition in common, when the predetermined activation condition is met, multiple types of predetermined skills The one avatar may be able to select one or more of the skills. In this case, an action may be implemented based on one or more types of predetermined skills selected by the one avatar. Note that when two or more types of predetermined skills are selected, operations based on the selected predetermined skills may be performed sequentially or may be performed in combination.

In the example shown in FIG. 7, when a predetermined activation condition is met, a user interface with a musical note mark M70 appears, and two songs ("Song A" and "Song A" and " It includes user interfaces B71 and B72 that allow selection of either song B''). Note that the user interface of the musical note mark M70 and/or the user interfaces B71 and B72 may appear automatically when a predetermined activation condition is met, or may appear in response to user input from the user. It's okay. Note that in this case, when multiple types of predetermined skills that are associated with a common predetermined activation condition are associated with one avatar, all of the multiple types of predetermined skills may be selectable. , only some of them may be selectable. In this case, the selectable predetermined skills may change depending on the attributes of the avatar and its costume. For example, when the avatar's costume is "formal attire," only formal songs may be selectable, and when the avatar's costume is "heavy metal style," only rock-style songs may be selectable. .

Further, one predetermined activation condition may be associated with two or more predetermined objects at the same time. For example, one predetermined activation condition may include that the avatar possesses one predetermined object and approaches another one predetermined object.

Moreover, a plurality of activation conditions may be associated with one predetermined skill. For example, a plurality of activation conditions such as wearing pants, wearing sneakers, and wearing a jersey may be set for the skill "running". In this case, the predetermined objects associated with the plurality of activation conditions associated with one common predetermined skill are different from each other. That is, in the case of the above example, the predetermined objects associated therewith are pants, sneakers, and jersey, and are different from each other.

Further, the predetermined activation condition may be related to the predetermined object itself, or may be related to the action or movement of the predetermined object. For example, when the predetermined skill is "automatic reaction", the predetermined activation condition may be satisfied when a specific avatar is in a predetermined positional relationship as a predetermined object, and the specific avatar performs a specific action. . In this case, the automatic reaction may reflect the characteristics of the avatar, such as a "ladylike reaction," an "exaggerated reaction," or a "cold reaction." Furthermore, if a plurality of types of automatic reactions are associated with one avatar as a predetermined skill, the type of automatic reaction may be selectable by the one avatar. However, in this case, if the first avatar's costume is a kimono, there will be restrictions such as being able to only select a "ladylike reaction".There are certain skills that can be selected depending on the attributes of the avatar and its costume. May be limited.

As a supplement, automatic reactions are based on the social graph of SNS, specifically past contact history, and the relationship between the user and the other party, such as a first time meeting, an acquaintance, a close user, a super celebrity who is unilaterally following, and the social graph. The mode and frequency of reactions may change depending on the distance. For example, for an avatar that has the skill "Oshikatsu" (an activity to support the person you like) and is set to "Lady", the target "Super Celebrity" (Number of followers) When a user (who has an extremely small number of followers) comes into view, it automatically plays a flashy motion such as ``making heart-shaped eyes and waving their hands wide,'' but for other users, it performs normal actions. Something like this. Also, if the skill ``Friendly Laughter'' is high, it will generate a smile and greeting motion to everyone, but if the user also has the skill ``Melancholy,'' unless the user is particularly close to the user, he or she will make a motion to smile and greet everyone. It is also possible to set variations and priorities that combine multiple skills, such as removing the expression or reducing changes in facial expressions.

Further, one predetermined activation condition may include a condition that two or more types of specific predetermined skills are associated with each other. In this case, for example, if the user does not acquire both the musical score of a specific song and the skills to play the piano, a restriction can be realized, such as not being able to play even if the user goes near the piano.

Further, the predetermined skill may be related to various behaviors such as how to walk, how to speak, how to eat, etc. For example, when the predetermined skill is "walk gracefully (elegantly)", the predetermined activation condition may be related to the costume (object) of the avatar at that time as a predetermined object. For example, the predetermined activation condition may be satisfied when the avatar's costume is a costume (for example, a dress or a kimono) that suits or requires a "ladylike (elegant)" way of walking. In this case, even if the avatar Y1 is wearing the same dress, if the predetermined skill "walk gracefully" is not associated, then the avatar Y1 as shown schematically in FIG. 8A, On the other hand, if the predetermined skill ``walk gracefully (elegantly)'' is associated, it will be a ``ladylike (elegant)'' walk as schematically shown in Figure 8B. It becomes a way of walking.

Further, although not shown, when the predetermined skill is "run fast", the predetermined activation condition may be related to shoes (objects) worn by the avatar at that time, as the predetermined object. For example, the predetermined activation condition may be satisfied when the shoes worn by the avatar are specific athletic shoes. The particular athletic shoe may optionally be an athletic shoe that includes a history of use by a famous fast user (eg, an Olympic medalist) based on the object's ownership history. Note that the predetermined skill "run fast" is activated in response to a predetermined input from the user (an input instructing to "run") or an input from an acceleration sensor or the like built into a device such as a smartphone of the terminal device 20. It's okay.

Furthermore, the predetermined skills that can be selected may be limited depending on the attributes of an object such as an avatar's costume (object). For example, ``Run'' and ``Walk energetically! '', ``Skip'', ``Walk gracefully'', and ``Walk like a man with crotch legs''. When the user obtains a pair of high heels (such as high heels), a list of selectable walking styles may be displayed with the object. In this case, the user may be able to move in a walking manner selected by the user. However, in this case, if the object is a kimono, the predetermined skills that can be selected may be limited depending on the attributes of the object, such as a restriction that only skipping and walking gracefully can be selected. Furthermore, the options may be further narrowed down depending on the attributes of the avatar (gender, body type, etc.).

Also, although not shown, if the predetermined skill is "Karaoke", the predetermined activation condition may be satisfied when the avatar obtains the music score, goes near the microphone (object), and picks up the microphone. . In this case, a list of songs of the obtained musical score may be displayed, and choreography and dance suitable for the song selected by the user may be realized as movements based on predetermined skills. Note that in the case of karaoke, it may be possible to output audio based on user input. That is, the user may be able to select options such as singing the song himself or producing the sound of the performance himself.

Further, although not shown, when the predetermined skill is "transformation", the predetermined activation condition may be satisfied when a predetermined spell (virtual reality medium) is acquired. In addition to acquiring a spell, when the user selects a spell, that is, when the user's avatar wears a predetermined object, the user selects from options such as "cast a spell," or inputs the following: This condition may be satisfied when the user inputs the wording of the spell (such as "abracadabra") by inputting text or speaking. In these cases, the avatar obtains or equips (wears) a predetermined object (for example, a dress or suit for costumes), selects a spell, and transforms into a dress or squadron suit. A movement that transforms as an action (for example, a movement accompanied by an effect) may be realized as a movement based on a predetermined skill. Note that if the user does not have the predetermined skill "transformation" or the predetermined spell, the action may be simply changing clothes. Further, the predetermined skill of "transformation" may be processed for a limited period or number of times by consuming a paid object. Additionally, the effect of the skill may not be displayed until the first processing, and may be displayed under a different name such as "???" in order to arouse the user's interest and encourage them to consume related objects. .

Here, if the predetermined skill is the same, the behavior based on the predetermined skill may be common to all avatars to which the predetermined skill is associated, but preferably, the characteristics of each avatar can be expressed. It may be possible to be different for each avatar or for multiple avatars, so that For example, multiple animations may be prepared depending on the attributes of the avatar (eg, size, gender, etc.).

As a common skill, it may be distributed via an external system such as an NFT marketplace, or it may be distributed as an object in another game system, an advertisement, an incentive linked to the purchase of a product, or the viewing of a video. In that case, it may be played back as a more standardized animation process, or the avatar's size (invisible elements such as limbs, hair length, weight, etc.), personality rating, and attributes of the actual user operating the avatar (male/female) may be reproduced. , age, region, language) may allow for differences in the actions to be played. This is intended to avoid the fact that the same gesture has different meanings in different cultures (e.g., thumb up, fingers in a circle, middle finger up, little finger up, etc.) ).

As for multiple animations, depending on the attributes (personality information. The avatar's personality refers to, for example, hair color, clothing, gender, etc.) of the avatar or the object it is attached to, for example, if it is a lively attribute, it will be an animation that moves actively. If the data is a quiet attribute, animation data such as slow or small movement may be prepared. Further, the attributes of the avatar or the object to be worn may be set by the user selecting the attribute (or personality) from a plurality of options. Furthermore, when a plurality of avatar IDs are associated with one user ID, the user can arbitrarily set each attribute (or personality) for each avatar.

Incidentally, drawing using animation has the advantage that by creating the animation in advance, real-time drawing can be realized with a reduced processing load, but on the other hand, it has the disadvantage that it is difficult to express the characteristics of each avatar.

Therefore, in this embodiment, animation drawing may be applied only to a portion of each part of the avatar. In this case, first, we will explain the simplest drawing process that ignores joints. For example, in the case of an animation that expresses "I like" by holding up the thumb, the important parts are the shapes of the right hand and fingers. In this case, the joints beyond the right wrist can be represented by a previously prepared finger shape (arrangement of joint bending angles), and the joints on the arm side of the wrist can use data such as motion capture.

Next, to explain the drawing process that focuses on the joints, only a predetermined part of each movable part of the avatar is drawn using animation, and other parts are drawn using predetermined parts with a focus on the joints. It may be drawn in a manner that is linked to the movement of. In this case, when expressing other parts including their positions, they may be drawn using, for example, inverse kinematics (hereinafter sometimes abbreviated as "IK"). In this case, the avatar may be represented by any joint model. In this case, the joint model used is any model related to the joints of the avatar, for example, a model represented by a plurality of joints and a skeleton (link) between the joints. In this embodiment, a joint model as shown in FIG. 9A is used as an example. The joint model shown in FIG. 9A is a 16-joint model in which the head has one joint, and the torso (torso), both arms, and both legs each have three joints, and the head has both ends as joints. The other three points, both ends and the middle point, are defined as joints. Specifically, the joint model consists of 16 joints A0 to A15 and 15 skeletons B1 to B15 (also referred to as "parts B1 to B15") that connect the joints. As can be understood, for example, joints A4 and A7 are joints of the left and right shoulders, and joint A2 is a joint related to the cervical vertebrae. Further, joints A14 and A15 are left and right hip joints, and joint A0 is a joint related to the lumbar vertebrae. In this case, the avatar can be drawn as a model in which multiple movable parts are connected via multiple joints, as shown in FIG. 9B. The example shown in FIG. 9B is a model in which 11 movable parts Bp1 to Bp11 related to the avatar are connected via a plurality of joints (represented by black circles). Note that each movable part of the avatar does not necessarily correspond to each actual movable part of a person, and may be omitted, or other movable parts that humans do not have, such as wings or a tail, may be omitted. May be added.

Here, the drawing process may be performed using a method other than the IK method, such as a method in which the motion data of each part is stored in association with a skill, and when the skill is demonstrated, an animation is drawn based on the motion data. Alternatively, the animation data of the skill may be video data, and the video may be played back when the skill is activated. For example, an animation GIF file may be defined, and a specific video file may be selected and played.

The details of the drawing process will be explained as follows.

(1) Draw an avatar based on the animation data of each skill.

(2) The animation data includes motion data for each part of the avatar. There are cases in which the avatar is drawn based on the motion data of each skill, cases in which it is generated by an IK controller, and cases in which it is generated by logic.

(3) Note that the avatar may be drawn by operating the IK controller using logic or physical simulation. When an IK target whose target is set to the starting point bone is moved, the starting point bone and the target bone are set to move in conjunction with the movement of the target.

(4) In the case of the above method, only the movement of the target may be set in the animation data of each skill (eg, the movement of a lifting ball). When the skill is activated, the target and the linked bones are moved using data on the movement of the target and IK set in advance for the bones of the avatar to draw the avatar.

(5) In addition to structures connected as joints such as bones, the neck, line of sight, effects, etc. may be manipulated in the same way.

(6) If the skill is implemented using logic, branching using IF statements for each part of the given motion data, analysis of the audio stream, image recognition such as a reference music video, embedded data such as lyrics, or The animation to be played is loaded using judgments made by machine learning using a learning set that combines them.

(7) Motions may be set for each part of the body, such as hands, legs, and torso, using Unity layers and avatar masks. For example, regarding animation data of a certain skill, the animation of some parts of the avatar may be changed by setting an avatar mask, and may be stored as separate animation data. It is also possible to combine the motion set in the avatar mask and the IK method, and draw using a blending ratio of 0 to 1 or a blending model that changes over time (linear interpolation or spline interpolation).

The steps described above are explained as follows.

I. [Start]

(1) Data from motion capture

(2) Generation by logic

(3) Generation by IK
II. [Prioritization by characteristics]
For each of (1), (2), and (3) above, a contribution rate of -1 to +1 is set (blend parameter) depending on the avatar, the object worn by the avatar, the type selected by the user, and the like.

III. [Definition by avatar mask] x [blend parameter]
IV. [Animation generation]
This enables the following operations, for example.

・When playing the piano, emphasis is placed on performance according to (1), but (2) can sometimes lead to a posture that prevents the player from playing according to the score.

・When playing the piano, (2) means that the person plays according to the score, but (1) does not make movements such as shaking the head in time with the music.

・When lifting a soccer ball, (3) always chases the ball, but (1) occasionally looks around.

Note that in the above, "characteristics" can be understood as "switching algorithms involved in automating the expression of detailed motions among the avatar's actions." By selecting which is prioritized between logic generation, calculation generation, and motion capture data, it is possible to create a variety of expressions, such as whether there is more motion capture or less motion capture, etc., without having to explicitly select it. It can be trained and calibrated based on whether the movement is close to or far from the calculated movement, which can lead to diversification of individual avatar expressions.

Note that parameters related to the joint model (for example, degrees of freedom of joints, etc.) may differ for each avatar. Further, the lengths, thicknesses, etc. of the plurality of movable parts may also differ for each avatar.

The predetermined part is a part related to a predetermined skill, and may be a part whose movement is most complicated in an action based on the predetermined skill. For example, if the predetermined skill is lifting a soccer ball as shown in FIG. 4, the predetermined part may be a part (legs or head) that handles a soccer ball. According to such a configuration, for parts other than the predetermined part that are unrelated to the movement of the predetermined part, instead of drawing based on inverse kinematics or in combination with inverse kinematics, user operations can be applied to parts other than the predetermined part. Drawing based on this is also possible. In this case, it becomes possible to express the characteristics of each avatar. For example, if the predetermined skill is lifting a soccer ball as shown in FIG. 4, hand movements and the like may be drawn based on user operations. Similarly, if the predetermined skill is piano performance as shown in FIG. 5, it may be possible to draw head movements and the like based on user operations. Furthermore, in any case, instead of or in combination with drawing based on user operations, drawing based on tracking information acquired by motion capture may be realized.

Note that drawing based on user operations on a touch panel or the like may be realized in the same way as tracking information acquired by motion capture. For example, if the input unit 24 is a touch panel, by swiping up, down, left, or right, it is possible to instruct the position of the head, the movement of the hips, etc., and it is possible to create an effect in which the avatar's whole body sways. The same goes for button operations, and by creating a rhythm with your feet and neck, you can create an effect where your whole body sways.

In addition, if the specification allows the drawing of the avatar's facial expressions (e.g. movements of the eyes, mouth, etc.) and the accompanying parts (hair, hair ornaments, etc.), the avatar's facial expressions and movements based on predetermined skills can be drawn. The movement of the parts associated therewith may similarly be drawn by animation or may be drawn based on user operations. In addition, if the avatar's facial expressions (e.g. movements of eyes, mouth, etc.) can be drawn, the avatar's facial expressions during movement based on predetermined skills may be drawn using the user's facial image information or motion capture (face capture). ) may be drawn based on the tracking information obtained. Facial expressions are created using not only the user's own motion capture data, but also lip sync data generated from the voice input from the microphone and recorded sound sources, as well as trigger data that detects timing through audio signal processing. You may. It is difficult to open and close the mouth shape of a general-purpose avatar in order to control the appearance, but for example, it is difficult to open and close the mouth widely to control the appearance, but for example, it is possible to adjust the opening and closing of the mouth that is characteristic at specific singing timings, the way the hands are clenched into fists, the redness of the face, the sweat, etc. Cartoon-like exaggerated/emphasized expressions such as comic marks may be applied.

Further, regarding the movement of a predetermined part, animation drawing may be realized in a drawing mode that can differ for each avatar so that the characteristics of each avatar can be expressed. In this case, for example, the posture of a predetermined region at one point in time (hereinafter referred to as the "first posture") and the posture of the predetermined region at the next point in time (hereinafter referred to as "second posture") are different. When defined by animation data, the movement from the first posture to the second posture may be expressed using a function that defines the speed of movement of a predetermined part.

For example, FIG. 10 schematically shows a change from finger state A (first posture) to finger state B (second posture). In FIG. 10, the angle θ formed by two parts of the finger joint (represented by black ●) (hereinafter also referred to as "joint angle θ") is greater than 0 (state A), so joint angle θ= The temperature changes to 0 degrees (state B). In this case, a function selected from a plurality of types of functions F101 to F104 as shown in FIG. 11 according to the avatar's personality (for example, hair color, clothing, gender, etc.) may be used. Furthermore, for example, attributes of the avatar, such as an avatar wearing casual clothes playing the piano cheerfully, may be associated with the avatar's personality, such as hair color, clothing, gender, etc. Furthermore, the user may be able to select how to play the piano (that is, which function to apply). The functions F101 to F104 define changes in the joint angle θ according to time, with the horizontal axis representing time and the vertical axis representing the joint angle θ. This allows the user's characteristics, such as careful playing or aggressive playing, to be reflected in the fingering. Note that in FIG. 11, the function F101 is a linear function, but the other functions F102 to F104 are nonlinear functions. Such a function may be an easing function. Such a function is preferably applied to movements of fingers (predetermined parts) during piano performance, but can be applied to movements of any other predetermined parts. In addition, although four types of functions F101 to F104 are shown as an example in FIG. 11, more types of functions may be prepared, or fewer types of functions may be prepared. Furthermore, although only one joint angle θ is shown in FIGS. 10 and 11, a larger number of joint angles may be changed based on a similar function depending on the degree of freedom of movement of each joint. .

There are subtle differences in musical performance expressions and dance motions based on conventional music theory. For example, when there are four consecutive beats, there is a technique to make the first beat strong (strong beat) and the last beat weak and flowing (weak beat). Even if it is a basic continuous movement within the same performance or dance motion, accents such as strong beats and weak beats have a large impression on the viewer, and as a scientific phenomenon, they have a wide and certain effect. Is recognized. For example, by making the accents of the strong beats and weak beats the same, it is possible to create more beginner-like expressions, and by making the strong beats louder, including more front-driven movements, it is possible to parametrically implement emotional expressions. Become. This has the effect of eliminating the need to perform conventional manual animation work for all avatar animations.

Incidentally, an action based on a predetermined skill may continue for a relatively long time. For example, if the predetermined skill is lifting a soccer ball as shown in FIG. 4, the action based on the predetermined skill will continue for at least several seconds. Further, when the predetermined skill is piano performance as shown in FIG. 5, the operation based on the predetermined skill basically continues until the performance of the music is finished.

Therefore, the movements of the avatars may be drawn in a drawing manner that can be different for each avatar so that the characteristics of each avatar appear only for a specific period during an action based on a predetermined skill. In this case, the avatar may be drawn based on a user's operation only for a specific period of time, or may be drawn based on tracking information obtained by motion capture.

FIG. 12 is a diagram illustrating an example of how period attributes change during an operation based on a predetermined skill. The attribute M1 indicates a period during which animation drawing is applied to the movement of a predetermined body part, and the attribute M4 indicates a period during which the avatar's movement can be drawn in a drawing mode that may vary from avatar to avatar. When the predetermined skill is lifting a soccer ball as shown in FIG. may be set. Further, when the predetermined skill is piano performance shown in FIG. 5, the period of attribute M4 may be set for an interlude period or a pause period (rest period). In this case, during the period of attribute M4, the avatar can create an effect in which the whole body sways by keeping the rhythm with the feet and neck. For example, based on user operations or motion capture information, the avatar's entire body can sway by rhythmically moving its legs and neck, allowing each avatar to express their individuality.

FIG. 13 is a schematic flowchart showing an example of a specific drawing method when the predetermined skill is playing a musical instrument. FIG. 14 is an explanatory diagram of synchronous playback by a plurality of avatars.

In the example shown in FIG. 13, standard MIDI file data (SMF data) to be played is set (step S1400). The SMF data includes various information such as channel, event (key press or key release), time, pitch, and strength. Then, MIDI data is read (step S1402). When reading MIDI data, the MIDI data may be converted into data that is easy to handle. Next, track chunk classification processing is performed (step S1404). For example, the type of musical instrument and the type of hand used when playing (right hand, left hand, or both hands) are specified. Note that, when realizing real-time performance, the number of track chunks may be one (that is, it may be converted to format 0).

Next, event classification processing (step S1406) is executed, and it is determined whether the sound starts or ends. Next, data acquisition processing (step S1408) is executed to acquire pitch and velocity. Next, it is determined whether or not it is a chord (step S1410), and if it is a chord, a chord classification process (step S1412A) is executed, and if it is not a chord, a fingering confirmation process (step S1412B) is executed. do. The code classification process is a process of classifying various codes such as Am code, G code, and D7 code. The fingering confirmation process may utilize fingering data that may be used separately, or may be realized by estimation processing. Alternatively, the data may be classified in units based on music theory, such as one measure or four notes, and processed by a classifier. Similarly, similar processing units may be grouped together using machine learning, or a classifier may be generated that realizes various performances by minimizing the base motion by estimating subspecies. Next, wrist position calculation processing (step S1414) is executed. For example, the position of the wrist (see joints A6 and A9 in FIG. 9A) may be calculated based on the instrument size and pitch using the joint model described above. At this time, the distance to the wrist (rotation angle around the wrist) and the speed of the music being played (BPM = Beats Per Minute) may be taken into consideration. Furthermore, if the arm length is insufficient, the position of the upper body (including the arm) may be linked based on the above-mentioned inverse kinematics.

Next, difficulty determination processing (step S1416) is executed. The difficulty determination process may be performed based on the distance from the previous sound (movement distance) and the time until the next pronunciation calculated from the BPM. For example, in the case of a piano, the determination may be made based on the time and distance from key release to next key press. In this case, the difficulty determination process may be executed in such a manner that the longer the travel distance and the shorter the time, the higher the difficulty level. In addition, in the case of instruments such as pianos and percussion instruments, where the size of the front movement when hitting the key, the moment of impact, and the swinging movement are important, the time until the next sound calculated from the BPM and the avatar's character A special motion or playback speed may be set for the forward drive motion or swing motion. Note that the results of the difficulty determination process may be used for automatically generating facial expressions and the like. For example, for difficult fingering, facial expressions such as a pained expression may be realized. Furthermore, the avatar's face may change its expression to match the melody; for example, it may smile if the song is lively and rhythmic, or close its eyes if the song is slow. Next, animation event creation processing (step S1418) is executed. In the animation event creation process, movement of the wrist and deformation of the shape of the fingers are drawn as animation.

The processing shown in FIG. 13 can reduce the processing load because it is not necessary to track all of the user's movements, and can efficiently draw the avatar's movements (motions) that reflect the user's individuality. (playback) possible. Note that all steps of the process shown in FIG. 13 do not necessarily need to be executed, and some of them may be omitted depending on the desired animation process.

Furthermore, the processing shown in FIG. 13 can be applied to a plurality of musical instruments. For example, in the case of a guitar, processing for determining whether it is a tablature or a chord may be included. If it's tablature, you can assume that the same pitches have the same fingering. However, it is possible to play the same pitch in different places, and you can decide which direction to play by calculating the distance to the next note. Furthermore, the right hand may repeat the same movement, but it may also stop the movement or hold down the strings by reading rest information. Guitar rests may be clearly muted, and although they are difficult to see in the video, it is preferable to reproduce mute movements as finger movements. On the other hand, open strings (in the case of long notes) are difficult to visually recognize in the video, but it is preferable that the strings express the resounding sound. In the case of open strings, it is possible to use data from motion capture to represent unused fingers and arms, and to express vibrating strings with visual effects such as particles and flickering, or signals to vibration devices. . Furthermore, in the case of a guitar, the movement of the avatar's arm may be determined based on the above-mentioned inverse kinematics. Also, the case of drums may be basically the same as the case of guitar. However, in the case of drums, arm movements associated with irregular rhythms (for example, so-called "daka-da-ka-dan") rather than simple 8-beat or 4-beat rhythms may be adapted separately. Furthermore, this processing can be used for determining correct answers in rhythm games and music games in the metaverse, and for learning systems that allow three-dimensional subjective observation to learn correct performance movements. It is possible to detect, save, share, publish, and distribute the differences between user input and MIDI musical score operations as arrangements and individuality.

Furthermore, since it can be applied to multiple musical instruments, it is also possible to distribute musical instrument performances in real time. For example, a combination of piano + drums + guitar + bass + singing is possible, and if it is a piano, it is also possible to play duet. In this case, real-time generation for the purpose of distribution can be realized by generating an animation in an animation event using the acquired standard MIDI file data. Further, by adjusting the BPM (Beat per Minutes) and adjusting the start timing, synchronized playback as schematically shown in FIG. 14, for example, can be realized. In the example shown in FIG. 14, a user related to avatar A and a user related to avatar B are different users, and a form is shown in which collaboration distribution within the metaverse space is realized. In this case, the avatar associated with user A satisfies the predetermined activation condition by approaching a predetermined object A (drum in this example) with song X as a predetermined skill. On the other hand, the avatar associated with user B satisfies the predetermined activation condition by approaching a predetermined object B (in this example, a piano) with song X as a predetermined skill. As a result, the avatar associated with user A performs a drum performance as an action based on the predetermined skill, and the avatar associated with user B performs a piano performance as an action based on the predetermined skill. These performances are then synchronously reproduced based on BPM.

Furthermore, in the case of synchronous playback, it is also possible to have a player (user) like a conductor control the BPM (playback in accordance with the player's time). Furthermore, participatory collaboration is also possible, for example, during a performance by three people, other avatars can join later. Note that the participation timing may be limited to the prelude, interlude period, or the like. Further, the data may be stored asynchronously by being superimposed on pre-saved performance data of another person, or may be performed simultaneously using a preset world standard time as an accurate metronome click sound. This form of collaboration can be applied not only to music, but also to other activities and competitions (such as dance). Note that regarding the drawing process during synchronous playback, mixing (stereophonic sound) may be performed on the terminal device 20 side by client rendering, or synthesis may be performed on the server device 10 side. Additionally, other recording systems may be used, such as computer vision and audio signal analysis from video recording files. Further, tracking information based on motion capture may be locally saved on the terminal device 20 and shared with other users via the server device 10. In this case, simultaneous real-time playback or time asynchronous playback based on the saved data may be performed. Playback (shadow play) becomes possible. At this time, tracking information for each user (tracking information acquired by motion capture) may be aligned by measuring delays between users.

It is also possible to play and sing in combination with lip sync (Lip Synchronization). In this case, voice information from the user may be used.

In this embodiment, as described above, the action based on the predetermined skill can be an action that reflects the characteristics of the avatar, so when the action based on the predetermined skill is started, the field of view of the virtual camera related to the surrounding avatars is changed. It's okay. That is, the field of view of the virtual camera related to surrounding avatars may be automatically changed so that the avatars performing actions based on predetermined skills can be clearly seen. Furthermore, a virtual space image including an avatar that performs an action based on a predetermined skill may be additionally displayed on the display unit 23 of the terminal device 20 of the surrounding avatar. For example, a sub-screen may be added superimposed on or adjacent to the originally displayed main screen. This has the effect that surrounding avatars can easily understand the predetermined actions of avatars in the same virtual space. Further, instead of or in addition to the field of view of the virtual camera related to surrounding avatars, the field of view of the avatar that performs an action based on a predetermined skill may be automatically changed. In this case as well, the virtual space image including the reactions of the surrounding avatars may be additionally displayed on the display unit 23 of the terminal device 20 of the avatar performing the action of the predetermined skill.

When the field of view of an avatar that performs an action based on a predetermined skill is automatically changed, the field of view of the avatar that performs an action based on a predetermined skill is such that it can clearly see the predetermined action (for example, the automatic reaction described above) of surrounding avatars. It may be changed automatically. Based on the excitement of the songs and the audience's reactions that have been detected and set in advance, the avatar's specific actions (jumping at the same time, playing with their backs together, etc.), the audience's psyllium, fireworks, smoke, lighting, etc. Camera motions may be played to make the production more effective. Note that if only the field of view of the virtual camera related to surrounding avatars performing an action is changed (for example, if the field of view of an avatar performing an action based on a predetermined skill is not changed), the user's (predetermined skill It is possible to prevent the user (whose avatar performs actions based on the above) from getting sick. You can also make a specific avatar look only at you. For example, the skill "Blow a kiss to a user who gave a large gift" will not be directed to a specific user, but will be directed to all users who have given a gift of a certain amount, "Blow a kiss to yourself." It can be rendered at an angle that makes it look like someone has just been blown a kiss. It can be applied not only to music, but also to greetings, handshakes, line of sight, and direction cameras (in the case of an HMD, the user's viewpoint is substituted for that).

For example, the field of view of an avatar playing the piano as an action based on a predetermined skill changes so that it can see surrounding avatars that perform automatic reactions (such as nearby avatars clapping); It may be assumed that there is no particular change. In this case, the field of view of the peripheral avatar is arbitrary. In addition, in the field of view of the avatar playing the piano, surrounding avatars that perform automatic reactions are displayed, but the actual users corresponding to the surrounding avatars that perform automatic reactions are moving freely, and the world (reality) of the user Then, the user may be making any movement (for example, he or she may not be making any movements or looking at the avatar playing the piano). Here, intimacy between users may be expressed by frequently directing the gaze of surrounding avatars.

Alternatively, when an action based on a predetermined skill is started, guidance information for viewing an avatar performing an action based on a predetermined skill may be generated for surrounding avatars. Changes in the visual field of the virtual camera and guidance information become effective in the metaverse space. The guide information may be shown in text or voice, or may display information indicating the direction in which the avatar performing the action is located. For example, the guide information provides hints for decisions when a user in the metaverse space does not know what to do or where to go. It also serves as an opportunity to communicate with other avatars. In addition, in the case of a non-metaverse space for distribution, the field of view of the virtual camera is set based on the distribution avatar, so when the distribution avatar performs an action based on a predetermined skill, the viewing user can You can watch the action being performed.

Next, with reference to FIG. 15 and subsequent figures, an example of the functional configuration of each of the server device 10 and the terminal device 20 related to the actions of the avatar based on the above-mentioned predetermined skills will be described.

FIG. 15 is a schematic block diagram showing the functions of the server device 10 related to the actions of the avatar based on the above-mentioned predetermined skills. FIG. 16 is an explanatory diagram showing an example of data in the object information storage section 140. FIG. 17 is an explanatory diagram showing an example of data in the skill-related data storage section 142. FIG. 18 is an explanatory diagram showing an example of data in the user information storage section 144. FIG. 19 is an explanatory diagram showing an example of data in the avatar information storage section 146. In addition, in FIG. 16 (and similar figures such as FIG. 17 below), "***" indicates a state in which some information is stored, and "..." indicates a state in which similar information is repeatedly stored. shows.

As shown in FIG. 15, the server device 10 includes an object information storage section 140, a skill-related data storage section 142 (an example of first to third storage sections), a user information storage section 144 (an example of a fourth storage section), It includes an avatar information storage section 146, a user input acquisition section 150 (an example of an input acquisition section), an avatar processing section 152, an activation condition determination section 154 (an example of a determination section), and a visual field change processing section 156.

Note that in FIG. 15, the object information storage section 140, skill-related data storage section 142, user information storage section 144, and avatar information storage section 146 can be realized by the server storage section 12 of the server device 10 shown in FIG. Further, the functions of each section from the user input acquisition section 150 to the visual field change processing section 156 can be realized by the server control section 13 and the server communication section 11 of the server device 10 shown in FIG.

The object information storage unit 140 stores object information regarding each object (including the above-mentioned predetermined object) placed in the virtual space. For example, as shown in FIG. 16, the object information may include object attributes, position information, and drawing information for each object ID. The object ID is an ID that is automatically generated when each object is generated. The object attribute represents the attribute of each object, and the position information is position information of each object (position information in virtual space). Note that the position information regarding an object whose object attribute is a moving object may be updated as appropriate. The drawing information may include information necessary to draw each object.

The skill-related data storage unit 142 stores skill-related data related to each of the above-mentioned predetermined skills. For example, as shown in FIG. 17, the skill-related data may include, for each skill ID, skill attributes, activation condition information, animation data (an example of predetermined data for drawing motions), and reproduction information.

The skill ID is an ID that is automatically generated when each predetermined skill is generated. The skill attributes represent attributes of a predetermined skill, and the attributes of the predetermined skill may include information such as whether the skill is for collaboration or not. The activation condition information may include information representing the above-described predetermined activation condition.

The animation data includes animation data for drawing an action based on the above-mentioned predetermined skill. Regarding animation drawing, in addition to generating a video in advance and playing it back, the animation data includes motion data (data on how to move each part) of the avatar and object used to draw the animation. , the avatar or object may be drawn based on the motion data of the avatar or object. As described above, the animation data may be data related only to the movement of a predetermined part of the avatar. In this case, animation data can be generated so that it can be shared among a plurality of avatars, and data efficiency can be improved. For example, the animation data may be generated based on a model as described above with reference to FIGS. 9A and 9B, which can be shared among a plurality of avatars. Note that the animation data may be generated based on tracking information obtained when an actual person (user) performs an actual action related to a predetermined skill. Note that animation drawing is not limited to IK drawing.

The playback information includes basic information representing playback start conditions and playback end conditions of animation data. Further, when a specific period is set, such as the period of attribute M4 described above with reference to FIG. 12, the reproduction information may include information representing the length, start timing, etc. of the specific period. Furthermore, the reproduction information may include information for synchronous reproduction. Furthermore, in order to represent the user's individuality, the reproduction information may include personality information for reflecting the user's individuality in actions based on predetermined skills.

The user information storage unit 144 stores information regarding each user. Information regarding each user also stores object information owned by the user. Information regarding each user may be generated, for example, at the time of user registration, and may be updated as appropriate thereafter. For example, in the example shown in FIG. 18, the user information storage unit 144 stores each item of user name, avatar ID, possessed skill information, conversation information, activity information, friend information, and preference information in correspondence for each user ID. It is being

The user ID is an ID automatically generated at the time of user registration.

The user name is a name registered by each user and is arbitrary.

The avatar ID is an ID representing the avatar used by the user. A plurality of avatar IDs may be stored for one user. The avatar ID may be associated with avatar drawing information (see FIG. 19) for drawing the corresponding avatar. Note that the avatar drawing information associated with one avatar ID may be able to be added, edited, etc. based on input from the corresponding user.

Possessed skill information represents one or more predetermined skills associated with the corresponding avatar. Possessed skill information may be updated when the avatar acquires a new predetermined skill or loses a predetermined skill.

Conversation information represents information regarding the content of utterances made by the corresponding avatar in the virtual space. The conversation information may be data converted into text. The conversation information may also include information indicating which language is spoken (for example, locale ID). Further, the conversation information may include information regarding first-person address, wording, dialect, and the like.

The activity information includes information representing various activity histories in the virtual space. The various activities may include information representing not only special activities such as event participation and event hosting, but also regular activities such as access to a virtual space, time spent in the virtual space, and the like. Note that the activity information may be used to determine whether a predetermined skill acquisition condition (that is, a predetermined acquisition condition) is met.

The friend information may be information (for example, a user ID) that can identify a user in a friend relationship. Friend information may include information indicating the presence or absence and degree of interaction and friendship between users. Note that the friend information may be information on follows, followers, and mutual followers (a state in which they follow each other).

Preference information represents the preferences of the corresponding user. The preference information is optional, but may include the user's preferred language settings and preferred keywords. Further, it may be possible for the user to set things he/she likes or dislikes in advance, and in this case, the preference information may include such settings. Preference information may also include user profile information. The preference information may be selected via a user interface generated on the terminal device 20 and provided to the server device 10 as a JSON (JavaScript Object Notation) request or the like. Further, preference information may be automatically extracted based on conversation information, action history, or the like. Here, the data in the user information storage unit 144 may be used to reflect the user's individuality in actions based on predetermined skills, as described above. For example, since the conversation information represents the user's personality, it may be used as personality information to reflect the user's personality in actions based on predetermined skills. The same applies to preference information and the like. Personality information may be used, for example, as described above, to prepare multiple animation data according to personality information, or as shown in FIG. 11, to prepare multiple functions according to personality information. may be used for.

The avatar information storage unit 146 stores avatar drawing information for drawing each user's avatar. In the example shown in FIG. 19, in the avatar drawing information, each avatar ID is associated with a face part ID, a hairstyle part ID, a clothing part ID, and the like. Appearance-related part information such as face part ID, hairstyle part ID, and clothing part ID are parameters that characterize the avatar, and may be selected by each user. All or part of the avatar drawing information is stored in association with the user ID, and can be selected from among them and associated with each avatar. At that time, object information associated with the user ID is also associated with (attached to) each avatar. For example, a plurality of types of appearance information such as face part ID, hairstyle part ID, clothing part ID, etc. related to the avatar are prepared. Regarding facial part IDs, a part ID is prepared for each type of face, such as face shape, eyes, mouth, nose, etc., and information related to facial part IDs is managed by combining the IDs of each part that makes up the face. may be done. In this case, each avatar can be drawn not only on the server device 10 but also on the terminal device 20 side based on each ID related to appearance linked to each avatar ID.

Here, the data in the avatar information storage unit 146 may be used to reflect the user's personality in actions based on predetermined skills, as described above. For example, the avatar's clothes, hairstyle, etc. may be used as personality information to reflect the user's personality in actions based on predetermined skills, since they represent the user's personality. As described above, personality information such as the avatar's clothing may be associated with the avatar's attributes such as how to play the piano. Furthermore, the personality information may be used to prepare multiple pieces of animation data or multiple functions.

Further, although not shown in FIG. 19, the avatar information storage unit 146 may store various parameters related to the joint model (positions of each joint, range of motion, etc.) for each avatar.

The user input acquisition unit 150 acquires various user inputs input by each user via the input unit 24 of the terminal device 20. The various inputs are as described above, and may include tracking information obtained by motion capture.

The avatar processing unit 152 causes the avatar to perform an action based on a predetermined skill when a predetermined activation condition is met.

The avatar processing section 152 includes an avatar motion processing section 1521 and a drawing processing section 1522.

The avatar motion processing unit 1521 determines the motion of the avatar (change in position, movement of each part, etc.) for each avatar based on various inputs from each corresponding user.

The drawing processing unit 1522 generates an image for viewing on the terminal device 20 (terminal image), which is an image of a virtual space including an avatar. The drawing processing unit 1522 generates an image for each avatar (image for the terminal device 20) based on the virtual camera associated with each avatar. In this embodiment, the drawing processing unit 1522 draws an avatar that performs an action based on a predetermined skill, or an avatar and a predetermined object. In this case, the drawing processing unit 1522 can draw the action based on the predetermined skill with a reduced processing load based on the skill-related data in the skill-related data storage unit 142 shown in FIG.

The activation condition determination unit 154 determines whether or not a predetermined activation condition is met when a predetermined skill is associated with one avatar. The predetermined activation conditions are as described above. The activation condition determining unit 154 may determine whether a predetermined activation condition is satisfied based on activation condition information in the skill-related data in the skill-related data storage unit 142 shown in FIG. 17.

When one avatar performs an action based on a predetermined skill, the visual field change processing unit 156 determines the position of other avatars located around the one avatar (i.e., surrounding avatars) based on the position information of the one avatar. Automatically change the field of view. Specifically, the visual field change processing unit 156 changes each value of the imaging parameter of the virtual camera associated with the peripheral avatar so that the one avatar falls within the visual field of the peripheral avatar. As a result, as described above, surrounding avatars can easily notice an avatar that performs an action based on a predetermined skill, and interaction between the two can be promoted. Further, at this time, the visual field change processing unit 156 may change the line of sight of the avatar that performs an action based on a predetermined skill to position information of surrounding avatars that perform a reaction. Specifically, the visual field change processing unit 156 changes each value of the imaging parameter of the virtual camera that is associated with the avatar that performs an action based on a predetermined skill so that the peripheral avatar comes within the visual field of the avatar. You may let them.

In addition, as described above, the visual field change processing unit 156 is configured to perform processing when some of the surrounding avatars react (for example, react by clapping or waving their hands) to an action based on a predetermined skill performed by one avatar. Additionally, the field of view of the avatar that performs actions based on predetermined skills may be changed. That is, the field of view of the avatar performing an action based on a predetermined skill may be changed so that surrounding avatars performing a reaction come within the field of view of the avatar performing an action based on the predetermined skill. Note that when the reaction is the automatic reaction described above, the field of view of the surrounding avatar that performs the automatic reaction does not need to be changed. Thereby, it is possible to prevent the user from becoming motion sickness (particularly "sickness" when wearing a head-mounted display) related to the surrounding avatars due to automatic changes in the field of view. For example, if surrounding avatar B is talking with another surrounding avatar C without looking at avatar A that performs an action based on a predetermined skill, surrounding avatar B will automatically react when seen from avatar A. However, the peripheral avatar B itself is still conversing with another peripheral avatar C. In this case, the actions of the surrounding avatar B in the terminal images related to the respective users of the avatar A and the surrounding avatar B will be different.

In addition, as described above, the visual field change processing unit 156 is configured to perform processing when some of the surrounding avatars react (for example, react by clapping or waving their hands) to an action based on a predetermined skill performed by one avatar. Additionally, the field of view of other avatars (further peripheral avatars) located around the peripheral avatar making the reaction may be changed. That is, the field of view of the further peripheral avatar may be changed so that the peripheral avatar that performs the reaction is included in the field of view of the further peripheral avatar. In this case, the additional surrounding avatars may be part of the surrounding avatars for the avatar that performs an action based on a predetermined skill, or may include different avatars.

Note that the division of functions between the server device 10 and the terminal device 20 described above is for server rendering only as an example, and can be variously changed as described above. That is, some or all of the functions of the server device 10 may be realized by the terminal device 20 as appropriate. In server rendering, a video drawn by the server control unit 13 (avatar processing unit 152) of the server device 10 is transmitted to the terminal device 20, and the video is displayed on the display unit 23 of the terminal device 20, that is, an operation is performed from the terminal device 20. Information is collected, a video is generated by the server device 10, and is broadcast to each terminal device 20 (it is visualized from the beginning), but in addition to this, client rendering, browser rendering, In each case, the results are as follows.

First, in the case of client rendering, the terminal device 20 receives and holds images such as avatar objects and drawing programs from the server device 10 in advance. The terminal device 20 receives avatar identification information, motion data, and audio data from the server device 10 . The terminal device 20 calls an already held image such as an avatar according to the received identification information and motion data, and renders a moving image, that is, generates and displays a moving image.

Next, in the case of browser rendering, first download an HTML document, download necessary items such as images and a program (JavaScript) that runs on the browser to the server device 20 described in the HTML document, and The processing data of each terminal is received from the computer and drawn using a browser-side program (JavaScript). Although the language is HTML, it is similar to client rendering by an app, but once you download image data such as avatars, you can leave it in the app, but the browser stores it in RAM, and when you close it, it is saved on the terminal device 20 side. The difference is that nothing remains. Browsers that can save only URLs using cookies must download avatar images and the like each time. Where to download the avatar from, that is, from which server, is described in the HTML document.

Next, with reference to FIG. 20 and subsequent figures, an example of the operation of the virtual reality generation system 1 related to the operation of the avatar based on the above-mentioned predetermined skill will be described.

FIG. 20 is a timing chart schematically showing an example of the operation of the virtual reality generation system 1 related to the avatar's movement based on the above-mentioned predetermined skill.

FIG. 20 is a schematic flowchart showing an example of a process executed by the server device 10 in relation to the avatar's movement based on the above-mentioned predetermined skill. The process shown in FIG. 20 is a process related to a specific avatar (hereinafter also referred to as "target avatar"), and may be repeatedly executed at predetermined intervals. The process shown in FIG. 20 may be executed in parallel for each avatar in the virtual space. In the following, it is assumed that one or more predetermined skills are associated with the target avatar.

In step S200, the server device 10 obtains the current cycle's user input from the user corresponding to the target avatar.

In step S201, the server device 10 determines whether the animation playback flag F1 is "0". A state in which the animation playback flag F1 is "1" represents a state in which animation data is being played in relation to the avatar's movement based on the above-mentioned predetermined skill, and a state in which the animation playback flag F1 is "0" is Represents other states. If the determination result is "YES", the process advances to step S202; otherwise, the process advances to step S214.

In step S202, the server device 10 extracts activation condition information regarding one or more predetermined skills associated with the target avatar.

In step S204, the server device 10 extracts object information regarding each object placed around the target avatar. In this case, the object whose object information is to be extracted may be only the object related to the activation condition information extracted in step S202.

In step S206, the server device 10 determines a predetermined skill related to one or more predetermined skills associated with the target avatar based on the activation condition information obtained in step S202 and the object information obtained in step S204. Determine whether the activation conditions are met. If the determination result is "YES", the process advances to step S208; otherwise, the process advances to step S212.

In step S208, the server device 10 sets the animation playback flag F1 to "1".

In step S210, the server device 10 sets animation data and reproduction information (see FIG. 17) regarding a predetermined skill (hereinafter also referred to as "target predetermined skill") for which a predetermined activation condition is satisfied.

In step S212, the server device 10 causes the target avatar to perform normal actions (for example, running, walking, jumping, etc.) based on various inputs by the user. Note that when no user input is obtained (that is, when the user has not made any input), the target avatar may be maintained in a predetermined stationary state.

In step S214, the server device 10 determines whether a condition for ending the reproduction of the animation data is satisfied during the reproduction. The playback end condition may be satisfied at the end of the animation data, or may be satisfied based on a predetermined user input (end instruction) or the like before the end of the animation data. If the determination result is "NO", the process advances to step S216; otherwise, the process advances to step S218.

In step S216, the server device 10 causes the target to perform an operation based on a predetermined skill.

In step S218, the server device 10 ends the operation based on the target predetermined skill that was being executed in step S216 until the previous processing cycle.

In step S220, the server device 10 resets the animation playback flag F1 to "0".

FIG. 21 is a schematic flowchart showing an example of a drawing process regarding a target avatar that is executed in connection with the process shown in FIG. 20.

In step S230, the server device 10 determines whether the animation playback flag F1 is "1". If the determination result is "YES", the process advances to step S231; otherwise, the process advances to step S235.

In step S231, the server device 10 determines whether or not it is within a predetermined time immediately after the animation playback flag F1 becomes "1". That is, it is determined whether the elapsed time from the start of the movement based on the target's predetermined skill is within a predetermined time. In the server device 10, the predetermined time is a time for making surrounding avatars aware of the action based on the target's predetermined skill, and may be a fixed value or a variable value. If the determination result is "YES", the process advances to step S232; otherwise, the process advances to step S233.

In step S232, the server device 10 forcibly (and temporarily) directs the lines of sight of surrounding avatars toward the target avatar. That is, the line of sight of the virtual camera is changed so that the target avatar is included within the field of view of the virtual camera for the user related to the surrounding avatars. Note that the surrounding avatars may include avatars within a predetermined distance from the target avatar. The line of sight of the surrounding avatars may be temporarily directed toward the target avatar (for example, only for a predetermined time in step S231). Note that the server device 10 does not need to change the line of sight of all surrounding avatars, and may change only the line of sight of surrounding avatars that are likely to be interested in the action based on the predetermined skill of the target. In this case, the processing load can be more efficiently reduced than in the case where the line of sight of all surrounding avatars is changed. Whether or not the user is interested in an action based on a predetermined skill may be determined based on user information (for example, preference information) of surrounding avatars. For example, when the action based on the predetermined skill is lifting a soccer ball, surrounding avatars who like soccer may be extracted based on the preference information, and only the line of sight of the extracted surrounding avatars may be changed.

In step S233, the server device 10 determines whether there is a virtual camera that the target avatar can see within its field of view, among the virtual cameras for users related to surrounding avatars. That is, it is determined whether there are any nearby avatars viewing the target avatar. If the determination result is "YES", the process advances to step S234; otherwise, the process advances to step S235.

In step S234, the server device 10 performs a process (hereinafter also simply referred to as "animation playback process") of animation data related to the target avatar based on the animation data and playback information set in step S210 (see FIG. 20). Execute. A specific example of the animation playback process will be described later with reference to FIG. 22. In this embodiment, in step S234, the target avatar or the target avatar and the predetermined object are drawn.

In step S235, the server device 10 executes normal drawing processing. At this time, if the animation playback flag F1 is "0" and the target avatar exists within the field of view of the user's virtual camera related to the surrounding avatars, the normal processing in step S212 (see FIG. 20) is performed. The target avatar in action will be drawn.

Note that among the processes shown in FIG. 21, a part of the drawing process regarding the target avatar may be changed. For example, if it is desired to direct the viewpoints of surrounding avatars toward the target avatar regardless of the predetermined time from the start of the action based on the predetermined skill, step S231 and step 233 may be omitted. In that case, the flowchart of FIG. 20 changes from step S230 to step S232 to step S234. Further, even if step S231 and step 233 are omitted or not, step S232 and step S234 may be performed simultaneously or in parallel. Furthermore, if the viewpoints of the surrounding avatars are not directed toward the target avatar, in addition to steps S231 and S233, step S232 may also be omitted.

FIG. 22 is a schematic flowchart showing an example of the animation playback process (step S234 in FIG. 21).

In step S240, the server device 10 performs avatar drawing processing based on the animation data of each skill.

As described above, several methods may be adopted for the drawing process, but when inverse kinematics is used in step S240, first, in step S241, the server device 10 A drawing process is performed on the above-mentioned predetermined portion based on reproduction of animation data. Next, in step S242, the server device 10 performs a drawing process for each part of the target avatar other than the predetermined part based on inverse kinematics based on the position and orientation of the predetermined part. .

In step S244, the server device 10 reflects the characteristics of other parts of the target avatar (for example, facial expressions, etc.) based on the tracking information acquired by motion capture among the user inputs related to the target avatar. Performs drawing processing for. In the drawing process, the tracking information of motion capture may be used to draw not only the body part but also the period described using FIG. 12 . Note that if the characteristics are not reflected, step S244 may be omitted.

In step S246, the server device 10 uses the drawing processing results from steps S240 to S244 to generate images for each terminal (images for viewing on the terminal device 20) corresponding to the line of sight from each virtual camera.

The features of each of the embodiments described above or the differences from motions developed in the prior art are summarized as follows.
- Contains elements that react to the environment and relationships with other users (friends).
・In an asymmetrical relationship between broadcasters and viewers, there is an element in which the reaction changes depending on the support of the viewers.
・Contains elements that can be widely used in general-purpose music, such as MIDI files, audio signal processing, and video recording.
- It includes the possibility of clarifying the handling of videos containing performances (music copyright processing, differential complementation and reuse of motion data) based on the elements of UGC based on the user's own individuality.
- You can resell the UGC you create, and if it becomes popular, you will receive incentives depending on the number of views.
・By transferring skill objects (specifically, collaborating as a virtual trainer), communication usage with users with specific popular skills can be strengthened.
・Similarly, it is also good to be consistent with existing games by removing skills in competitive games and duplicating them in cooperative games.

Although each embodiment has been described in detail above, it is not limited to a specific embodiment, and various modifications and changes can be made within the scope of the claims. It is also possible to combine all or a plurality of the components of the embodiments described above.

For example, in the embodiment described above, the three elements of the predetermined skill, the predetermined activation condition, and the predetermined object are the relationships in which the predetermined skill is associated with the predetermined activation condition, and the predetermined object is associated with the predetermined activation condition. It is. Therefore, equivalently, it can be said that a predetermined object is associated with a predetermined skill via a common predetermined activation condition. That is, it can also be said that a predetermined skill is associated with a predetermined object, and the predetermined object is associated with a predetermined activation condition. In this case, one or more skill IDs may be stored in the object information storage unit 140 in association with the object ID. In this case, when a predetermined activation condition related to a specific predetermined object is satisfied regarding the avatar to which the predetermined skill is associated, an action based on the predetermined skill associated with the specific predetermined object may be realized. . For example, if a soccer ball (a predetermined object) is associated with various predetermined skills such as lifting, shooting, and dribbling, an avatar to which lifting is associated as a predetermined skill is within a predetermined distance from the soccer ball. If the avatar is located at , the predetermined activation condition related to the soccer ball is satisfied for the avatar. Therefore, in this case, the avatar can perform an action based on lifting (predetermined skill). Further, when an avatar to which lifting and shooting are associated as predetermined skills is located within a predetermined distance from a soccer ball, a predetermined activation condition related to the soccer ball is satisfied for the avatar. Therefore, in this case, the avatar can perform an action based on the selected predetermined skill of lifting or shooting.

In addition to the embodiments described above, it is also possible to implement duplication and transfer of skill objects through collaboration. For example, you can continue to observe an avatar lifting a soccer ball at close range, have a conversation for a certain amount of time, play a mini-game, accomplish a certain task in a cooperative play game, or defeat your opponent in a competitive play game. Through actions such as , the aforementioned skill objects can be rationally duplicated, transferred, or taken away. Many applicable examples can be set, such as contests at music events (skills that can only be used by those who win the air guitar contest) and special moves in fighting games.

1 Virtual reality generation system 3 Network 10 Server device 11 Server communication section 12 Server storage section 13 Server control section 20, 20A, 20B Terminal device 21 Terminal communication section 22 Terminal storage section 23 Display section 24 Input section 25 Terminal control section 30A, 30B Studio unit 140 Object information storage section 142 Skill-related data storage section 144 User information storage section 146 Avatar information storage section 150 User input acquisition section 152 Avatar processing section 1521 Avatar movement processing section 1522 Drawing processing section 154 Activation condition determination section 156 Visual field change processing Department

Claims (21)

an avatar processing unit that processes avatar movements in virtual space;
a first storage unit that stores predetermined skills related to the avatar's actions and that can be associated with the avatar;
a second storage unit that stores activation conditions associated with the predetermined skill;
a determination unit that determines whether the activation condition is met when the predetermined skill is associated with one target avatar;
The activation condition is related to a predetermined object in the virtual space,
The avatar processing unit is an information processing system that causes the target avatar to perform an action based on the predetermined skill when the activation condition is satisfied. The information processing system according to claim 1, wherein the activation condition is related to a positional relationship between the predetermined object and the target avatar. The information processing system according to claim 2, wherein the activation condition is satisfied when a positional relationship between the predetermined object and a predetermined part of the target avatar becomes a predetermined relationship. The predetermined object includes a predetermined item that can be worn by the avatar,
The information processing system according to claim 1, wherein the activation condition is satisfied when the target avatar wears the predetermined item. The first storage unit stores a plurality of types of the predetermined skills,
The information processing system according to claim 1, wherein the second storage unit stores a plurality of types of the activation conditions in such a manner that one or more of the activation conditions are associated with one of the predetermined skills. further comprising an input acquisition unit that acquires a user input associated with the target avatar,
The avatar processing unit is configured to perform one of the plurality of predetermined skills related to the one common activation condition when one common activation condition associated with the plurality of predetermined skills is satisfied. The information processing system according to claim 5, wherein the target avatar is caused to perform an action based on one or more of the predetermined skills selected based on user input. The information processing system according to claim 5, wherein at least two of the plurality of types of the activation conditions are associated with one of the predetermined skills, and the predetermined objects associated with each of the activation conditions are different from each other. . further comprising a third storage unit that stores predetermined data for action drawing in association with the predetermined skill;
The avatar processing unit includes a drawing processing unit that draws the target avatar that performs an action based on the predetermined skill,
The information processing system according to claim 1, wherein the drawing processing unit draws the predetermined object and the target avatar based on the predetermined data. further comprising an input acquisition unit that acquires a user input associated with the target avatar,
The drawing processing unit draws a predetermined part of each movable part of the target avatar based on the predetermined data, and also draws at least a part of the other movable parts in response to the user input. The information processing system according to claim 8, wherein the information processing system performs drawing based on the information processing system. The information processing system according to claim 9, wherein the drawing processing unit draws the predetermined part and the predetermined object in an interlocking manner. The information processing system according to claim 10, wherein the drawing processing unit draws a part of each movable part of the target avatar that is different from the predetermined part in a manner linked to movement of the predetermined part. The information processing system according to claim 9, wherein the drawing processing unit is capable of drawing the predetermined region based on the user input for at least a part of the predetermined period during the operation based on the predetermined skill. further comprising a fourth storage unit that stores personality information related to the personality of the user associated with the avatar or the avatar itself;
The information processing system according to claim 8, wherein the drawing processing unit draws the target avatar based on the predetermined data and the personality information related to the target avatar during an operation based on the predetermined skill. The information processing system according to claim 8, wherein the predetermined data includes animation data. The information processing system according to any one of claims 8 to 13, wherein the predetermined data includes data shared among a plurality of avatars including the target avatar. The first storage unit stores a plurality of types of the predetermined skills,
The information processing system according to claim 8, wherein the third storage unit stores a plurality of types of the predetermined data in such a manner that one or more of the predetermined data are associated with one of the predetermined skills. The method further comprises a visual field change processing unit that changes the visual field of other avatars located around the target avatar based on position information of the target avatar when the target avatar performs an action based on the predetermined skill. The information processing system according to item 1. The information processing system according to claim 17, wherein the visual field change processing unit changes the visual field of the other avatar so that the target avatar is within the visual field of the other avatar. When another avatar located around the target avatar performs a predetermined action related to the action based on the predetermined skill performed by the target avatar, the other avatar enters the field of view of the target avatar. The information processing system according to claim 1, further comprising a visual field change processing unit that changes the visual field of the target avatar. A computer-readable storage medium for storing information,
avatar processing that processes avatar movements in virtual space;
Judgment processing and
The program that the computer executes is stored,
The determination process is
Determining whether a predetermined skill regarding the behavior of the avatar in the virtual space is associated with the target avatar;
In the case where the predetermined skill is associated with the target avatar, determining the success or failure of an activation condition associated with the predetermined skill and associated with a predetermined object in the virtual space. ,
The avatar processing includes:
A computer-readable storage medium for storing information, the storage medium comprising causing the target avatar to perform an action based on the predetermined skill when the activation condition is satisfied. Execute avatar processing to process avatar movements in virtual space,
including performing a determination process;
The determination process is
Determining whether a predetermined skill related to the avatar's movement in the virtual space is associated with the target avatar;
When the predetermined skill is associated with the target avatar, the method includes determining whether or not an activation condition associated with the predetermined skill and associated with a predetermined object in the virtual space is successful. ,
The avatar processing includes:
An information processing method executed by a computer, including causing the target avatar to perform an action based on the predetermined skill when the activation condition is satisfied.