JP7442107B2 - Video playback device, video playback method, and video distribution system - Google Patents

Description

The disclosure herein mainly relates to a video playback device, a video playback method, and a video distribution system.

2. Description of the Related Art A video distribution system is known in which a viewing user who views a video can participate in the video using an avatar. For example, Japanese Unexamined Patent Publication No. 2012-120098 (Patent Document 1) describes including the avatar of a viewing user in a distributed video.

Japanese Patent Application Publication No. 2012-120098

In a video distribution service that distributes videos, it is possible to increase the engagement of viewing users by reflecting feedback from viewing users in the videos being distributed. For example, Patent Document 1 described above describes displaying comments from a viewing user in association with the avatar of the viewing user. In the system of Patent Document 1, viewing users provide feedback to the distributed video by posting comments, and this feedback is reflected in the distributed video.

If the distributed content can include an animation of an avatar that moves in response to not only comments from the viewing user but also the actions of the viewing user, it is possible to further increase the engagement of the viewing user. However, in order to move the avatar according to the actions of the viewing user, motion data indicating the actions of the viewing user must be sent in real time from the video playback device of the viewing user to a video generation device that generates the video. Is required. In order to cause the avatar to perform movements that reflect the user's actions, it is necessary to include bone data that indicates the positions and orientations of the user's bones in the motion data that indicates the user's actions. The position and orientation of bones are expressed as three-dimensional vectors. In order to express the movement of a human body, bone data for 20 or more bones may be required.

As described above, since the amount of bone data is large, it is difficult to transmit motion data including bone data indicating the posture of the viewing user without delay depending on the bandwidth of the transmission path and the degree of congestion. In particular, since the uplink from the user equipment has a smaller transmission capacity than the downlink used for content distribution, delays are likely to occur in the transmission of bone data using the uplink. As a result, it is difficult to generate an avatar animation that reflects the actions of the viewing user in real time based on bone data sent from the viewing user, and to include the generated animation in the video being distributed.

The bone data of each user is sent to the other user's device not only when viewing users participate in the video being distributed as avatars, but also in systems where users communicate through avatars that move according to their own movements. There is a need. Therefore, in a system where users communicate through avatars that move according to their movements, it is necessary to generate animations of avatars that move smoothly and reflect the movements of each user when a delay occurs in the transmission path. is difficult.

It is an objective of the present disclosure to provide technical improvements that solve or alleviate at least some of the problems of the prior art described above.

One of the more specific objectives of the present invention is to make it possible to generate an avatar animation related to the user's posture based on information with a smaller amount of data than before.

Other objects of the disclosure herein will become apparent upon reference to the specification as a whole. The invention disclosed in this specification may solve the problems understood from the description of the mode for carrying out the invention in this specification, instead of or in addition to the above problems. good.

A video playback device according to one aspect includes one or more computer processors, and the one or more computer processors play the video received from the video distribution device by executing computer-readable instructions, and the one or more computer processors play the video received from the video distribution device, and transmitting motion data including a posture feature amount representing a feature of the posture to the video distribution device; receiving a video including an animation of the user's avatar generated based on the posture feature amount from the video distribution device; A classifier that classifies the user's posture based on posture features determines whether the user's posture belongs to a predetermined reference posture, and there is a delay in transmitting the motion data to the video distribution device. While this is occurring, the reference posture to which the user's posture is determined to belong is identified, and reference posture identification data having a data amount smaller than the posture feature amount is transmitted to the video distribution device.

In the video playback device according to one aspect, the classifier determines whether the user's posture matches the reference posture based on an evaluation function using the posture feature amount as a variable.

In one aspect, the video playback device presents the user with an image representing the reference posture, and learns, as training data, a posture feature amount representing the posture taken by the user with respect to the presented image. Create a vessel.

In one aspect of the video playback device, one or more registered animations are registered for the user's avatar. In one aspect, the video playback device receives a video including the registered animation corresponding to the reference posture specified based on the reference posture identification data while the delay is occurring.

In one aspect, the video playback device generates a user animation of the avatar based on the posture feature acquired in time series, registers the user animation as the registered animation, and generates a sample video including the user animation. Then, the reference posture corresponding to the user animation is determined based on an image of a reference frame selected from a plurality of frames constituting the sample video.

In one aspect, the video playback device calculates the posture feature amount regarding the user at each of a plurality of feature points regarding the user based on a predetermined frame rate, and calculates the posture feature amount at each of the plurality of feature points in a first frame. A first RMS that is the root mean square of the feature amount is calculated, and a second RMS that is the root mean square of the posture feature amount at each of the plurality of feature points in a second frame chronologically later than the first frame. A third RMS, which is the root mean square of the posture feature at each of the plurality of feature points, is calculated in a third frame chronologically later than the second frame, and the second RMS and the third RMS are calculated. When the sign of the first RMS difference, which is the difference from the 1RMS, and the sign of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the user The reference posture corresponding to the animation is determined.

In one aspect, the video playback device includes first evaluation data including the posture feature acquired in time series based on the user's movement with respect to the user animation, and based on other movements of the user with respect to the user animation. The reference posture corresponding to the user animation is determined by comparing the second evaluation data including the posture feature amount acquired in time series.

In one aspect, the posture feature amount includes bone data that represents the position and orientation of the user's bones as a three-dimensional vector.

In one aspect, the motion data includes facial feature amounts representing facial features of the user. In one aspect, the video playback device determines whether the user's facial expression belongs to a predetermined reference facial expression using another classifier that classifies the user's facial expression based on the facial feature amount, and the video playback device determines whether the user's facial expression belongs to a predetermined reference facial expression, and While the expression is being generated, the reference expression to which it is determined that the user's expression belongs is identified, and reference expression identification data having a data amount smaller than the facial feature amount is transmitted to the video distribution device.

In one aspect, the reference posture identification data is transmitted in place of the motion data while the delay occurs.

In one aspect, the motion data is transmitted in real time.

A video distribution system according to one aspect includes one or more computer processors and distributes a video including a user's avatar to the user's video playback device. The one or more computer processors execute computer-readable instructions to receive motion data including posture features representing the user's posture from the video playback device via the transmission path, and to determine the posture features. An animation of the user's avatar generated based on the amount is included in the video and distributed, and a classifier that classifies the user's posture based on the posture feature determines that the user's posture belongs to a reference posture. In this case, while there is a delay in transmitting the motion data to the video distribution device, the reference posture is identified and the reference posture identification data having a data amount smaller than the posture feature amount is transmitted to the video playback device. A registered animation of the user's avatar generated based on the reference posture identification data is included in the video and distributed.

A video playback method according to one aspect is performed by one or more computer processors executing computer-readable instructions. The method includes a step of playing a video received from a video distribution device, a step of transmitting motion data including a posture feature representing a user's posture to the video distribution device, and a step of transmitting motion data including a posture feature representing a user's posture from the video distribution device to the posture feature. the user's posture belongs to a predetermined reference posture by a classifier that classifies the user's posture based on the posture feature amount; and identifying the reference posture to which it is determined that the posture of the user belongs and determining the posture feature amount while there is a delay in transmitting the motion data to the video distribution device. and transmitting reference posture identification data having a smaller amount of data than the video distribution device.

A video playback device according to one aspect of the present invention calculates a posture feature amount related to the user at each of a plurality of feature points related to the user of the video playback device based on a predetermined frame rate, and A first RMS, which is the root mean square of the posture feature at each point, is calculated, and the square of the posture feature at each of the plurality of feature points is calculated in a second frame chronologically later than the first frame. A second RMS that is the mean square root is calculated, an RMS difference that is the difference between the second RMS and the first RMS is calculated, and when the RMS difference is larger than a predetermined threshold, the second frame is sent to the video distribution device. transmitting the posture feature amount, receiving a video including an animation of the user's avatar generated based on the posture feature amount from the video distribution device, and playing the video.

A video playback device according to one aspect of the present invention calculates a posture feature amount related to the user at each of a plurality of feature points related to the user of the video playback device based on a predetermined frame rate, and A first RMS, which is the root mean square of the posture feature at each point, is calculated, and the square of the posture feature at each of the plurality of feature points is calculated in a second frame chronologically later than the first frame. calculating a second RMS that is the root mean square, and calculating a third RMS that is the root mean square of the posture feature amount at each of the plurality of feature points in a third frame chronologically later than the second frame; When the sign of the first RMS difference, which is the difference between the second RMS and the first RMS, and the sign of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the posture characteristic in the third frame and receives a video including an animation of the user's avatar generated based on the posture feature from the video distribution device, and plays the video.

According to the embodiment, it is possible to generate an avatar animation related to the user's posture based on information with a smaller amount of data than before.

FIG. 1 is a block diagram illustrating a video distribution system according to an embodiment. FIG. 2 is a diagram conceptually showing a three-dimensional skeletal model. 2 is a diagram illustrating reference posture management data stored in the video distribution system of FIG. 1. FIG. FIG. 2 is a diagram conceptually showing a three-dimensional skeletal model corresponding to a starting posture. FIG. 3 is a diagram conceptually showing a three-dimensional skeletal model corresponding to a trigger posture. 2 is a diagram illustrating avatar data stored in the video distribution system of FIG. 1. FIG. 2 is a diagram illustrating animation management data stored in the video distribution system of FIG. 1. FIG. FIG. 3 is a diagram schematically showing frames forming a captured image. It is a table showing an example of the feature amount and its RMS at a plurality of feature points in each frame. FIG. 2 is a diagram illustrating an example of a video displayed on the video playback device 10 in an embodiment. FIG. 2 is a diagram illustrating an example of a video displayed on the video playback device 10 in an embodiment. FIG. 2 is a flowchart showing a part of the process in a video playback method according to an embodiment. FIG. 2 is a block diagram showing a video distribution system according to another embodiment. 12 is a diagram illustrating reference facial expression management data stored in the video distribution system shown in FIG. 11. FIG. 12 is a diagram illustrating registered facial expression management data stored in the video distribution system shown in FIG. 11. FIG. FIG. 2 is a block diagram showing a video distribution system according to another embodiment. FIG. 2 is a block diagram showing a video distribution system according to another embodiment. 3 is a table showing an example of feature amounts at a plurality of feature points in each frame.

Hereinafter, various embodiments of the present invention will be described with appropriate reference to the drawings. Identical or similar components in multiple drawings are provided with the same reference numerals.

A video distribution system according to an embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing a video distribution system 1 according to an embodiment. FIGS. 2, 4a, and 4b are diagrams conceptually showing a three-dimensional skeleton model, and FIGS. FIG. 6 is a diagram for explaining information stored in the video distribution system 1.

The video distribution system 1 includes a video playback device 10 and a video distribution device 20. The video playback device 10 and the video distribution device 20 are connected to each other via a network 50 so that they can communicate with each other. The video distribution system 1 may include a storage 60. A viewing user who is a user of the video playback device 10 can view the video distributed from the video distribution device 20 using the video playback device 10 . The video distribution system 1 may include two or more video playback devices. A viewing user who views a video distributed from the video distribution device 20 using the video playback device 10 can have his or her avatar displayed on the video. In other words, viewing users can participate in the video being distributed via their own avatars. Viewing users can have their own avatars displayed on the video being distributed, and can interact with the video distribution users (or characters in the video being distributed) and other viewing users via the avatars.

First, the video playback device 10 will be explained. The video playback device 10 is an information processing device such as a smartphone. In addition to a smartphone, the video playback device 10 may be a mobile phone, a tablet terminal, a personal computer, an electronic book reader, a wearable computer, a game console, or any other information processing device capable of playing back videos.

The video playback device 10 is used by viewing users for video viewing and other purposes. The video playback device 10 includes a computer processor 11, a communication I/F 12, a storage 13 that stores various information, a display 14 that displays the video to be played, and a sensor unit 15. The video distribution device 10 may include components other than those described above, such as a sound collecting microphone. The video playback device 10 plays back the video distributed from the video distribution device 20.

The computer processor 11 is an arithmetic device that loads an operating system and various programs that implement various functions into memory from the storage 13 or other storage, and executes instructions included in the loaded programs. The computer processor 11 is, for example, a CPU, MPU, DSP, GPU, various other arithmetic devices, or a combination thereof. The computer processor 11 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. Although computer processor 11 is illustrated as a single component in FIG. 1, computer processor 11 may be a collection of multiple physically separate computer processors. In this specification, a program described as being executed by the computer processor 11 or instructions included in the program may be executed by a single computer processor, or may be executed in a distributed manner by a plurality of computer processors. Good too. Further, the program executed by the computer processor 11 or the instructions included in the program may be executed by a plurality of virtual computer processors. The functions realized by the computer processor 11 will be described later.

The communication I/F 12 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof. The video playback device 10 can send and receive data to and from other devices via the communication I/F 12.

Storage 13 is a storage device accessed by computer processor 11 . The storage 13 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various other storage devices capable of storing data. Various programs can be stored in the storage 13. At least some of the programs and various data that can be stored in the storage 13 may be stored in a storage that is physically separate from the video playback device 10 (for example, the storage 60).

Display 14 includes a display panel and a touch panel. The display panel is a liquid crystal panel, an organic EL panel, an inorganic EL panel, or any other display panel capable of displaying images. The touch panel is configured to be able to detect a touch operation (contact operation) by a player. The touch panel can detect various touch operations such as tap, double tap, and drag by the player. The touch panel may include a capacitive proximity sensor and be configured to be able to detect a non-contact operation by the player. The video distributed from the video distribution device 20 is displayed on the display 14.

The sensor unit 15 includes one or more sensing devices that detect the posture of the user of the video playback device 10. This sensing device may include an RGB camera, a depth sensor, and other devices capable of detecting the posture of the user. Sensor unit 15 may include a computer processor. The computer processor of the sensor unit 15 may generate three-dimensional skeletal data representing the user's posture by analyzing the data acquired by the sensing device. A computer processor included in the sensor unit 15 may generate three-dimensional skeletal data by executing dedicated software. The three-dimensional skeleton data representing the user's posture detected by the sensor unit 15 is an example of the "posture feature amount" described in the claims. The sensor unit 15 may generate three-dimensional skeleton data using Light Coding technology such as Kinect (trademark) provided by Microsoft Corporation. In other words, the sensor unit 15 irradiates a random pattern of infrared rays onto a target such as a user, acquires the depth of the target by analyzing the image, and generates three-dimensional skeletal data by analyzing the depth. good. The feature quantity representing the user's posture detected using the Light Coding technique is not limited to three-dimensional skeleton data, and other feature quantities representing the user's posture may be used.

Three-dimensional skeletal data is data for expressing a human body or a part of the human body as a three-dimensional skeletal model. A three-dimensional skeletal model models the skeleton of a human body or a part of the human body using a plurality of bones corresponding to the axes of bones of the human body and a plurality of joints connecting each bone. The three-dimensional skeleton model will be further explained with reference to FIG. FIG. 2 is an explanatory diagram for explaining the concept of a three-dimensional skeleton model. In the example shown in FIG. 2, the three-dimensional skeleton model 100 includes eight bones and joints connecting these bones. A bone in a three-dimensional skeleton model is represented as a line segment connecting two points on three-dimensional coordinates. Therefore, each bone is represented by three-dimensional vectors V1 to V8 representing effective line segments corresponding to each bone in a three-dimensional space (XYZ space). The three-dimensional skeleton model that constitutes the three-dimensional skeleton model includes three-dimensional vectors representing each of the bones included in the three-dimensional model. Although a three-dimensional skeletal model of the upper body of a human body is shown in FIG. For example, it may be a model of an arm, lower body, etc.). Furthermore, in the example of FIG. 2, the upper half of the human body is represented by eight bones, but the upper half of the human body may be represented by more or less than eight bones. A three-dimensional skeletal model that includes a larger number of bones may include bones that represent finger bones. With this, finger movements can also be detected.

Information stored in the storage 13 will be explained. In the illustrated embodiment, the storage 13 stores the reference posture management data 13a and other various information necessary for using the services provided by the video distribution device 20. In the video distribution system 1 according to one embodiment, a user's reference posture is determined in advance. In the video distribution system 1, only one reference posture may be determined, or a plurality of reference postures may be determined. Each of the one or more reference postures may be a set of a plurality of postures for identifying a series of actions of the user. For example, the reference posture includes a starting posture that indicates the starting posture of a series of user movements associated with the reference posture, and a trigger posture that indicates a specific posture after the user takes the starting posture in the series of motions. may be included. At least one of the starting posture and the trigger posture may be plural.

An example of the reference posture management data 13a is shown in FIG. In the example shown in FIG. 3, the reference posture includes a starting posture and a trigger posture. Therefore, in the storage 13, start attitude data indicating a starting attitude included in the reference attitude and trigger attitude indicating a trigger attitude included in the reference attitude are associated with reference attitude identification data for identifying the reference attitude. data may be stored.

The reference posture identification data is, for example, a reference posture ID that identifies a reference posture. The reference posture ID is, for example, an identification code expressed by several bits. The number of bits of the reference posture ID can be determined according to the number of reference postures used in the video distribution system 1. The reference posture ID is represented by, for example, data with an information amount of 10 bits or less. The reference posture ID may be represented by data with an information amount of 5 bits or less, 4 bits or less, 3 bits or less, or 2 bits or less. Therefore, the standard posture identification data (standard posture ID) has a much smaller amount of data than the three-dimensional skeleton data.

The starting attitude data is data indicating the starting attitude. The starting posture data is, for example, three-dimensional skeleton data indicating the starting posture. An example starting position is shown in Figure 4a. The starting posture in the illustrated example is a posture in which the right palm is thrust forward so that the right fist is at the same height as the right shoulder. The starting posture data may be three-dimensional skeleton data modeling this starting posture. The starting posture data may include three-dimensional vector data indicating the position and orientation of each bone in the starting posture. The three-dimensional skeleton data indicating the starting pose may include eight bones as schematically shown in Figure 4a. The vectors T1 to T8 representing these eight bones correspond to the bone vectors V1 to V8 included in the three-dimensional skeleton data of the viewing user generated based on the detection data of the sensor unit 15, respectively. When data other than three-dimensional skeleton data is used as the posture feature, the starting posture data is data representing the starting posture among the posture features used.

Trigger attitude data is data indicating a trigger attitude. The trigger posture data is, for example, three-dimensional skeleton data indicating the trigger posture. An example of a trigger position is shown in Figure 4b. The trigger posture in the illustrated example is a posture in which the right arm is thrust forward diagonally upward. The trigger posture data may be three-dimensional skeleton data modeling this trigger posture. The three-dimensional skeleton data indicating the trigger posture may include three-dimensional vector data indicating the position and orientation of each bone. The three-dimensional skeleton data indicating the trigger posture may include eight bones as schematically shown in FIG. 4b. These eight bone vectors T1 to T8 correspond to bone vectors V1 to V8 included in the viewing user's three-dimensional skeleton data generated based on the detection data of the sensor unit 15, respectively. When data other than three-dimensional skeleton data is used as the posture feature, the trigger posture data is data representing the trigger posture among the posture features used.

Next, the video distribution device 20 will be explained. The video distribution device 20 is, for example, a video distribution server that distributes a video to the video playback device 10 via a downlink of the network 50. The video distribution device 20 includes a computer processor 21, a communication I/F 22, and a storage 23 that stores various information. The video distribution device 20 may include components other than those described above, such as a sound collection microphone.

The computer processor 21 is an arithmetic device that loads an operating system and various programs that implement various functions into memory from the storage 23 or other storage, and executes instructions included in the loaded programs. The description given above regarding computer processor 11 also applies to computer processor 21 to the extent possible. The functions realized by the computer processor 21 will be described later.

The communication I/F 22 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof. The video distribution device 20 can transmit and receive data to and from other devices via the communication I/F 22.

Storage 23 is a storage device accessed by computer processor 21 . The storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various other storage devices capable of storing data. Various programs can be stored in the storage 23. At least some of the programs and various data that can be stored in the storage 23 may be stored in a storage that is physically separate from the video distribution device 20 (for example, the storage 60).

The storage 23 can store object data 23a, avatar data 23b, animation management data 23c, and various other information necessary for generating and distributing moving images other than the above.

The object data 23a may include asset data for constructing a virtual space that constitutes a moving image. The object data 23a includes data for drawing the background of the virtual space constituting the video, data for drawing various objects displayed in the video, and data for drawing various objects displayed in the video other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space. In addition to the above, the object data 23a may include data indicating a gift object to be displayed in the video based on a display request from the viewing user of the video playback device 10. Gift objects may include effect objects, normal objects, and decoration objects. Viewing users can purchase desired gift objects. Details of the gift object displayed in the video are described in the specification of Japanese Patent No. 6446154. Also in the video distribution system 1 of the present application, a gift object can be displayed in a video similarly to the description in the specification of Patent No. 6446154.

An example of the avatar data 23b is shown in FIG. As shown in FIG. 5, the avatar data 23b can include avatar identification information of the avatar used by the viewing user in the video distribution system 1, and avatar display information for displaying the avatar in the video. . In other words, the storage 23 may store avatar display information for displaying the avatar in association with the avatar identification information of the avatar. The avatar identification information is, for example, an avatar ID that identifies an avatar. A user of the video playback device 10 can set his or her own avatar in the video distribution system 1. In order to manage avatars for each user, the avatar ID may be stored in the storage 23 in association with a user ID that identifies the user. For example, an avatar is displayed in a video as an image imitating a human or an animal. Avatar display information is information used to display an avatar in a video. Avatar information includes, for example, parts information showing images of the head, hairstyle, facial parts (eyes, nose, mouth, etc.), torso, clothes, accessories, items, and other parts that make up the avatar. Contains skin information for specifying the appearance of other avatars. Users can register their own avatars by selecting their favorite part images. The avatar display information may include 2D display information for displaying the avatar in 2D in the video, and 3D display information for displaying the avatar in 3D in the video. The 3D display information includes parts information that shows images of parts to display the avatar three-dimensionally in a video, bone data to express the avatar's movement in three dimensions, and other parts that display the avatar three-dimensionally. Contains publicly known information used for display.

As described above, the video distributed from the video distribution device 20 may include an animation of the avatar of the user of the video playback device 10. An animation of a certain user's avatar may be generated to reflect the posture and movement of the user in real time based on posture feature amounts indicating the posture of the user, as described later.

A registered animation registered in advance may be used as the avatar animation. Registered animations are not generated in real time during video distribution to follow the movements of the viewing user, but are registered or defined in advance before the video is distributed or before the avatar animation needs to be displayed. It's an animation. An example of animation management data 23c for managing registered animations is shown in FIG. As illustrated, the animation management data 23c includes reference posture identification data, registered animation identification data (registered animation ID) for identifying registered animations, and animation definition data for specifying avatar animations. In the storage 23, registered animation identification data and animation definition data are stored in association with reference posture identification data. The reference posture identification data may be the reference posture ID as described above. When a plurality of reference posture IDs are used in the video distribution system 1, a plurality of registered animations may be registered according to the number of reference posture IDs. One registered animation selected from the plurality of registered animations can be included in the video. The animation definition data may be data that describes bone data indicating the positions and orientations of the bones of the avatar in chronological order.

The registered animation may be generated based on a request from a viewing user. Hereinafter, for convenience of explanation, a registered animation registered based on a request from a viewing user will be referred to as a "user animation." When a viewing user requests registration of a user animation, the video playback device 10 can prompt the viewing user to perform a movement corresponding to the animation desired to be registered. The video playback device 10 acquires posture feature amounts (for example, three-dimensional skeleton data) of the posture taken by the viewing user over a predetermined period of time in response to the display of the image representing the reference posture. Acquisition of this posture feature amount is performed by the posture data acquisition section 11b or the sensor unit 15 at predetermined sampling time intervals. In this way, animation definition data indicating the movement corresponding to the animation that the viewing user desires to register is obtained. This animation definition data may have the same data format as the animation definition data of the registered animation stored as the animation management data 23c. The video playback device 10 transmits the animation definition data that defines the user animation thus obtained to the video distribution device 20, and registers this user animation in the video distribution device 20 as a new registered animation.

When the video distribution device 20 receives animation definition data that defines a user animation, it stores the user animation in the storage 23 as part of the animation management data 23c. Specifically, the video distribution device 20 issues a registered animation ID that identifies the user animation for which registration is requested, and stores the animation definition data received from the video playback device 10 in association with the registered animation ID. . Furthermore, the video distribution device 20 requests the video playback device 10 to determine a reference posture that is a trigger for including the user animation whose registration is requested in the video.

In response to a request from the video distribution device 20, the video playback device 10 determines a reference posture that becomes a trigger for causing its own avatar to perform a movement according to the newly registered user animation. In this specification, a reference posture that serves as a trigger for moving an avatar according to a newly registered user animation may be referred to as an additional reference posture. Additional reference postures can be determined in various ways. For example, a sample video containing a newly registered user animation is generated, and the viewing user who has requested the registration of the user animation is asked to select one or more candidate frames from among the multiple frames included in this sample video. The additional reference posture can be determined based on the image of the avatar included in the selected frame. The viewing user can select a favorite frame from a plurality of frames. The viewing user may, for example, select a frame that includes an avatar in a memorable posture, a frame that includes an avatar in a characteristic posture, or a frame other than these. can. When two frames are selected from the plurality of frames, the posture of the avatar included in the chronologically earlier frame of the two frames is determined as the starting posture, and the avatar posture included in the chronologically earlier frame is determined as the starting posture, The posture of the avatar included in a certain frame may be determined as the trigger posture.

Another method of determining the additional reference posture will be explained. The video playback device 10 instructs the viewing user who has made the user animation registration request to perform the movement corresponding to the user animation multiple times. This instruction may be given by voice or screen display. The video playback device 10 acquires posture feature amounts (for example, three-dimensional skeleton data) corresponding to the movements performed by the viewing user in response to this instruction. Specifically, three-dimensional skeletal data representing the posture of the viewing user moving in accordance with the user animation is acquired at a predetermined sampling interval during a predetermined measurement period. As a result, two sets of three-dimensional skeleton data are obtained that represent, in time series, the postures of the viewing user who performed movements corresponding to the user animation from the start of the measurement to the end of the measurement. Next, among these two sets of three-dimensional skeleton data, three-dimensional skeleton data acquired at the same timing after the start of measurement are compared, and an additional reference posture is determined based on the comparison result. For example, the total value of the angles made by the vectors of the corresponding bones of the 3D skeleton data acquired at the same timing after the start of measurement among the two sets of 3D skeleton data is calculated, and the total value of this angle is the smallest. The posture corresponding to the three-dimensional skeleton data (whichever of the two sets of skeleton data may be adopted) can be used as the additional reference posture.

Next, the functions of the video playback device 10 will be explained. The functions of the video playback device 10 are realized by the computer processor 11 executing computer-readable instructions included in a program. The computer processor 11 functions as a video playback section 11a, an attitude data acquisition section 11b, a transmission section 11c, a classification section 11d, and a delay monitoring section 11e by executing computer-readable instructions included in the program. At least some of the functions implemented by computer processor 11 may be implemented by a computer processor other than computer processor 11. At least some of the functions realized by the computer processor 11 may be realized by the computer processor 21 of the video distribution device 20 or other computer processors included in the video distribution system 1.

The video playback unit 11a plays back the video distributed from the video distribution device 20. The reproduced moving image is displayed on the display 14.

The posture data acquisition unit 11b acquires posture feature amounts representing the posture characteristics of a user (also referred to as a "viewing user") who views a video using the video playback device 10. The posture feature quantity representing the posture characteristics of the viewing user may be three-dimensional skeletal data representing the posture of the viewing user, that is, three-dimensional vector data indicating the position and orientation of the viewing user's bones. The posture data acquisition unit 11b generates three-dimensional skeletal data of the viewing user based on the detection data detected by the sensor unit 15, for example. The viewing user's three-dimensional skeleton data may be generated at predetermined sampling time intervals. If the sensor unit 15 includes a processor and the three-dimensional skeleton data is generated by the processor of the sensor unit 15, the posture data acquisition section 11b does not need to be executed as a function of the processor 11.

The posture feature amount acquired by the posture data acquisition unit 11b may be a feature amount other than three-dimensional skeleton data. For example, as described above, a random pattern of infrared rays may be irradiated onto a target such as a user using the Light Coding technology, and the depth of the target may be obtained by analyzing the image, and the depth may be used as the posture feature amount. A detection value detected by the sensor unit 15 and various values calculated based on this detection value can be used as the posture feature amount.

The posture data acquisition unit 11b may obtain the posture feature amount from a captured image including a plurality of frames obtained by capturing the movement of the user of the video playback device 10 at a predetermined frame rate in the following manner. Specifically, the posture data acquisition unit 11b extracts a plurality of feature points related to the user in each frame of the captured image. Positions suitable for expressing the user's posture and movements are extracted as feature points. A plurality of feature points may also be referred to as a feature point group. FIG. 7 schematically shows a frame f1, which is one of a plurality of frames constituting a captured image captured by the video playback device 10. As illustrated, the frame f1 includes a captured image U1 of the user. P1 to P6 shown in FIG. 7 indicate extracted feature points, respectively. The number and positions of feature points are not limited to those shown in FIG. 7. For example, when a random dot pattern is projected onto the body or face of the user of the video playback device 10 using an infrared laser, and the user on whom this random dot pattern is projected is imaged with a camera capable of photographing the infrared region, the random dot pattern The whole becomes a feature point group.

The posture data acquisition unit 11b can obtain a posture feature amount (image vector) for each of the plurality of extracted feature points. The posture feature amount at each of the feature points P1 to P6 may be the depth of each feature point or three-dimensional coordinates. When three-dimensional coordinates are used as posture features, a three-dimensional world coordinate system is set in the captured image, and a relative amount normalized based on the captured image and the world coordinate system is used as the posture feature. Ru. The normalized posture feature amount is expressed as a relative amount from 0 to 1.0, for example. The posture feature amount of each feature point P1 to P6 in frame f1 is the posture feature amount of each feature point P1 to P6 obtained in a frame chronologically earlier than frame f1 (for example, immediately before) and the posture feature amount obtained in frame f1. The amount of change may be expressed as a difference from the posture feature amount of each of the feature points P1 to P6. When using the amount of change in posture feature between frames as another posture feature, and it is necessary to distinguish between the two, the posture feature in a certain frame is called an "intra-frame feature," and the posture feature between frames is called an "intra-frame feature." The posture feature amount expressed by the amount of change in the feature amount may be referred to as the "inter-frame feature amount." Unless otherwise specified or unless otherwise understood from the context, the term "posture feature" includes both "intra-frame feature" and "inter-frame feature." The posture feature amount of each feature point P1 to P6 may be normalized to a range of 0 to 1.0 and expressed as a float array. In this case, the posture feature amount at each of the feature points P1 to P6 becomes an element of the array.

The transmitter 11c transmits the posture feature amount representing the posture of the viewing user acquired by the posture data acquisition section 11b or the sensor unit 15 to the video distribution device 20. When the transmitter 11c receives the posture feature amount from the posture data acquisition section 11b or the sensor unit 15, it immediately transmits it to the video distribution device 20. In other words, the transmitter 11c can transmit the posture feature amount of the viewing user to the video distribution device 20 in real time. As described above, the posture feature amount is acquired at a predetermined sampling time interval or a predetermined frame rate. Thereby, the posture feature amount can be generated at each predetermined sampling time interval or at each frame rate. Therefore, the posture feature amount of the viewing user that is continuously acquired over a predetermined time interval can represent the movement of the viewing user's body in time series as digital data. The posture feature amount representing the posture of the viewing user may be transmitted to the video distribution device 20 together with data other than the posture feature amount. In this specification, data representing the posture and facial expression of the viewing user or data correlated with the posture and facial expression of the viewing user may be collectively referred to as "motion data" of the viewing user. The viewing user's motion data is transmitted from the video playback device 10 to the video distribution device 20 via a transmission path including the network 50. Motion data may be sent in packets. That is, the transmitter 11c may transmit a packet including motion data to the video distribution device 20.

In one embodiment, the classification unit 11d uses a classifier that classifies the posture of the viewing user to determine the posture of the viewing user according to a predetermined standard based on the posture feature amount (for example, three-dimensional skeleton data) of the viewing user. Determine whether it belongs to the posture. This classifier is, for example, a linear classifier. When the reference posture includes a starting posture and a trigger posture, the classification unit 11d can determine whether the posture of the viewing user belongs to the reference posture and the trigger posture. After determining whether the viewing user's posture belongs to the starting posture of the reference postures at time t1, the classification unit 11d determines that the user's posture at the time t1 belongs to the starting posture, Subsequently, it may be determined whether the posture of the viewing user at time t2, which is after time t1, belongs to the trigger posture.

In one embodiment, in order to obtain training data, the classification unit 11d displays an image representing a reference posture on the display 14, and instructs the viewing user to take a posture according to the image representing the reference posture by using a voice or a screen display. can be instructed through. The video playback device 10 sequentially acquires posture feature amounts of movements performed by the viewing user in response to this instruction using the posture data acquisition section 11b or the sensor unit 15. The classification unit 11d can create a classifier by learning, as training data, the posture feature amount of the posture taken by the viewing user in response to the image representing the reference posture.

For example, the classification unit 11d in one embodiment determines whether the posture of the viewing user indicated by the posture feature from the posture data acquisition unit 11b or the sensor unit 15 matches a predetermined reference posture. can be determined based on the evaluation function using as a variable. If it is determined that the orientation of the viewing user matches the reference orientation based on the evaluation function, the orientation of the viewing user belongs to the reference orientation. Whether the viewing user's posture matches the starting posture is determined by the posture feature amount representing the viewing user's posture generated in the posture data acquisition unit 11b or the sensor unit 15 and the posture feature amount representing the starting posture (starting posture data). It can be determined based on. When the posture feature amount is three-dimensional skeleton data, vectors V1 to V8 of each bone included in the three-dimensional skeleton data of the viewing user and vectors T1 to T1 of each bone corresponding to vectors V1 to V8 of the starting posture data. The smaller the total value of the angle with T8, the higher the similarity between the viewing user's posture detected by the sensor unit 15 and the starting posture. In other words, there is a negative correlation between the magnitude of the angle θ formed by the 3D vector included in the 3D skeleton data representing the viewing user's posture and the corresponding vector of the 3D skeleton model representing the reference posture, and the similarity of the postures. There is. Therefore, focusing on this point, a new evaluation function f is defined for normalizing the magnitude of the angle θ defined by the two three-dimensional skeleton models. An example of the evaluation function f is shown in the following formula (1).

In the above formula, (Ti・Vi) indicates the inner product of the corresponding vectors, (∥Ti∥∥Vi∥) indicates the product of the magnitudes of each vector, and k is the vector constituting the three-dimensional skeletal model. Indicates the number of elements in That is, according to the evaluation function f shown in the above equation (1), the average value of the cosine (=cosθ) of the angle θ formed by the corresponding vectors of the two three-dimensional skeleton models is obtained. Here, if the corresponding vectors of the two 3D skeletal models completely match (angle θ = 0), cos θ = 1, and if the directions of the corresponding vectors of the two 3D skeletal models are opposite (angle θ=180°), cos θ=−1, so the possible range of the evaluation function f is −1≦f≦+1. In this case, for example, by linearly assigning the value range (-1≦f≦+1) of the evaluation function f to a percentage value (0 to 100%), the degree of similarity can be expressed as a percentage. This predetermined threshold may be, for example, 90%. The determination of the similarity between two three-dimensional skeleton data is also disclosed in Japanese Patent Application Laid-Open No. 2013-37454.

The classification unit 11d determines that the posture of the viewing user is the starting posture when the degree of similarity between the posture feature amount representing the posture of the viewing user calculated as described above and the posture feature amount representing the starting posture is greater than or equal to a predetermined threshold. It may be determined that they match.

Each time the posture data acquisition section 11b or the sensor unit 15 acquires the posture feature amount of the viewing user, the classification section 11d determines whether or not the posture of the viewing user indicated by the posture feature amount belongs to the starting posture. good. In this case, it is determined whether the posture of the viewing user matches the starting posture at the same frequency as the sampling rate for acquiring the posture feature amount of the viewing user.

It can be similarly determined whether the viewing user's posture and the trigger posture match. That is, the classification unit 11d calculates the degree of similarity between the posture feature amount representing the posture of the viewing user at time t2 and the posture feature amount representing the trigger posture, and when this degree of similarity is greater than or equal to a predetermined threshold, It may be determined that the posture matches the trigger posture. The above explanation for determining whether the viewing user's posture matches the starting posture also applies to the process for determining whether the viewing user's posture matches the trigger posture as much as possible.

The delay monitoring unit 11e monitors the transmission delay of motion data transmitted from the video playback device 10 to the video distribution device 20 by the transmission unit 11c. For example, the delay monitoring unit 11e adds a timestamp to an actual packet including motion data before transmission, and the timestamp added at the time of transmission and the timestamp added when this actual packet is received by the video distribution device 20. The transmission time of the actual packet on the transmission path between the video playback device 10 and the video distribution device 20 can be determined using the time stamp. The delay monitoring unit 11e can determine that a delay has occurred in the transmission path when this transmission time is equal to or longer than a predetermined reference time or when it is longer than the reference time. After once determining that a delay has occurred in the transmission path, the delay monitoring unit 11e determines that the delay has been resolved when the transmission time becomes shorter than a predetermined reference time or becomes equal to or less than the predetermined reference time. It can be determined that When the video distribution device 20 is a server and the video playback device 10 is a client, data is transmitted from the video playback device 10 to the video distribution device 20 using an uplink. In this case, the delay monitoring unit 11e monitors whether a delay has occurred in the uplink of the transmission path between the video playback device 10 and the video distribution device 20. The delay monitoring unit 11e may measure the transmission time of an actual packet as described above, or may measure the transmission time using a pseudo packet that does not include motion data. While a delay occurs in the transmission of motion data transmitted from the video playback device 10 of a certain viewing user, a delay occurs in the transmission of motion data transmitted from the video playback device 10 of another viewing user. It is possible that it will be determined that there is no such thing. Therefore, the delay in the transmission path may be determined for each viewing user. Further, whether or not a transmission delay has occurred may be determined by the video distribution device 20 based on a time stamp included in the packet. The video playback device 10 may receive the determination result of the transmission delay in the video distribution device 20.

The function of the transmitter 11c will be explained again. The transmitting unit 11c can transmit the reference attitude identification data (reference attitude ID) stored in the storage 13 when the delay monitoring unit 11e determines that a delay has occurred in the transmission path. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As mentioned above, the reference attitude identification data may include starting attitude data and trigger attitude data. The transmitter 11c can transmit trigger attitude data as reference attitude identification data. In one embodiment, when the classification unit 11d determines that the orientation of the viewing user belongs to the trigger orientation, the transmission unit 11c selects the trigger orientation as reference orientation identification data while a delay occurs in the transmission of motion data. The reference posture ID to be identified is transmitted to the video distribution device 20. If the viewing user's posture is determined to belong to the trigger posture within a predetermined interval after the classification section 11d determines that the viewing user's posture belongs to the starting posture, the transmitting section 11c transmits motion While a delay occurs in data transmission, a reference posture ID for identifying the trigger posture is transmitted to the video distribution device 20. By transmitting a reference posture ID when it is determined that the posture of the viewing user after the posture of the viewing user belongs to the starting posture is also determined to belong to the trigger posture, the posture of the viewing user is determined to belong to only one of the starting posture or the trigger posture. Compared to the case where the reference posture ID is transmitted in response to the determination, it is possible to prevent or suppress the viewing user from unintentionally transmitting the reference posture ID. "While a delay is occurring in the transmission of motion data" means the period from when the delay monitoring unit 11e determines that a delay has occurred in the transmission path until it is determined that the delay has been resolved. It's okay. The transmitter 11c does not need to transmit the reference posture ID for identifying the starting posture to the video distribution device 20.

Next, the functions realized by the computer processor 21 will be explained in more detail. The computer processor 21 functions as a video generation section 21a, a video distribution section 21b, and an animation generation section 21c by executing computer-readable instructions included in the distribution program. At least some of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the video distribution system 1. At least a part of the functions realized by the computer processor 21 may be realized by, for example, the computer processor 11 of the video playback device 10 or a computer processor provided in the video distribution system 1 other than that.

The video distribution device 20 can distribute various types of videos. In the following, it is assumed that the video distribution device 20 distributes a video including animation of a character object generated based on the movement of an actor.

The video generation unit 21a detects the movement of the actor using a motion sensor attached to the actor. The video generation unit 21a can generate an animation of a character that moves in synchronization with the movement of the actor's body detected by the motion sensor. The video generation unit 21a may acquire face motion data that is a digital representation of the actor's facial movements. In this case, the video generation unit 21a can generate an animation of a character whose expression changes in synchronization with the movement of the actor's face. The video generation unit 21a can construct a virtual space using the object data 23a, and generate a video including this virtual space and an animation of a character corresponding to the actor. The video generation unit 21a can synthesize the generated video with the actor's voice obtained from the microphone. Generation of a moving image including animation of a character that moves in synchronization with the movements of the actor's body and facial expressions is disclosed in detail in the specification of Japanese Patent No. 644615.

When the video generation unit 21a receives a participation request for having an avatar participate in the video from a viewing user who is viewing the video, the video generation unit 21a can generate a video so as to include the avatar of the viewing user. The participation request from the viewing user may include a user ID that identifies the viewing user. The video generation unit 21a specifies an avatar ID that identifies the avatar of the viewing user based on the user ID included in the participation request, and based on the parts information stored in the storage 23 in association with the avatar ID. can generate an avatar object representing the avatar of the viewing user.

The video distribution unit 21b distributes the video generated by the video generation unit 21a. This video is distributed to the video playback device 10 via the network 50. The received video is played back in the video playback device 10.

A display example of a video distributed from the video distribution device 20 to the video playback device 10a and being played back in the video playback device 10 is shown in FIG. 9a. A video played on the video playback device 10 may be displayed on the display 14. As shown in the figure, the video 60 displayed on the video playback device 10 includes a character object 51 representing an actor's character, a floor object 54a on which the character object 51 stands, and a screen object 54b defining the rear end of the stage. Avatar objects 56a to 56j representing avatars of viewing users who are viewing the video 60 are included. The character object 51 can move in virtual space in synchronization with the movement of the actor's body. Since the video 60 is distributed from the video distribution device 20 to a large number of video playback devices 10, the image 60 includes a large number of avatars. In FIG. 9a, it is assumed that ten viewing users are participating in the video via avatars. The number of viewing users who can participate in the video may be more than 10, or at least less than 10.

Operation buttons for accepting user operations may be displayed on the display 14 so as to overlap the video 60. In the example shown in FIG. 9a, a gift button 61 for gifting, an evaluation button 62 for providing an evaluation, and an avatar participation button 63 for applying for participation in the video 60 by the avatar. is displayed superimposed on the video 60. A gift button 61, an evaluation button 62, and an avatar participation button 63 are displayed so as to be selectable by the viewing user. In the video 60, operation buttons other than these may be displayed. A viewing user who is viewing the video 60 can send a desired gift to the distributor who is distributing the video 60 or the actor appearing in the video 60 via the character 51 by selecting the gift button 61. I can do it. By selecting the evaluation button 62, the viewing user can transmit evaluation information indicating that the video 60 has been positively evaluated to the video distribution device 20. Evaluation information from various viewing users may be aggregated, and the aggregate results may be displayed together with the video 60. By selecting the avatar participation button 63, the viewing user can send a participation request to the video distribution device 20 requesting that his or her avatar participate in the video 60.

The animation generation unit 21c generates an animation of avatars of viewers participating in the video 60. In one embodiment, the animation generation unit 21c generates an animation of the avatar of the viewing user based on the posture feature amount of the viewing user received from the video playback device 10. The posture feature amount (for example, three-dimensional skeleton data) that is received continuously in time represents the movement of the viewing user's body in time series. Therefore, by continuously receiving the posture feature amount of the viewing user from the video playback device 10, the animation generation unit 21c generates an avatar that moves in synchronization with the body movement of the viewing user based on the posture feature amount. Animation can be generated. When the animation of the avatar is generated by the animation generation unit 21c, the video generation unit 21a generates the video 60 to include the animation of the avatar.

If a transmission delay occurs in the uplink transmission path from the video playback device 10 to the video distribution device 20, the video distribution device 20 may not be able to continuously receive posture features of the viewing user in time. be. If an attempt is made to create an avatar animation based only on the posture features of the viewing user transmitted from the video playback device 10, if a transmission delay occurs in the transmission path, the animation of the avatar that moves to reflect the viewing user's movements may not be created. There is a possibility that it will not be possible to generate the data. On the other hand, the animation generation unit 21c in one embodiment generates the animation of the avatar based on the reference posture identification data (for example, the reference posture ID) from the video playback device 10, without relying only on the posture feature amount of the viewing user. can be generated. Specifically, upon receiving the reference posture identification data from the video playback device 10, the animation generation unit 21c identifies the registered animation associated with the received reference posture identification data by referring to the animation management data 23c. The avatar animation can be generated based on the animation definition data associated with the registered animation ID of the specified registered animation. As described above, the reference posture identification data is transmitted from the video playback device 10 when the delay monitoring unit 11e of the video playback device 10 determines that there is a delay in the uplink transmission of motion data including posture features. It is transmitted to the video distribution device 20. Therefore, even if the animation generation unit 21c is unable to receive the posture feature amount while there is a delay in the transmission of motion data, the animation generation unit 21c receives the posture feature amount from the video playback device 10 while the delay is occurring. Avatar animation can be generated based on reference posture identification data.

The generation of avatar animation will be explained. The avatar animation generated based on the posture feature amount (for example, three-dimensional skeleton data) of the viewing user expresses the movement of the avatar that reflects the movement of the viewing user. For example, assume that the viewing user lifts his or her right hand diagonally upward from a position where it is placed on the waist. In this case, an animation of the avatar of the viewing user is generated based on the posture feature amount of the viewing user. By including this animation in the video, the avatar lifts its right hand (the part corresponding to the avatar's right hand) diagonally upward from the position it rests on its waist, similar to the movement of the viewing user in the video. .

On the other hand, if a delay occurs in the transmission path, generation of the avatar animation based on the posture feature amount may be interrupted. For example, if a delay occurs in the transmission path, the transmission of the posture feature by the transmitter 11c may be interrupted, and in response, the animation generator 21c generates an avatar animation based on the posture feature. may also be interrupted. As described above, the video distribution device 20 can determine whether there is a delay based on the time stamp included in the packet received from the video playback device 10. When a delay occurs in the transmission path, it becomes difficult for the video distribution device 20 to timely receive posture feature amounts with a large amount of data. Even in such a case, if the animation of the avatar based on the posture feature is continued, the movement of the avatar in the video may become unnatural. Such unnatural movements of the avatar are undesirable because they degrade the quality of the video being distributed. When a delay occurs in the transmission path, by interrupting the generation of avatar animation based on posture features, it is possible to prevent the avatar from making unnatural movements. Even if the generation of the avatar animation based on the posture feature amount is interrupted, the animation generation unit 21c specifies the registered animation ID of the avatar based on the reference posture identification data from the video playback device 10, and generates the specified registered animation ID. An avatar animation can be generated based on the animation definition data associated with the ID. If the viewing user does not take a posture that matches the reference posture while a delay occurs in the transmission path, the video distribution device 20 does not receive the reference posture identification data. In this case, the animation generation unit 21c does not need to generate the animation of the avatar of the viewing user. When the animation generation unit 21c does not generate an animation of the avatar, the avatar remains stationary within the video. In another embodiment, if the video distribution device 20 does not receive the reference posture identification data while a delay occurs on the transmission path, the video distribution device 20 sends the video distribution device 20 an instruction to perform a movement in accordance with the basic motion determined for the avatar. You may also create an animation for your avatar. The basic movements of the avatar refer to predetermined movements such as shaking the hands up and down, shaking the hands from side to side, and jumping. This basic operation may be set in common for a plurality of viewing users. The basic movements of the avatar are generated by the animation generation unit 21c in that they can be performed by the avatar without receiving information regarding the movement of the avatar (three-dimensional skeleton data or reference posture identification data) from the video playback device 10. It's different from animation.

When the animation generation unit 21c generates the animation of the avatar, the animation generation unit 21a generates a video including the animation, and the video distribution unit 21b distributes the video including the avatar animation to the video playback device 10. When the animation 60 shown in FIG. 9a includes the animation of the avatar object 56a, the animation 60 including the animation of the avatar object 56a is displayed. For example, when an animation is generated in which the avatar object 56a raises its right hand, the avatar object 56a raises its right hand in the moving image 60, as shown in FIG. 9b.

As described above, the video distribution device 20 may request the video playback device 10 to determine an additional reference posture that becomes a trigger for including the user animation whose registration is requested in the video. Although some aspects of the method for determining the additional reference posture have been described above, another aspect of determining the additional reference posture will be described with further reference to FIG. 8. In the example below, a feature point of a user's image is extracted, and an additional reference posture is determined using the root mean square of the posture feature amount at the feature point. First, the video playback device 10 instructs the user to perform a movement that the user wants to register as a user animation, and based on the instruction, captures a plurality of frames by capturing an image of the user performing the movement that the user wants to register at a predetermined frame rate. Obtain a captured image containing the image. The video playback device 10 obtains posture feature amounts for each of the plurality of extracted feature points. Extraction of feature points in each of a plurality of frames of a captured image of a user has already been described with reference to FIG. 7, so a repeated explanation will not be given.

FIG. 8 shows a table including posture feature amounts for each of the 24 frames from the 0th frame to the 23rd frame. In the table of FIG. 8, the posture feature amount of each of the feature points P1 to P6 is shown in columns "P1" to "P6", respectively. This posture feature amount is normalized to a range of 0 to 1.0. The video playback device 10 calculates the root mean square (RMS) of the posture feature amount of each feature point P1 to P6 for each frame. If the posture feature at each of the N feature points P ₁ to P _N is x _i (where i takes a value of 1 to N), then the RMS of the posture feature at the feature points P1 to P _N ( ) is expressed by the following formula.

FIG. 8 shows RMS(x) calculated for each frame. In addition to the RMS(x), the video playback device 10 indicates, for each frame, the average value of the posture feature amount at each feature point and the difference between the RMS(x) of the current frame and the RMS(x) of the previous frame. An RMS difference may also be calculated. The video playback device 10 may obtain, for each frame, a positive/negative flag indicating whether the RMS difference is positive or negative, and an inversion flag indicating whether the sign of the RMS difference has been reversed. For example, the RMS difference in the second frame is the difference between the RMS (x ₂ ) in the second frame and the RMS (x ₁ ) in the first frame. For example, the positive/negative flag is "1" when the RMS difference is positive, and is "0" when the RMS difference is negative. For example, the inversion flag is "1" when the sign of the RMS difference is reversed, and becomes "0" when the sign of the RMS difference is not reversed (that is, the sign is the same as that of all frames). FIG. 8 includes these average values, RMS differences, positive/negative flags, and inversion flags. In FIG. 8, the sign of the RMS difference in the 4th frame, 10th frame, 16th frame, 17th frame, 19th frame, and 21st frame is reversed from that of the immediately preceding frame, so the reversal flag is set as "1" is set. Since there is no frame preceding the 0th frame, the RMS difference for the 0th frame is blank.

In a frame where the reversal flag is "1", the sign of the RMS difference is reversed from the previous frame, so it is estimated that the user made a large movement in that frame. Large movements that can be detected by the reversal flag are often periodic movements such as waving a hand or blinking an eye. Therefore, in an embodiment of the present invention, a frame for which the inversion flag is set to "1", that is, a frame in which the sign of the RMS difference in posture features is reversed from the previous frame, is set as the start frame, and the RMS It becomes easy to pay attention to the interval until the sign of the difference is reversed again (until the reversal flag becomes "1"). The posture of the user in this starting frame can be set as the starting posture. Further, a frame in which the reversal flag is set to "1" after the start frame can be set as a trigger frame, and the user's posture in this trigger frame can be set as the trigger posture. In this way, the user's motion and posture can be analyzed by focusing on the period from when the reversal flag becomes "1" until the next time when the reversal flag becomes "1". In the example shown in FIG. 8, the reversal flag becomes "1" for the first time in the fourth frame, and the reversal flag becomes "1" for the second time in the tenth frame. Therefore, it is possible to extract periodic motions and their trigger postures with the user's posture in the fourth frame as the starting posture and the user's posture in the tenth frame as the trigger posture at a low calculation cost. The starting attitude data indicating the starting attitude and the trigger attitude data indicating the trigger attitude are calculated from the Root Mean Squared Error (RMSE), the Root Mean Squared Percentage Error (RMSPTE), or the captured image. or can be estimated.

In the example shown in FIG. 8, the reversal flag is set to "1" in both the 16th frame and the 17th frame. Since the time interval between successive frames is very short, it may not be appropriate to use the user's pose in each of these adjacent frames as the starting pose and the trigger pose. Therefore, a lower limit number of frames may be determined, which is the lower limit of the number of frames that should exist between the start frame and the trigger frame. The lower limit number of frames may be set to 3 frames or more, for example, taking into account compression of communication packets, differences with neighboring frames, and other factors. A lower limit on the time interval between the start frame and the trigger frame may be defined. The lower limit of the time interval between the start frame and the trigger frame is, for example, 1 second or more, 2 seconds or more, 3 seconds or more, 4 seconds or more, taking into account the operating frequency, the target user's behavior, and other factors. It is good also as a lower limit of 2 seconds or more, 5 seconds or more, or other than these. If the lower limit number of frames is 3 frames, there are 5 frames that exceed the lower limit frame number between the 4th frame and the 10th frame in FIG. 8, so the 10th frame can be treated as a trigger frame. I can do it. If the reversal flag is set to "1" in the 7th frame, the 7th frame is not set as the trigger frame because there are only 2 frames that are less than the minimum number of frames between the 4th frame and the 7th frame. Then, the 10th frame in which the inversion flag becomes "1" can be set as the trigger frame. However, since the RMSPE from the 10th frame to the 16th frame, which is the next target frame, is approximately 1 or less, the 10th frame may be determined to be noise and not be processed as a trigger frame.

As described above, by determining additional reference postures (starting posture and trigger posture) using the root mean square of the posture features at the user's feature points, user animation can be registered without presenting candidate frames or sample videos. In addition, a starting posture and a trigger posture corresponding to the user animation can be determined.

Next, with reference to FIG. 10, a part of the processing included in the video playback method according to one embodiment will be described. FIG. 10 is a flow diagram showing a part of the process in the video playback method in one embodiment. In the video playback method of FIG. 10, it is assumed that the viewing user is playing back the video using the video playback device 10 and has his or her avatar participating in the video. In other words, the video that the viewing user is viewing includes his or her own avatar. The process shown in FIG. 10 relates to the transmission of posture feature amounts and reference posture identification data to the video distribution device 20, which is performed while viewing a video.

First, in step S11, data regarding the posture feature amount of the viewing user who is viewing the video is acquired. Furthermore, in step S11, the acquired posture feature amount is immediately transmitted to the video distribution device 20. In step S11, motion data including posture features and other data may be transmitted to the video distribution device 20. The viewing user's posture feature amount may be three-dimensional skeleton data representing the posture of the viewing user. The posture feature amount may be acquired at every predetermined sampling time interval. The posture feature amount of the viewing user is acquired by, for example, the posture data acquisition section 11b or the sensor unit 15 described above. The posture feature amount is transmitted, for example, by the above-mentioned transmitter 11c.

Next, in step S12, based on the posture feature amount of the viewing user acquired in step S11, it is determined whether the posture of the viewing user belongs to a predetermined reference posture. This determination is performed, for example, by the classification section 11d described above. If the viewing user's posture does not belong to the reference posture, the process returns to step S11, and acquisition of posture feature amounts is continued in step S11. If it is determined that the viewing user's posture belongs to any of the reference postures, the process proceeds to step S13.

In step S13, it is determined whether there is a delay in the transmission of motion data on the transmission path between the video playback device 10 and the video distribution device 20. This determination is made, for example, by the delay monitoring unit 11e described above. If it is determined that no delay has occurred in the transmission path, the process returns to step S11, and acquisition of posture feature amounts is continued in step S11. If it is determined that a delay has occurred in the transmission path, the process proceeds to step S14.

In step S14, reference posture identification data that identifies the reference posture to which the posture of the viewing user was determined to belong in step S12 is transmitted to the video distribution device 20. Transmission of the reference posture identification data is performed by, for example, the above-mentioned transmitter 11c. If it is determined in step S12 that the viewing user's posture belongs to the starting posture, and then it is further determined that the posture belongs to the trigger posture, trigger posture data identifying this trigger posture is transmitted.

In parallel with the processing in each step from step S11 to step S14, the video is continuously distributed from the video distribution device 20 to the video playback device 10. During video distribution, the processes of steps S11 to S14 are repeated. If it is determined in step S13 that no delay has occurred in the transmission path, the video to be distributed includes an avatar animation generated based on the posture feature amount of the viewing user. Conversely, if it is determined in step S13 that a delay has occurred in the transmission path, the video to be distributed does not include the registration of the avatar specified based on the reference posture identification data transmitted to the video distribution device 20. Animation may be included.

Step S13 may be executed before step S11 or between step S11 and step S12. In addition to this, the order of the processes shown in FIG. 10 may be changed as appropriate. Further, in addition to the processes in steps S11 to S14, processes not explicitly described in FIG. 10 may be performed.

Next, other embodiments of the present invention will be described with reference to FIGS. 11 to 13. FIG. 11 shows a block diagram of a video distribution system 101 according to another embodiment of the present invention. The video distribution system 101 includes a video playback device 110 and a video distribution device 120. The video playback device 110 of the video distribution system 101 is different from the video playback device 10 of the video distribution system 1 in that the computer processor 11 functions as a facial feature acquisition unit 11f, and the storage 13 stores reference facial expression management data 13b. different. The video distribution device 120 of the video distribution system 101 differs from the video distribution device 20 of the video distribution system 1 in that the storage 23 stores registered facial expression management data 23d. In the following, differences between the embodiment of FIG. 11 and the embodiment of FIG. 1 will be explained. In the embodiment of FIG. 11, descriptions of matters common to the embodiment of FIG. 1 will be omitted.

The facial feature amount acquisition unit 11f acquires facial feature amounts representing facial features of a viewing user who views a video using the video playback device 110. Specifically, the facial feature acquisition unit 11f extracts features representing facial features from the image including the viewing user's face acquired by the sensor unit 15, for example, according to HOG, SIFT, SURF, or other known algorithms. Calculate the amount. For example, the facial feature amount calculated by SIFT is calculated as a 128-dimensional feature amount. In this way, the facial feature amount has a large amount of data, similar to the posture feature amount acquired by the posture data acquisition unit 11b.

FIG. 12 shows an example of reference facial expression management data. As shown in FIG. 12, the storage 13 stores standard facial expression identification data indicating the standard facial expression in association with standard facial expression identifying data for identifying the standard facial expression. The standard facial expression identification data is, for example, a standard facial expression ID that identifies a standard facial expression. The reference facial expression ID is, for example, an identification code expressed by several bits. The number of bits of the standard facial expression ID can be determined according to the number of standard facial expressions used in the video distribution system 1. The reference facial expression ID is represented by, for example, data with an information amount of 10 bits or less. The reference facial expression ID may be represented by data with an information amount of 5 bits or less, 4 bits or less, 3 bits or less, or 2 bits or less. Therefore, the standard facial expression identification data (standard facial expression ID) has a much smaller amount of data than the facial feature amount.

The standard facial expression identification data is data indicating a standard facial expression. One or more reference facial expressions can be set in the video distribution system 1. The standard facial expression is, for example, a facial expression with eyes wide open facing the front. The reference facial expression identification data is data that describes a reference facial expression, and has the same data structure as the facial feature amount calculated by the facial feature amount acquisition unit 11f. When the facial feature amount calculated by the facial feature amount acquisition unit 11f is a SIFT feature amount calculated by the SIFT algorithm, the reference facial expression identification data is the one in which the reference facial expression is described in the same data format as the SIFT feature amount. be.

The classification unit 11d uses a classifier that classifies the facial expressions of the viewing user to determine whether the facial expression of the viewing user belongs to a predetermined reference facial expression based on the facial feature amount of the viewing user. In one embodiment, in order to obtain training data, the classification unit 11d can display an image representing a reference facial expression on the display 14 and prompt the viewing user to make a facial expression according to the image representing the standard facial expression. The content reproduction device 110 uses the facial feature acquisition unit 11f to acquire the facial feature amount of the facial expression made by the viewing user in response to the display of the image representing the reference facial expression. The classification unit 11d can create a classifier by learning, as training data, the facial features of the facial expressions made by the viewing user with respect to the image representing the reference facial expression.

The transmitting unit 11c transmits to the video distribution device 120 motion data including facial features representing facial features of the viewing user acquired by the facial feature acquiring unit 11f. The transmitting unit 11c transmits the standard facial expression identification data (standard facial expression ID) stored in the storage 13 to video distribution when the delay monitoring unit 11e determines that a delay has occurred in the transmission of motion data on the transmission path. The information may be transmitted to device 120. If the delay monitoring unit 11e determines that a delay has occurred in the transmission path, the reference facial expression identification data may be transmitted to the video distribution device 120 instead of the facial feature amount.

In the video distribution device 20, the animation generation unit 21c generates animations of avatars of viewers participating in the video 60. In one embodiment, the animation generation unit 21c generates an animation of the avatar of the viewing user based on facial feature amounts of the viewing user received from the video playback device 110. The animation generation unit 21c continuously receives facial features of the viewing user from the video playback device 110, and generates animations in synchronization with facial movements (changes in facial expression) of the viewing user based on the facial features. It is possible to generate avatar animations that change facial expressions.

The animation generation unit 21c in one embodiment generates an avatar animation including facial movements based on the standard facial expression identification data (for example, standard facial expression ID) from the video playback device 110 by referring to the registered facial expression management data 23d. can be generated. FIG. 13 shows an example of registered facial expression management data 23d in the embodiment of FIG. 11. As shown in FIG. 13, the registered facial expression management data 23d includes reference facial expression identification data (registered facial expression ID), registered facial expression identification data (registered facial expression ID) that identifies registered facial expressions, and information about movements of facial expressions of the avatar. and animation definition data for specifying the included animation. This animation definition data is data that defines the movements of the avatar's facial expressions. The animation definition data included in the registered facial expression management data 23d may be data that describes the positions of the feature points of the avatar's face in time series. When receiving the standard facial expression identification data from the video playback device 110, the animation generation unit 21c identifies the registered facial expression identification data that is associated with the received standard facial expression identification data by referring to the animation management data 23d. An animation of the avatar including facial movements can be generated based on the animation definition data associated with the specified registered facial expression identification data.

Next, referring to FIG. 14, another embodiment of the present invention will be described. FIG. 14 shows a block diagram of a video distribution system 201 according to another embodiment of the present invention. Generally speaking, in the video distribution system 1 shown in FIG. 1, posture features representing the posture of the viewing user are transmitted in real time, whereas in the video distribution system 201 shown in FIG. The two differ in that the posture feature amount of the viewing user is transmitted only when the viewing user makes a characteristic movement. The video distribution system 201 includes a video playback device 210 and a video distribution device 220. In the video playback device 210 of the video distribution system 201, the computer processor 11 functions as a determination unit 11g and a transmission unit 11h. The video playback device 210 does not need to have the reference facial expression management data 13b, and the video distribution device 220 does not need to have the animation management data 23c. In the embodiment of FIG. 14, descriptions of matters common to the embodiment of FIG. 1 will be omitted.

The determination unit 11g determines whether the user of the video playback device 210 has made a characteristic movement. Specifically, the posture data acquisition unit 11b calculates the posture feature amount of the feature point of the user of the video playback device 210 at a predetermined frame rate, and the determination unit 11g calculates the posture feature amount based on the RMS difference of the posture feature amount of this frame. It is determined whether the user has made a characteristic movement. The calculation of the posture feature amount for each frame and the calculation of the RMS difference have already been explained with reference to FIG. 8 . In one embodiment, the determination unit 11g determines that the user performed a characteristic movement in a frame in which the sign of the RMS difference is reversed. As mentioned above, if the sign of the RMS difference in a certain frame is reversed from that of the previous frame, it is assumed that the user performed a characteristic movement involving reciprocation in that frame, such as waving his hand or nodding his head. Ru. For example, when the posture data acquisition unit 11b calculates posture feature amounts for 24 frames shown in FIG. It is determined that the user made a characteristic movement in the 17th frame, the 19th frame, and the 21st frame.

In one embodiment, the determination unit 11g determines that the user has made a large movement in a frame in which the RMSPE is greater than a predetermined threshold. This threshold value may be, for example, 1.0. The threshold value used by the determination unit 11g can be changed as appropriate. In the example of FIG. 8, RMSPE is larger than the threshold value of 1.0 in the fourth frame to the seventh frame. Therefore, the determination unit 11g can determine that the user made a large movement from the fourth frame to the eighth frame.

The transmitting unit 11h selects a frame in which the determining unit 11g determines that the user made a characteristic movement, and transmits the posture feature amount in the selected frame to the video distribution device 220. The frame selected by the transmitter 11h may be referred to as a "selected frame." The selected frames may be only the frames in which the determination unit 11g has determined that the user has performed a characteristic movement, or the frames in which the determination unit 11g has determined that the user has performed a characteristic movement and the frames that follow. Or it may be multiple frames. In this way, the transmitter 11h is configured to transmit the posture feature amount in the selected frame to the video distribution device 220, while not transmitting the posture feature amount in frames other than the selected frame to the video distribution device 220.

In the video distribution device 220 that has received the posture feature amount transmitted by the transmitter 11h, the animation generation unit 21c generates an animation of the avatar of the user of the video playback device 210 based on the received posture feature amount. The method of generating the avatar animation based on the posture feature amount is the same as the generation method in the video distribution system 1. In the video distribution system 201, there is a period in which the posture feature amount is not transmitted from the video playback device 210 to the video distribution device 220. Specifically, the posture feature amount is not transmitted to the video distribution device 220 in frames other than the selected frame. During this period in which the posture feature amount is not transmitted, the animation generation unit 21c may create an animation of the avatar so that the avatar moves in accordance with the basic motion determined for the avatar. The basic movements of the avatar are as described, and are predetermined movements such as shaking hands.

In the video distribution system 201, the posture feature amount is transmitted from the video playback device 210 to the video distribution device 220 only in the selected frame. can be reduced.

Next, referring to FIG. 15, another embodiment of the present invention will be described. FIG. 15 shows a block diagram of a video distribution system 301 according to another embodiment of the present invention. The video distribution system 1 differs from the video distribution system 1 in that a video playback device generates a video based on data necessary for video generation, such as posture feature amounts. Specifically, the video distribution system 301 includes a posture data acquisition device 310, a video distribution device 320, and a video playback device 330. Posture data acquisition device 310, video distribution device 320, and video playback device 330 are connected via network 50. In the video distribution system 301, it is assumed that the video playback device 320 generates a video including an avatar animation generated based on the movement of the user of the posture data acquisition device 310, and that the generated video is played back. . In other words, the user of the posture data acquisition device 310 can play back a video including an avatar that moves based on the user's own movements on the video playback device 320 used by the viewing user. The user of the posture data acquisition device 310 can interact with the user of the video playback device 320 via the avatar. Regarding this embodiment, the user of the posture data acquisition device 310 may be simply referred to as a "distribution user." Although FIG. 15 shows one posture data acquisition device 310, one video distribution device 320, and one video playback device 330 for explanation, the video distribution system 301 may include a plurality of these devices.

The posture data acquisition device 310 acquires posture feature amounts representing the posture characteristics of the user of the posture data acquisition device 310 . The posture feature quantity representing the posture characteristics of the viewing user may be three-dimensional skeletal data representing the posture of the viewing user, that is, three-dimensional vector data representing the positions and orientations of the bones of the viewing user. The posture data acquisition device 310 may include the sensor unit 15 described above. The posture data acquisition device 310 can generate three-dimensional skeletal data of the viewing user based on the detection data detected by the sensor unit 15. Posture data acquisition device 310 transmits the obtained posture feature amount to video distribution device 320. The posture data acquisition device 310 may transmit audio data representing the voice of the distribution user to the video distribution device 320.

The video distribution device 320 includes a computer processor 21, a communication I/F 22, and a storage 23, like the video distribution device 20 described above. The video distribution device 320 differs from the video distribution device 20 in the functions realized by the computer processor 21 and the data stored in the storage 23. Therefore, the functions realized in the computer processor 21 of the video distribution device 320 and the data stored in the storage 23 will be explained below.

As illustrated, the storage 23 of the video distribution system 301 can store object data 23a, avatar data 23b, reference posture management data 323a, and various other information necessary for generating and distributing videos other than the above. . The object data 23a and the avatar data 23b may be the same as the object data 23a and the avatar data 23b stored in the video distribution system 1. The reference posture management data 323a may be the same as the reference posture management data 13a in the video distribution system 1. That is, as shown in FIG. 3, the reference attitude management data 323a includes reference attitude identification data that identifies a reference attitude, and start attitude data and trigger attitude data that are stored in association with this reference attitude identification data. can be included.

The computer processor 21 functions as a posture feature acquisition section 321a, a transmission section 321b, a classification section 321c, and a delay monitoring section 321d by executing computer-readable instructions. At least some of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the video distribution system 301.

The posture feature acquisition unit 321a acquires posture features representing the posture of the user of the posture data acquisition device 310 from the posture data acquisition device 310.

The transmitter 321b transmits the posture feature amount representing the posture of the user of the posture data acquisition device 310 acquired from the posture data acquisition device 310 to the video playback device 330. The transmitter 321b may transmit the posture feature amount to the video playback device 330 immediately upon receiving the posture feature amount from the posture data acquisition device 310. In other words, the transmitter 321b can transmit the distribution user's posture feature amount to the video playback device 330 in real time. The transmitter 321b may transmit a packet including this posture feature amount to the video playback device 330. The transmitter 321b can transmit to the video playback device 330 data necessary for generating a video in the video playback device 330 in addition to the posture feature amount. For example, the transmitter 321b can transmit at least part of the object data 23a and the avatar data 23b to the video playback device 330. The object data 23a and the avatar data 23b may be stored in advance in the video playback device 330 instead of being transmitted to the video playback device 330 by the transmitter 321b. In this case, it is not necessary to transmit the object data 23a and avatar data 23b from the video distribution device 320 to the video playback device 330. Further, the transmitter 321b may transmit audio data representing the distribution user's voice, which the video distribution device 320 has acquired from the generated data acquisition device 310, to the video playback device 330.

The classification unit 321c uses a classifier that classifies the posture of the distribution user to determine whether the posture of the viewing user belongs to a predetermined reference posture based on the posture feature amount (for example, three-dimensional skeleton data) of the viewing user. Determine whether or not. This classifier is, for example, a linear classifier. The classification unit 321c may realize substantially the same function as the classification unit 11d of the video distribution system 1. For example, when the reference posture includes a starting posture and a trigger posture, the classification unit 321c can determine whether the distribution user's posture belongs to the reference posture and the trigger posture, respectively. .

The delay monitoring unit 321d monitors the transmission delay of motion data transmitted from the video distribution device 320 to the video playback device 330 by the transmission unit 321c. The delay monitoring unit 321d may realize substantially the same function as the delay monitoring unit 11e of the video distribution system 1. For example, the delay monitoring unit 321d adds a timestamp to an actual packet including motion data before transmission, and the timestamp added at the time of transmission and the timestamp added when this actual packet is received by the video playback device 330. The transmission time of the actual packet on the transmission path between the video distribution device 320 and the video playback device 330 can be determined using the time stamp. The presence or absence of a transmission delay may be determined by the video playback device 330 based on the time stamp included in the packet. The video playback device 320 may receive the determination result of the transmission delay in the video distribution device 330.

The function of the transmitter 321b will be explained again. The transmitter 321b transmits the reference attitude identification data (reference attitude ID) stored in the storage 23 to the video playback device 330 when the delay monitor 321d determines that a delay has occurred in the transmission path. be able to. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As mentioned above, the reference attitude identification data may include starting attitude data and trigger attitude data. The function of the transmitter 321b executed when this transmission delay occurs is the same as the function executed by the transmitter 11c when a transmission delay occurs.

The video playback device 330 includes a computer processor 11, a communication I/F 12, and a storage 13, similar to the video playback device 10 described above. The video playback device 330 differs from the video playback device 10 in the functions realized by the computer processor 11 and the data stored in the storage 23. Therefore, the functions realized in the computer processor 21 of the video playback device 330 and the data stored in the storage 23 will be explained below.

Animation management data 313a is stored in the storage 13 of the video playback device 330. The animation management data 313a may be the same as the animation management data 13a stored in the video distribution system 1. For example, as shown in FIG. 6, the animation management data 313a includes reference posture identification data, registered animation identification data (registered animation ID) for identifying registered animations, and animation definitions for specifying avatar animations. has data.

The computer processor 11 functions as an animation generation unit 331a, a video generation unit 331b, and a video playback unit 331c by executing computer-readable instructions. At least some of the functions realized by the computer processor 11 may be realized by a computer processor other than the computer processor 11 of the video distribution system 301.

The animation generation section 331a may realize substantially the same function as the animation generation section 21c of the video distribution system 1. For example, the animation generation unit 331c can generate an animation of the avatar of the distribution user of the posture data acquisition device 310. In one embodiment, the animation generation unit 331c generates an animation of the distribution user's avatar based on the posture feature amount and avatar data 23b of the distribution user received from the video distribution device 320. The animation generation unit 331c continuously receives the posture feature amount of the viewing user from the video distribution device 320, and generates an animation of the avatar that moves in synchronization with the body movement of the viewing user based on the posture feature amount. can be generated.

The video generation unit 331b can construct a virtual space using the object data 23a acquired from the video distribution device 320, and generate a video in which the avatar generated by the animation generation unit 331a moves within this virtual space. This video can be synthesized with the voice of the distribution user.

The video playback section 331a plays back the video generated by the video generation section 331b. As a result, a video including an animation of the distribution user's avatar is displayed on the display 14 of the video playback device 330.

A process implemented by the animation generation unit 331a when a transmission delay occurs in the transmission path that transmits data from the video distribution device 320 to the video playback device 330 will be described. If a transmission delay occurs in the transmission path from the video distribution device 320 to the video playback device 330, there is a risk that it will not be possible to generate an avatar animation that moves smoothly while reflecting the movements of the distribution user. The animation generation unit 331a can generate an avatar animation based on reference posture identification data (for example, reference posture ID) from the video distribution device 320. Specifically, upon receiving the reference posture identification data from the video playback device 10, the animation generation unit 331a identifies the registered animation associated with the received reference posture identification data by referring to the animation management data 313a. The avatar animation can be generated based on the animation definition data associated with the registered animation ID of the specified registered animation. Therefore, even if the animation generation unit 331a cannot receive the posture feature amount while there is a delay in the transmission of motion data, the animation generation unit 331a can receive the posture feature amount from the video distribution device 320 while the delay is occurring. An avatar animation can be generated based on the reference posture identification data.

In one embodiment, the posture data acquisition device 310 may be configured to be able to realize the functions of the video playback device 320, and the video playback device 320 may be configured to be able to realize the functions of the posture data acquisition device 310. This allows users of both devices to communicate interactively via the avatars.

Next, a modification of the above video distribution system will be described with reference to FIG. 16. In the above embodiment, when the classification unit 11d determines that the viewing user's posture belongs to the standard posture, the user's posture is determined by transmitting a standard posture ID that identifies the standard posture instead of the posture feature amount. The amount of data represented is reduced. In the example shown in FIG. 16, an index (subscript) of a feature array determined as follows is used instead of the reference posture ID. That is, when a transmission delay occurs, the index of the feature array is transmitted instead of the posture feature amount representing the user's posture. The index of the feature array is data of about several bits, as described below, and can be expressed by the amount of information in the range of 10 bits or less, similar to the reference posture ID.

In the video distribution system 1, the feature array is determined as follows. First, as described with reference to FIGS. 7 and 8, the video playback device 10 acquires a captured image by capturing a user of the video playback device 10 at a predetermined frame rate, and in each frame of this captured image, A plurality of feature points related to the user are extracted. For example, as shown in FIG. 7, six feature points P1 to P6 are extracted. The video playback device 10 obtains an image vector related to the user's movement for each of the plurality of extracted feature points. The image vector at each of the feature points P1 to P6 is, for example, the depth at each feature point. The image vector of each feature point P1 to P6 may be normalized to a range of 0 to 1.0 and expressed as a float array. The image vectors of the feature points P1 to P6 obtained in this way are shown in FIG. In FIG. 16, image vectors for 10 frames are shown to simplify the explanation, but in reality, image vectors are obtained for a sufficient number of frames for statistical analysis.

Next, for each of the feature points P1 to P6, the sum Σ of the image vector elements in a predetermined interval is calculated. In the table of FIG. 16, the sum of the elements of the image vector in the 26-frame interval from the 0th frame to the 25th frame is shown in the row labeled "Σ". Next, the sum of the elements of the image vector of each feature point P1 to P6 is sorted in descending order. In FIG. 16, this sorting result is shown in the line labeled "RANK". In the example of FIG. 16, the items are sorted in the order of P4, P3, P6, P2, P1, and P5. This sorted array is expressed as {4,3,6,2,1,5}.

The video playback device 10 calculates a representative vector indicating a characteristic posture or movement of the user by learning samples of image vectors related to the user's movement. The representative vector is calculated using, for example, Lloyd's algorithm or other known algorithms. In other words, the representative vector indicates a characteristic posture of the user. Therefore, the user's posture can be specified based on this representative vector. The video playback device stores the calculated representative vector along with the index. The video playback device 10 may acquire a representative vector calculated by a device other than the device itself. In this embodiment, four representative vectors A to D are calculated, and the elements of each representative vector are sorted in ascending order to obtain an array of representative vectors. The arrays obtained by sorting the elements of representative vectors A to D in descending order are {1,3,6,2,4,5}, {4,3,6,2,1,5}{2,3, 6,4,5,1}{1,2,3,4,5,6}.

The video playback device 10 compares the input array {4, 3, 6, 2, 1, 5} obtained by sorting the image vectors obtained from the captured images with the array of each representative vector, and calculates the input array { 4,3,6,2,1,5}, select the closest representative vector array. In the example shown in FIG. 16, the input array {4,3,6,2,1,5} is the array of representative vector B {4,3,6,2,1, 5}, so {4,3,6,2,1,5} is selected as the feature array. The video playback device 10 transmits the index of the feature array selected in this way to the video distribution device 20.

The video distribution device 20 stores a codebook in which representative vectors A to D are associated with their arrays. The video distribution device 20 refers to the codebook and generates an animation of the user's avatar so as to take a posture corresponding to the representative vector B corresponding to the feature array acquired from the video playback device 10.

In the above example, the feature array is transmitted from the video playback device 10 to the video distribution device 20 for each unit section of 26 frames. This unit section can be adjusted as appropriate.

As described above, it has been explained that in the video distribution system 1, an index of a feature array is transmitted in place of the reference posture ID, and an avatar animation is generated based on the index of this feature array when transmission is delayed. Similarly, in the video distribution system 301, a feature array can be used instead of the reference posture ID. The feature array can also be used in place of the reference facial expression ID in the video distribution system 101.

The effects achieved by the above embodiment will be explained. In one aspect of the above, while a delay occurs in the transmission path, the reference posture identification data that can identify information about the viewing user's posture with a small amount of data is used in the video instead of the posture feature amount with a large amount of data. It is transmitted from the playback device 10 to the video distribution device 20. Thereby, when a delay occurs in the transmission path, information regarding the viewing user's posture can be transmitted to the video distribution device 20 using a small amount of reference posture identification data.

In one aspect, a registered animation can be registered regarding the avatar of a viewing user, and while a delay occurs in the transmission path, a registered animation corresponding to a reference posture specified based on reference posture identification data is registered. A video containing the content will be distributed. As a result, even if it is difficult to transmit the posture feature representing the posture of the viewing user to the video distribution device 20 due to a delay in the transmission path from the video playback device 10 to the video distribution device 20, the viewing user's Registered animations related to posture can be included in the video.

In the above aspect, the video playback device 330 generates an animation of an avatar that moves based on the movement of the distribution user based on the posture feature amount of the distribution user received from the video distribution device 320. While a delay occurs in the transmission path, reference posture identification data rather than the posture feature amount is transmitted from the video distribution device 320 to the video playback device 310. As a result, when a delay occurs in the transmission path, an animation of the avatar can be generated in the video playback device 330 using the reference posture identification data that can represent the movement of the distribution user with a small amount of data.

In one aspect, a user animation generated based on a user's movement can be registered as a registered animation. Thereby, it is possible to increase the variation of the avatar's movement when the posture feature cannot be used.

In one aspect, while a delay occurs in the transmission path, the video playback device uses reference facial expression identification data that can identify information about the posture of the viewing user using a small amount of data, rather than using facial features that have a large amount of data. 10 to the video distribution device 20. As a result, when a delay occurs in the transmission path, information regarding the facial expression of the viewing user can be transmitted to the video distribution device 20 using a small amount of reference facial expression identification data.

In the processing procedures described in this specification, particularly those explained using flowcharts, some of the steps constituting the processing procedure may be omitted, and some of the steps constituting the processing procedure may be omitted. It is possible to add steps that are not specified and/or change the order of the steps, and such omissions, additions, or changes in the order of processing procedures are also included in the present invention as long as they do not depart from the spirit of the present invention. within the scope of the invention.

Below, the invention described in the claims of the original application of this application at the time of filing will be added.
[1]
comprising one or more computer processors;
The one or more computer processors may execute computer readable instructions to:
Sending motion data including posture features representing characteristics of the user's posture to a video distribution device,
If there is no delay in transmitting the motion data to the video distribution device, a video including an avatar of the user performing a movement based on the posture feature amount is received from the video distribution device, and the motion data is transmitted to the video distribution device. If there is a delay in transmitting data to the video distribution device, receiving a video including the avatar that moves in accordance with the basic motion set for the user;
Play the video,
Video playback device.
[2]
The video includes avatars of other users who view the video.
Video playback device according to [1] [3]
the other user's avatar moves in accordance with the basic movement set for the other user;
The video playback device according to [2].
[4]
The basic operation set for the user is the same as the basic operation set for the other user,
The video playback device according to [3].
[5]
The posture feature amount includes bone data representing the position and orientation of the user's bones as a three-dimensional vector.
The video playback device according to any one of [1] to [4].
[6]
the motion data is transmitted in real time;
The video playback device according to any one of [1] to [5].
[7]
A video playback method performed by one or more computer processors executing computer-readable instructions, the method comprising:
transmitting motion data including posture features representing the user's posture to a video distribution device;
If there is no delay in transmitting the motion data to the video distribution device, a video including an avatar of the user performing a movement based on the posture feature amount is received from the video distribution device, and the motion data is transmitted to the video distribution device. If there is a delay in transmitting data to the video distribution device, receiving a video including the avatar that moves in accordance with basic actions set for the user;
a step of playing the video;
A video playback method comprising:
[8]
comprising one or more computer processors;
The one or more computer processors may execute computer readable instructions to:
Receive motion data including posture features representing characteristics of the user's posture from the video distribution device,
If there is no delay in transmitting the motion data to the video distribution device, the video distribution device generates a video including the avatar of the user performing a movement based on the posture feature amount, and transmits the motion data to the video distribution device. If there is a delay in transmitting data to the video distribution device, generate a video including the avatar that moves in accordance with the basic actions set for the user;
Play the video,
Video playback device.
[9]
comprising one or more computer processors;
The one or more computer processors may execute computer readable instructions to:
Image the user over multiple frames,
acquiring a posture feature representing a feature of the user's posture for each of the plurality of frames;
Selecting a selected frame including a frame in which the user made a characteristic movement from among the plurality of frames based on the posture feature amount of each frame included in the plurality of frames;
transmitting motion data including the posture feature amount in the selected frame to the video distribution device;
During the period when the selected frame is being transmitted to the video distribution device, a video including the user's avatar that makes a movement based on the posture feature amount is received from the video distribution device, and the selected frame is transmitted to the video distribution device. During a period when the video is not being transmitted to the device, receiving a video including the avatar performing movements according to basic actions set for the user;
Play the video,
Video playback device.

1, 101, 201, 301 Video distribution system 10, 110, 210, 330 Video playback device 11a Video playback section 11b Posture data acquisition section 11c Transmission section 11d Classification section 11e Delay monitoring section 11f Facial feature amount acquisition section 20, 120, 220 , 320 Video distribution device 21a Video generation section 21b Video distribution section 21c Animation generation section 310 Posture data acquisition device

Claims (10)

comprising one or more computer processors and storage for storing registered animation;
The one or more computer processors may execute computer readable instructions to:
Receive motion data including posture features representing characteristics of the user's posture from the video distribution device,
generating a video including an animation of the user's avatar generated based on the motion data;
a reference posture that identifies a reference posture that has a smaller amount of data than the posture feature and is determined to match the user's posture while the posture feature cannot be received during transmission of the motion data from the video distribution device; generating a video including the registered animation corresponding to the reference posture based on identification data;
Video playback device. The posture feature amount includes bone data representing the position and orientation of the user's bones as a three-dimensional vector.
The video playback device according to claim 1. The reference posture identification data is transmitted from the video distribution device in place of the motion data while the posture feature cannot be received.
The video playback device according to claim 1 or 2. The motion data includes facial feature amounts representing features of the user's face,
The one or more computer processors include:
determining whether the user's facial expression matches a predetermined reference facial expression using another classifier that classifies the user's facial expression based on the facial feature amount;
A criterion for identifying the reference facial expression that has a smaller data amount than the facial feature amount and is determined to match the user's facial expression while the facial feature amount cannot be received during transmission of the motion data from the video distribution device. generating an animation of the avatar including facial movements corresponding to the reference facial expression based on facial expression identification data;
The video playback device according to claim 1 or 2. The posture feature amount is acquired by a posture data acquisition device and transmitted from the posture data acquisition device to the video distribution device.
The video playback device according to any one of claims 1 to 4. The motion data received by the video distribution device from the posture data acquisition device is immediately transmitted.
The video playback device according to claim 5. Whether or not the user's posture matches the reference posture is determined in the video distribution device based on an evaluation function using the posture feature amount as a variable.
The video playback device according to any one of claims 1 to 6. The determination of whether the user's posture matches the reference posture is performed by a classifier that classifies the user's posture based on the posture feature amount.
The video playback device according to any one of claims 1 to 7. A video playback method performed by one or more computer processors executing computer-readable instructions, the method comprising:
a step of receiving motion data including posture feature amounts representing characteristics of the user's posture from the video distribution device;
generating a video including an animation of the user's avatar generated based on the motion data;
a reference posture that identifies a reference posture that has a smaller amount of data than the posture feature and is determined to match the user's posture while the posture feature cannot be received during transmission of the motion data from the video distribution device; generating a video including a registered animation corresponding to the reference posture based on identification data;
A video playback method comprising: a step of receiving motion data including posture feature quantities representing characteristics of the user's posture from the video distribution device to one or more computer processors;
generating a video including an animation of the user's avatar generated based on the motion data;
a reference posture that identifies a reference posture that has a smaller amount of data than the posture feature and is determined to match the user's posture while the posture feature cannot be received during transmission of the motion data from the video distribution device; generating a video including a registered animation corresponding to the reference posture based on identification data;
A video playback program that runs .