JP2021040305A - Video playback device, video playback method, and video distribution system - Google Patents

Abstract

To provide a video playback device, a video playback method, and a video distribution system that generate an animation of an avatar that moves smoothly by reflecting the movement of each user when a delay occurs in a transmission line in a system that communicates through the avatar that moves according to the movements of the user.SOLUTION: In a video distribution system 1, a video playback device plays back a video received from a video distribution device, transmits motion data including a facial feature amount representing a facial expression feature of a user to the video distribution device, receives a video including an animation of the user's avatar generated from the video distribution device on the basis of the facial feature amount, determines whether the user's facial expression matches a predetermined reference facial expression, and transmits, to the video distribution device, reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression while a delay occurs in transmission of the motion data to the video distribution device.SELECTED DRAWING: Figure 1

Description

The disclosure in this specification mainly relates to a video playback device, a video playback method, and a video distribution system.

A video distribution system is known in which a viewing user who watches a video can participate in the video using an avatar. For example, Japanese Patent Application Laid-Open No. 2012-121998 (Patent Document 1) describes that the avatar of the viewing user is included in the distributed moving image.

Japanese Unexamined Patent Publication No. 2012-121998

In a video distribution service that distributes a video, the engagement of the viewing user can be enhanced by reflecting the feedback from the viewing user in the distributed video. For example, Patent Document 1 describes that a comment from a viewing user is displayed in association with the avatar of the viewing user. In the system of Patent Document 1, the viewing user provides feedback to the video distributed by posting a comment, and this feedback is reflected in the distributed video.

If the distribution content can include not only comments from the viewing user but also an animation of the avatar that moves according to the movement of the viewing user, it is considered that the engagement of the viewing user can be further enhanced. However, in order to move the avatar according to the movement of the viewing user, motion data indicating the movement of the viewing user must be transmitted in real time to the video generation device that generates the video from the video playback device of the viewing user. Is required. In order for the avatar to perform a movement that reflects the user's movement, it is necessary to include bone data indicating the position and orientation of the user's bone in the motion data indicating the user's movement. The position and orientation of the bones are represented by a three-dimensional vector. Bone data of 20 or more bones may be required to express the movement of the human body.

As described above, since the amount of bone data is large, it is difficult to transmit motion data including bone data indicating the posture of the viewing user without delay depending on the band of the transmission line and the degree of congestion. In particular, since the uplink from the user device has a smaller transmission capacity than the downlink used for content distribution, a delay is likely to occur in the transmission of bone data using the uplink. As a result, it is difficult to generate an avatar animation that reflects the operation of the viewing user in real time based on the bone data transmitted from the viewing user, and to include the generated animation in the video being distributed.

Not only when the viewing user participates in the video being delivered as an avatar, but also in a system where users communicate with each other via an avatar that moves according to their own movement, the bone data of each user is transmitted to the device of the other user. There is a need. Therefore, in a system in which users communicate with each other via avatars that move according to their own movements, it is necessary to generate an animation of avatars that move smoothly by reflecting the movements of each user when a delay occurs in the transmission line. Is difficult.

An object of the present disclosure is to provide technical improvements that solve or alleviate at least some of the problems of the prior art described above.

One of the more specific objects of the present invention is to be able to generate an animation of an avatar related to a user's posture based on information of a smaller amount of data than before.

Other objectives of the disclosure herein will become apparent by reference to the entire specification. The invention disclosed in the present specification may solve the problem grasped from the description of the mode for carrying out the invention of the present specification in place of or in addition to the above-mentioned problem. Good.

The moving image playback device according to one aspect includes one or more computer processors, and the one or more computer processors play back a moving image received from the moving image distribution device by executing a computer-readable instruction, and the user Motion data including a posture feature amount representing a posture feature is transmitted to the video distribution device, and a video including the user's avatar animation generated based on the posture feature amount is received from the video distribution device, and the video is received. A classifier that classifies the user's posture based on the posture feature amount determines whether or not the user's posture belongs to a predetermined reference posture, and there is a delay in transmitting the motion data to the video distribution device. While it is being generated, the reference posture determined to belong to the user's posture is identified, and reference posture identification data having a data amount smaller than the posture feature amount is transmitted to the moving image distribution device.

In the moving image playback device according to one aspect, the classifier determines whether or not the posture of the user matches the reference posture based on the evaluation function with the posture feature amount as a variable.

The moving image playback device in one embodiment presents an image representing the reference posture to the user, and learns the posture feature amount representing the posture taken by the user with respect to the presented image as teacher data to classify the video. Create a vessel.

In the moving image playback device in one aspect, one or more registered animations are registered for the user's avatar. The moving image playback device in one aspect receives the moving image including the registered animation corresponding to the reference posture specified based on the reference posture identification data while the delay is occurring.

The moving image playback device in one aspect generates the user animation of the avatar based on the posture feature amount acquired in time series, registers the user animation as the registered animation, and generates a sample moving image including the user animation. Then, the reference posture corresponding to the user animation is determined based on the image of the reference frame selected from the plurality of frames constituting the sample moving image.

The moving image playback device in one embodiment calculates the posture feature amount related to the user at each of the plurality of feature points related to the user based on a predetermined frame rate, and the posture at each of the plurality of feature points in the first frame. The first RMS, which is the root mean square of the feature amount, is calculated, and in the second frame, which is time series after the first frame, the second RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated. Is calculated, and in the third frame after the second frame in chronological order, the root mean square of the posture feature amount at each of the plurality of feature points is calculated, and the second RMS and the second RMS are calculated. When the positive / negative of the first RMS difference, which is the difference from the 1 RMS, and the positive / negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the user is based on the posture feature amount in the third frame. The reference posture corresponding to the animation is determined.

The moving image playback device in one aspect is based on the first evaluation data including the posture feature amount acquired in time series based on the movement of the user with respect to the user animation, and other movements of the user with respect to the user animation. The reference posture corresponding to the user animation is determined by comparing with the second evaluation data including the posture feature amount acquired in time series.

In one aspect, the posture feature includes bone data representing the position and orientation of the user's bones in a three-dimensional vector.

In one aspect, the motion data includes a facial feature amount representing the facial features of the user. The moving image playback device in one embodiment determines whether or not the user's facial expression belongs to a predetermined reference facial expression by another classifier that classifies the user's facial expression based on the facial feature amount, and the delay is caused. While it is being generated, the reference facial expression determined to belong to the user's facial expression is identified, and the reference facial expression identification data having a data amount smaller than the facial feature amount is transmitted to the moving image distribution device.

In one aspect, the reference posture identification data is transmitted in place of the motion data while the delay is occurring.

In one aspect, the motion data is transmitted in real time.

The moving image distribution system according to one aspect includes one or a plurality of computer processors, and distributes a moving image including a user's avatar to the user's moving image playback device. By executing a computer-readable command, the one or more computer processors receive motion data including a posture feature amount representing the posture of the user from the moving image playback device via a transmission line, and the posture feature The user's avatar animation generated based on the amount is included in the moving image and distributed, and the classifier that classifies the user's posture based on the posture feature amount determines that the user's posture belongs to the reference posture. If so, while the transmission of the motion data to the moving image distribution device is delayed, the reference posture is identified and the reference posture identification data having a data amount smaller than the posture feature amount is generated by the moving image playback device. The registered animation of the user's avatar received from the above and generated based on the reference posture identification data is included in the moving image and distributed.

The moving image reproduction method according to one aspect is executed by executing a computer-readable instruction by one or more computer processors. The method includes a step of playing a moving image received from a moving image distribution device, a step of transmitting motion data including a posture feature amount representing a user's posture to the moving image distribution device, and a step of transmitting the motion data from the moving image distribution device to the posture feature amount. The user's posture belongs to a predetermined reference posture by a process of receiving a moving image including an animation of the user's avatar generated based on the above and a classifier that classifies the user's posture based on the posture feature amount. While there is a delay in the step of determining whether or not the motion data is transmitted to the moving image distribution device, the reference posture determined to belong to the user's posture is identified and the posture feature amount is identified. It includes a step of transmitting reference posture identification data of a smaller amount of data to the moving image distribution device.

The moving image playback device according to one aspect of the present invention calculates the posture feature amount related to the user at each of the plurality of feature points related to the user of the moving image playback device based on a predetermined frame rate, and the plurality of features in the first frame. The first RMS, which is the root mean square of the posture features at each of the points, is calculated, and the square of the posture features at each of the plurality of feature points in the second frame after the first frame in time series. The second RMS, which is the root mean square, is calculated, the RMS difference, which is the difference between the second RMS and the first RMS, is calculated, and when the RMS difference is larger than a predetermined threshold value, the second frame with respect to the video distribution device. The motion feature amount in the above is transmitted, a moving image including the animation of the user's avatar generated based on the posture feature amount is received from the moving image distribution device, and the moving image is reproduced.

The moving image playback device according to one aspect of the present invention calculates the posture feature amount related to the user at each of the plurality of feature points related to the user of the moving image playback device based on a predetermined frame rate, and the plurality of features in the first frame. The first RMS, which is the root mean square of the posture features at each of the points, is calculated, and the square of the posture features at each of the plurality of feature points in the second frame after the first frame in time series. The root mean square, which is the root mean square, is calculated, and in the third frame, which is time-series after the second frame, the root mean square, which is the root mean square of the posture features at each of the plurality of feature points, is calculated. When the positive / negative of the first RMS difference, which is the difference between the second RMS and the first RMS, and the positive / negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the posture feature in the third frame. The amount is transmitted, the moving image including the animation of the user's avatar generated based on the posture feature amount is received from the moving image distribution device, and the moving image is reproduced.

According to the embodiment, it is possible to generate an animation of an avatar related to a user's posture based on information of a smaller amount of data than before.

It is a block diagram which shows the moving image distribution system by one Embodiment. It is a figure which shows the 3D skeleton model conceptually. It is a figure explaining the reference posture management data stored in the moving image distribution system of FIG. It is a figure which conceptually shows the 3D skeleton model corresponding to the starting posture. It is a figure which conceptually shows the 3D skeleton model corresponding to a trigger posture. It is a figure explaining the avatar data stored in the moving image distribution system of FIG. It is a figure explaining the animation management data stored in the moving image distribution system of FIG. It is a figure which shows typically the frame which constitutes the captured image. It is a table which shows the feature quantity at a plurality of feature points in each frame, and an example of the RMS. It is a figure which shows the example of the moving image which is displayed on the moving image reproduction apparatus 10 in one Embodiment. It is a figure which shows the example of the moving image which is displayed on the moving image reproduction apparatus 10 in one Embodiment. It is a flow chart which shows a part flow of the process in the moving image reproduction method in one Embodiment. It is a block diagram which shows the moving image distribution system by another embodiment. It is a figure explaining the reference facial expression management data stored in the moving image distribution system shown in FIG. It is a figure explaining the registered facial expression management data stored in the moving image distribution system shown in FIG. It is a block diagram which shows the moving image distribution system by another embodiment. It is a block diagram which shows the moving image distribution system by another embodiment. It is a table which shows the example of the feature quantity at a plurality of feature points in each frame.

Hereinafter, various embodiments of the present invention will be described with reference to the drawings as appropriate. The same or similar components are designated by the same reference numerals in a plurality of drawings.

A moving image distribution system according to an embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing a moving image distribution system 1 according to an embodiment. FIGS. 2, 4a, and 4b are diagrams conceptually showing a three-dimensional skeleton model, and FIGS. 3, 5, and 4 b. FIG. 6 is a diagram for explaining information stored in the moving image distribution system 1.

The video distribution system 1 includes a video playback device 10 and a video distribution device 20. The video playback device 10 and the video distribution device 20 are connected to each other so as to be able to communicate with each other via the network 50. The video distribution system 1 may include a storage 60. A viewing user who is a user of the moving image playback device 10 can view the moving image distributed from the moving image distribution device 20 by the moving image playback device 10. The video distribution system 1 may include two or more video playback devices. A viewing user who watches a moving image distributed from the moving image distribution device 20 on the moving image playback device 10 can display his / her own avatar on the moving image. In other words, the viewing user can participate in the video being delivered via his or her own avatar. The viewing user can display his / her own avatar on the distributed video and interact with the distribution user (or the character in the distributed video) or another viewing user through the avatar.

First, the moving image reproducing device 10 will be described. The moving image playback device 10 is an information processing device such as a smartphone. In addition to smartphones, the moving image playback device 10 may be a mobile phone, a tablet terminal, a personal computer, an electronic book reader, a wearable computer, a game console, and various information processing devices capable of playing back moving images other than these.

The moving image playback device 10 is used by a viewing user for viewing a moving image and for other purposes. The moving image reproduction device 10 includes a computer processor 11, a communication I / F 12, a storage 13 for storing various information, a display 14 for displaying the reproduced moving image, and a sensor unit 15. The moving image distribution device 10 may include components other than the above, such as a sound collecting microphone. The moving image playback device 10 reproduces the moving image distributed from the moving image distribution device 20.

The computer processor 11 is an arithmetic unit that loads various programs that realize an operating system and various functions from the storage 13 or other storage into a memory and executes instructions included in the loaded programs. The computer processor 11 is, for example, a CPU, an MPU, a DSP, a GPU, various arithmetic units other than these, or a combination thereof. The computer processor 11 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, or MCU. Although the computer processor 11 is shown as a single component in FIG. 1, the computer processor 11 may be a set of a plurality of physically separate computer processors. In the present specification, the program described as being executed by the computer processor 11 or the instructions contained in the program may be executed by a single computer processor or may be executed by a plurality of computer processors in a distributed manner. May be good. Further, the program executed by the computer processor 11 or the instructions included in the program may be executed by a plurality of virtual computer processors. The functions realized by the computer processor 11 will be described later.

The communication I / F12 is implemented as hardware, firmware, communication software such as a TCP / IP driver or PPP driver, or a combination thereof. The moving image playback device 10 can transmit and receive data to and from other devices via the communication I / F12.

The storage 13 is a storage device accessed by the computer processor 11. The storage 13 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various storage devices other than those capable of storing data. Various programs can be stored in the storage 13. At least a part of the program and various data that can be stored in the storage 13 may be stored in a storage (for example, storage 60) that is physically separate from the moving image playback device 10.

The display 14 has a display panel and a touch panel. The display panel is a liquid crystal panel, an organic EL panel, an inorganic EL panel, or any other display panel capable of displaying an image other than these. The touch panel is configured to be able to detect a player's touch operation (contact operation). The touch panel can detect various touch operations such as tapping, double tapping, and dragging of the player. The touch panel may include a capacitive proximity sensor and may be configured to be capable of detecting a player's non-contact operation. The moving image distributed from the moving image distribution device 20 is displayed on the display 14.

The sensor unit 15 includes one or a plurality of sensing devices that detect the posture of the user of the moving image reproducing device 10. The sensing device may include an RGB camera, a depth sensor, and other devices capable of detecting the posture of the user. The sensor unit 15 may include a computer processor. The computer processor of the sensor unit 15 may generate three-dimensional skeleton data representing the posture of the user by analyzing the data acquired by the sensing device. The computer processor provided in the sensor unit 15 may generate three-dimensional skeleton data by executing dedicated software. The three-dimensional skeleton data representing the posture of the user detected by the sensor unit 15 is an example of the "posture feature amount" described in the claims. The sensor unit 15 may generate three-dimensional skeleton data using Light Coding technology, such as Kinect ™ provided by Microsoft Corporation. That is, even if the sensor unit 15 irradiates a target such as a user with a random pattern of infrared rays, acquires the depth of the target by analyzing the image, and generates three-dimensional skeleton data by analyzing the depth. Good. The feature amount representing the user's posture detected by using the Light Coding technique is not limited to the three-dimensional skeleton data, and other feature amounts representing the user's posture can be used.

The three-dimensional skeleton data is data for expressing the human body or a part of the human body with a three-dimensional skeleton model. The three-dimensional skeleton model models the skeleton of the human body or a part of the human body by a plurality of bones corresponding to the axes of the bones of the human body and a plurality of joints connecting each bone. The three-dimensional skeleton model will be further described with reference to FIG. FIG. 2 is an explanatory diagram for explaining the concept of the three-dimensional skeleton model. In the example shown in FIG. 2, the 3D skeletal model 100 includes eight bones and joints connecting these bones. The bones of the 3D skeleton model are represented as line segments connecting two points on the 3D coordinates. Therefore, each bone is represented by three-dimensional vectors V1 to V8 representing effective line segments corresponding to each bone in the three-dimensional space (XYZ space). The three-dimensional skeleton model constituting the three-dimensional skeleton model includes a three-dimensional vector representing each of the bones included in the three-dimensional model. FIG. 2 shows a three-dimensional skeleton model of the upper body of the human body, but the three-dimensional skeleton model may be a model of the entire skeleton of the human body, or a part other than the upper body of the human body ( For example, the arm, lower body, etc.) may be modeled. Further, in the example of FIG. 2, the upper body of the human body is represented by eight bones, but the upper body of the human body may be represented by more or less than eight bones. A 3D skeletal model containing a larger number of bones may include bones representing finger bones. Thereby, the movement of the finger can also be detected.

The information stored in the storage 13 will be described. In the illustrated embodiment, the storage 13 stores various information necessary for using the reference posture management data 13a and other services provided by the moving image distribution device 20. In the video distribution system 1 according to one embodiment, the reference posture of the user is determined in advance. In the video distribution system 1, only one reference posture may be defined, or a plurality of reference postures may be defined. Each of the one or more reference poses may be a set of poses to identify a series of user actions. For example, the reference posture includes a start posture that indicates the posture at the start of a series of movements of the user associated with the reference posture, and a trigger posture that indicates a specific posture after taking the start posture in the series of movements. May be included. At least one of the starting posture and the trigger posture may be plural.

An example of the reference posture management data 13a is shown in FIG. In the example shown in FIG. 3, the reference posture includes a start posture and a trigger posture. Therefore, in the storage 13, the start posture data indicating the start posture included in the reference posture and the trigger posture indicating the trigger posture included in the reference posture are associated with the reference posture identification data for identifying the reference posture. Data and can be stored.

The reference posture identification data is, for example, a reference posture ID for identifying the reference posture. The reference posture ID is, for example, an identification code represented by several bits. The number of bits of the reference posture ID can be determined according to the number of reference postures used in the moving image distribution system 1. The reference posture ID is represented by, for example, data having an amount of information of 10 bits or less. The reference posture ID may be represented by data having an amount of information of 5 bits or less, 4 bits or less, 3 bits or less, and 2 bits or less. Therefore, the amount of the reference posture identification data (reference posture ID) is significantly smaller than that of the three-dimensional skeleton data.

The starting posture data is data indicating the starting posture. The starting posture data is, for example, three-dimensional skeleton data indicating the starting posture. An example of the starting posture is shown in FIG. 4a. The starting posture in the illustrated example is a posture in which the right palm is projected forward so that the right fist is at the same height as the right shoulder. The starting posture data may be three-dimensional skeleton data modeling this starting posture. The starting posture data may include three-dimensional vector data indicating the position and orientation of each bone in the starting posture. The three-dimensional skeleton data showing the starting posture may include eight bones as schematically shown in FIG. 4a. The vectors T1 to T8 representing the eight bones correspond to the bone vectors V1 to V8 included in the three-dimensional skeleton data of the viewing user generated based on the detection data of the sensor unit 15, respectively. When data other than the three-dimensional skeleton data is used as the posture feature amount, the start posture data is data representing the start posture among the used posture feature amounts.

The trigger posture data is data indicating the trigger posture. The trigger attitude data is, for example, three-dimensional skeleton data indicating the trigger attitude. An example of the trigger posture is shown in FIG. 4b. The trigger posture in the illustrated example is a posture in which the right arm is pushed up diagonally forward. The trigger posture data may be three-dimensional skeleton data that models this trigger posture. The three-dimensional skeleton data indicating the trigger posture may include three-dimensional vector data indicating the position and orientation of each bone. The three-dimensional skeleton data showing the trigger posture may include eight bones as schematically shown in FIG. 4b. The eight bone vectors T1 to T8 correspond to the bone vectors V1 to V8 included in the three-dimensional skeleton data of the viewing user generated based on the detection data of the sensor unit 15. When data other than the three-dimensional skeleton data is used as the posture feature amount, the trigger posture data is data representing the trigger posture among the used posture feature amounts.

Next, the moving image distribution device 20 will be described. The video distribution device 20 is, for example, a video distribution server that distributes a video to the video playback device 10 via the downlink of the network 50. The video distribution device 20 includes a computer processor 21, a communication I / F 22, and a storage 23 that stores various information. The video distribution device 20 may include components other than the above, such as a sound collecting microphone.

The computer processor 21 is an arithmetic unit that loads various programs that realize an operating system and various functions from the storage 23 or other storage into a memory and executes instructions included in the loaded programs. The above description of the computer processor 11 also applies to the computer processor 21 as much as possible. The functions realized by the computer processor 21 will be described later.

The communication I / F 22 is implemented as hardware, firmware, communication software such as a TCP / IP driver or PPP driver, or a combination thereof. The video distribution device 20 can transmit / receive data to / from another device via the communication I / F 22.

The storage 23 is a storage device accessed by the computer processor 21. The storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various storage devices other than those capable of storing data. Various programs can be stored in the storage 23. At least a part of the program and various data that can be stored in the storage 23 may be stored in a storage (for example, a storage 60) that is physically separate from the moving image distribution device 20.

The storage 23 can store object data 23a, avatar data 23b, animation management data 23c, and various information necessary for generating and distributing moving images other than the above.

The object data 23a may include asset data for constructing a virtual space constituting a moving image. The object data 23a is data for drawing the background of the virtual space constituting the moving image, data for drawing various objects displayed in the moving image, and various objects displayed in the moving image other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space. In addition to the above, the object data 23a may include data indicating a gift object to be displayed in the moving image based on a display request from the viewing user of the moving image reproducing device 10. Gift objects can include effect objects, regular objects, and decoration objects. The viewing user can purchase the desired gift object. Details of the gift object displayed in the video are described in the specification of Japanese Patent No. 6446154. Also in the moving image distribution system 1 of the present application, the gift object can be displayed in the moving image as described in the specification of Japanese Patent No. 6446154.

An example of avatar data 23b is shown in FIG. As shown in FIG. 5, the avatar data 23b can include avatar identification information of the avatar used by the viewing user in the video distribution system 1 and avatar display information for displaying the avatar in the moving image. .. In other words, the storage 23 may store the avatar display information for displaying the avatar in association with the avatar identification information of the avatar. The avatar identification information is, for example, an avatar ID that identifies the avatar. The user of the moving image playback device 10 can set his / her own avatar in the moving image distribution system 1. In order to manage the avatar for each user, the avatar ID may be stored in the storage 23 in association with the user ID that identifies the user. The avatar is displayed in the moving image as, for example, an image imitating a human or an animal. The avatar display information is information used to display the avatar in the moving image. The avatar information includes, for example, part information showing images of the head, hairstyle, facial parts (eyes, nose, mouth, etc.), body, clothes, accessories, items, and other parts constituting the avatar. Other skin information is included to identify the appearance of the avatar. The user can register his / her own avatar by selecting a favorite part image. The avatar display information may include 2D display information for displaying the avatar in the moving image in 2D and 3D display information for displaying the avatar in the moving image in 3D. The 3D display information includes parts information showing images of parts for displaying avatars in a three-dimensional manner in a moving image, bone data for expressing the movement of avatars in three dimensions, and other avatars in three dimensions. Contains known information used for display.

As described above, the moving image distributed from the moving image distribution device 20 may include an animation of the user's avatar of the moving image reproducing device 10. As will be described later, the animation of a user's avatar may be generated so as to reflect the posture and movement of the user in real time based on the posture feature amount indicating the posture of the user.

A pre-registered registered animation may be adopted as the animation of the avatar. The registered animation is not generated in real time during the video distribution to follow the movement of the viewing user, but is pre-registered or defined before the video is distributed or before the avatar animation needs to be displayed. It is an animation. An example of animation management data 23c for managing registered animation is shown in FIG. As shown in the figure, the animation management data 23c includes reference posture identification data, registered animation identification data (registered animation ID) for identifying the registered animation, and animation definition data for specifying the animation of the avatar. In the storage 23, the registered animation identification data and the animation definition data are stored in association with the reference posture identification data. The reference posture identification data may be the reference posture ID as described above. When a plurality of reference posture IDs are used in the video distribution system 1, a plurality of registered animations may be registered according to the number of the reference posture IDs. One registered animation selected from the plurality of registered animations can be included in the video. The animation definition data may be data that describes bone data indicating the position and orientation of the avatar bones in chronological order.

The registration animation may be generated based on a request from the viewing user. Hereinafter, for convenience of explanation, the registered animation registered based on the request from the viewing user is referred to as "user animation". When the viewing user requests the registration of the user animation, the moving image playback device 10 can urge the viewing user to perform a movement corresponding to the animation desired to be registered. The moving image reproduction device 10 acquires the posture feature amount (for example, three-dimensional skeleton data) of the posture taken by the viewing user in response to the display of the image representing the reference posture over a predetermined time. The posture feature amount is acquired by the posture data acquisition unit 11b or the sensor unit 15 at predetermined sampling time intervals. In this way, animation definition data indicating the movement corresponding to the animation that the viewing user desires to register can be obtained. This animation definition data may have the same data format as the animation definition data of the registered animation stored as the animation management data 23c. The moving image playback device 10 transmits the animation definition data that defines the user animation acquired in this way to the moving image distribution device 20, and registers this user animation in the moving image distribution device 20 as a new registration animation.

When the video distribution device 20 receives the animation definition data that defines the user animation, the video distribution device 20 stores the user animation in the storage 23 as a part of the animation management data 23c. Specifically, the video distribution device 20 issues a registration animation ID that identifies the user animation for which registration is requested, and stores the animation definition data received from the video playback device 10 in association with the registered animation ID. .. Further, the moving image distribution device 20 requests the moving image reproducing device 10 to determine a reference posture that serves as a trigger for including the user animation for which registration is requested in the moving image.

In response to a request from the video distribution device 20, the video playback device 10 determines a reference posture that is a trigger for causing the avatar to perform a movement according to a newly registered user animation. In the present specification, the reference posture that triggers the movement of the avatar according to the newly registered user animation may be referred to as an additional reference posture. The additional reference posture is determined by various methods. For example, a sample video including a newly registered user animation is generated, and one or a plurality of candidate frames are selected from a plurality of frames included in the sample video for a viewing user who has made a registration request for the user animation. The additional reference posture can be determined based on the image of the avatar contained in the selected frame. The viewing user can select a favorite frame from a plurality of frames. The viewing user may select, for example, a frame containing an avatar in a memorable posture, a frame containing an avatar in a characteristic posture, or a frame other than these. it can. When two frames are selected from the plurality of frames, the posture of the avatar included in the frame that is ahead in time series of the two frames is determined as the start posture, and the posture is determined later in time series. The posture of the avatar included in a certain frame may be determined as the trigger posture.

Other methods of determining the additional reference posture will be described. The moving image playback device 10 instructs the viewing user who has requested the registration of the user animation to perform the movement corresponding to the user animation a plurality of times. This instruction may be given by voice or screen display. The moving image playback device 10 acquires a posture feature amount (for example, three-dimensional skeleton data) corresponding to the movement performed by the viewing user in response to this instruction. Specifically, three-dimensional skeleton data representing the posture of the viewing user moving in response to the user animation is acquired at a predetermined sampling interval in a predetermined measurement period. As a result, two sets of three-dimensional skeleton data representing the posture of the viewing user who has performed the movement corresponding to the user animation from the start of the measurement to the end of the measurement can be obtained. Next, of the two sets of three-dimensional skeleton data sets, the three-dimensional skeleton data acquired at the same timing after the start of measurement are compared with each other, and an additional reference posture is determined based on the comparison result. For example, out of two sets of 3D skeleton data, the total value of the angles formed by the corresponding bone vectors of the 3D skeleton data acquired at the same timing after the start of measurement is calculated, and the total value of these angles is the smallest. The posture corresponding to the three-dimensional skeleton data (any of the two sets of skeleton data may be adopted) can be used as the additional reference posture.

Next, the function of the moving image playback device 10 will be described. The function of the moving image playback device 10 is realized by the computer processor 11 executing a computer-readable instruction included in the program. The computer processor 11 functions as a moving image reproduction unit 11a, a posture data acquisition unit 11b, a transmission unit 11c, a classification unit 11d, and a delay monitoring unit 11e by executing a computer-readable instruction included in the program. At least a part of the functions realized by the computer processor 11 may be realized by a computer processor other than the computer processor 11. At least a part of the functions realized by the computer processor 11 may be realized by the computer processor 21 of the moving image distribution device 20 or a computer processor provided in the other moving image distribution system 1.

The moving image reproduction unit 11a reproduces the moving image distributed from the moving image distribution device 20. The reproduced moving image is displayed on the display 14.

The posture data acquisition unit 11b acquires a posture feature amount representing the posture characteristics of a user (sometimes referred to as a “viewing user”) who watches a moving image using the moving image playback device 10. The posture feature amount representing the posture feature of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user, that is, three-dimensional vector data indicating the position and orientation of the bones of the viewing user. The posture data acquisition unit 11b generates, for example, three-dimensional skeleton data of the viewing user based on the detection data detected by the sensor unit 15. The viewing user's three-dimensional skeleton data may be generated at predetermined sampling time intervals. When the sensor unit 15 includes a processor and the three-dimensional skeleton data is generated by the processor of the sensor unit 15, the attitude data acquisition unit 11b does not have to be executed as a function of the processor 11.

The posture feature amount acquired by the posture data acquisition unit 11b may be a feature amount other than the three-dimensional skeleton data. For example, as described above, a random pattern of infrared rays may be applied to a target such as a user by using the Light Coding technique, and the depth of the target may be obtained by analyzing the image, and the depth may be used as a posture feature amount. The detected value detected by the sensor unit 15 and various values calculated based on the detected value can be used as the posture feature amount.

The posture data acquisition unit 11b may obtain the posture feature amount as follows from the captured image including a plurality of frames obtained by capturing the movement of the user of the moving image reproduction device 10 at a predetermined frame rate. Specifically, the posture data acquisition unit 11b extracts a plurality of feature points related to the user in each frame of the captured image. Positions suitable for expressing the user's posture and movement are extracted as feature points. A plurality of feature points may be referred to as a feature point group. FIG. 7 schematically shows a frame f1 which is one of a plurality of frames constituting the captured image captured by the moving image reproducing device 10. As shown, the frame f1 includes a captured user image U1. P1 to P6 shown in FIG. 7 indicate the extracted feature points, respectively. The number and position of feature points are not limited to those shown in FIG. For example, when a random dot pattern is projected onto the body or face of the user of the moving image playback device 10 by an infrared laser and the user on which the random dot pattern is projected is imaged with a camera capable of photographing an infrared region, the random dots The whole becomes a feature point group.

The posture data acquisition unit 11b can obtain a posture feature amount (image vector) for each of the extracted plurality of feature points. The posture feature amount at each of the feature points P1 to P6 may be the depth of each feature point or the three-dimensional coordinates. When three-dimensional coordinates are used as the posture feature amount, a three-dimensional world coordinate system is set in the captured image, and the relative amount normalized based on the captured image and the world coordinate system is used as the posture feature amount. To. The normalized posture feature quantity is represented by, for example, a relative quantity of 0 to 1.0. The posture features of the feature points P1 to P6 in the frame f1 are obtained in the posture features of the feature points P1 to P6 obtained in the frame before (for example, immediately before) the frame f1 in time series and in the frame f1. It may be the amount of change represented by the difference between each feature point P1 to P6 and the posture feature amount. When it is necessary to distinguish between the two when the amount of change in the posture feature amount between frames is used as another posture feature amount, the posture feature amount in one frame is called "in-frame feature amount", and the posture between frames is called. The posture feature amount represented by the change amount of the feature amount may be called "inter-frame feature amount". Unless otherwise specified, or unless otherwise understood in the context, the term "postural feature" includes both "in-frame features" and "interframe features". The posture features of each feature point P1 to P6 may be normalized to the range of 0 to 1.0 and expressed as a float array. In this case, the posture feature amount at each of the feature points P1 to P6 is an element of the array.

The transmission unit 11c transmits the posture feature amount representing the posture of the viewing user acquired by the posture data acquisition unit 11b or the sensor unit 15 to the moving image distribution device 20. Upon receiving the posture feature amount from the posture data acquisition unit 11b or the sensor unit 15, the transmission unit 11c immediately transmits the posture feature amount to the moving image distribution device 20. In other words, the transmission unit 11c can transmit the posture feature amount of the viewing user to the moving image distribution device 20 in real time. As described above, the posture feature amount is acquired at a predetermined sampling time interval or a predetermined frame rate. Thereby, the posture feature amount can be generated at each predetermined sampling time interval or at each frame rate. Therefore, the posture feature amount of the viewing user continuously acquired over a predetermined time interval can express the body movement of the viewing user as digital data in chronological order. The posture feature amount representing the posture of the viewing user may be transmitted to the moving image distribution device 20 together with data other than the posture feature amount. In the present specification, data representing the posture and facial expression of the viewing user or data correlating with the posture and facial expression of the viewing user may be collectively referred to as "motion data" of the viewing user. The motion data of the viewing user is transmitted from the moving image playback device 10 to the moving image distribution device 20 via a transmission line including the network 50. The motion data may be packet-transmitted. That is, the transmission unit 11c may transmit a packet containing motion data to the moving image distribution device 20.

The classification unit 11d in one embodiment uses a classifier that classifies the posture of the viewing user, and the posture of the viewing user is determined in advance based on the posture feature amount (for example, three-dimensional skeleton data) of the viewing user. Determine if it belongs to the posture. This classifier is, for example, a linear classifier. When the reference posture includes the start posture and the trigger posture, the classification unit 11d can determine whether or not the posture of the viewing user belongs to the reference posture and whether or not the posture belongs to the trigger posture, respectively. After determining whether or not the posture of the viewing user belongs to the start posture of the reference postures at the time t1, the classification unit 11d determines that the posture of the user at the time t1 belongs to the start posture. Subsequently, it may be determined whether or not the posture of the viewing user at time t2 after time t1 belongs to the trigger posture.

In one embodiment, the classification unit 11d displays an image representing the reference posture on the display 14 in order to obtain teacher data, and displays a voice or a screen to the viewing user so as to take a posture according to the image representing the reference posture. Can be instructed through. The moving image playback device 10 sequentially acquires the posture feature amount of the movement performed by the viewing user in response to this instruction by the posture data acquisition unit 11b or the sensor unit 15. The classification unit 11d can create a classifier by learning the posture feature amount of the posture taken by the viewing user as teacher data in response to the image representing the reference posture.

The classification unit 11d in one embodiment determines, for example, whether or not the posture of the viewing user indicated by the posture feature amount from the posture data acquisition unit 11b or the sensor unit 15 matches a predetermined reference posture. Can be determined based on the evaluation function with. When it is determined that the posture of the viewing user matches the reference posture based on the evaluation function, the posture of the viewing user belongs to the reference posture. Whether or not the posture of the viewing user matches the starting posture is determined by the posture feature amount representing the posture of the viewing user and the posture feature amount representing the starting posture (starting posture data) generated by the posture data acquisition unit 11b or the sensor unit 15. Can be determined based on. When the posture feature amount is the three-dimensional skeleton data, the vectors V1 to V8 of each bone included in the three-dimensional skeleton data of the viewing user and the vectors T1 to each bone corresponding to the vectors V1 to V8 in the start posture data. The smaller the total value of the angles formed by T8, the higher the similarity between the posture of the viewing user detected by the sensor unit 15 and the starting posture. That is, there is a negative correlation between the magnitude of the angle θ formed by the 3D vector included in the 3D skeleton data representing the posture of the viewing user and the corresponding vector of the 3D skeleton model representing the reference posture and the similarity of the posture. There is. Therefore, paying attention to this point, a new evaluation function f for normalizing the magnitude of the angle θ defined by the two three-dimensional skeleton models is defined. An example of the evaluation function f is shown in the following equation (1).

In the above equation, (Ti · Vi) indicates the inner product of the corresponding vectors, (‖Ti‖‖Vi‖) indicates the product of the magnitudes of each vector, and k is the vector constituting the three-dimensional skeleton model. Indicates the number of elements of. That is, according to the evaluation function f shown in the above equation (1), the average value of the cosine (= cos θ) of the angle θ formed by the corresponding vectors of the two three-dimensional skeleton models can be obtained.
Here, when the corresponding vectors of the two 3D skeleton models exactly match (angle θ = 0), cos θ = 1, and the directions of the corresponding vectors of the two 3D skeleton models are opposite. Since (angle θ = 180 °) is cos θ = -1, the range in which the value of the evaluation function f can be taken is -1 ≦ f ≦ + 1. In this case, for example, if the range of values of the evaluation function f (-1 ≦ f ≦ + 1) is linearly assigned to the percentage value (0 to 100%), the similarity can be expressed as a percentage. This predetermined threshold can be, for example, 90%. The determination of the degree of similarity between the two three-dimensional skeleton data is also disclosed in Japanese Patent Application Laid-Open No. 2013-37454.

The classification unit 11d sets the posture of the viewing user to the starting posture when the similarity between the posture feature amount representing the posture of the viewing user calculated as described above and the posture feature amount representing the starting posture is equal to or more than a predetermined threshold value. It may be determined that they match.

Even if the classification unit 11d determines whether or not the posture of the viewing user indicated by the posture feature amount belongs to the starting posture each time the posture data acquisition unit 11b or the sensor unit 15 acquires the posture feature amount of the viewing user. Good. In this case, it is determined whether or not the posture of the viewing user matches the starting posture at the same frequency as the sampling rate for acquiring the posture feature amount of the viewing user.

Whether or not the posture of the viewing user and the trigger posture match can be determined in the same manner. That is, the classification unit 11d calculates the degree of similarity between the posture feature amount representing the posture of the viewing user and the posture feature amount representing the trigger posture at time t2, and when this similarity is equal to or greater than a predetermined threshold value, the viewing user It may be determined that the posture matches the trigger posture. The above description for determining whether or not the posture of the viewing user matches the start posture also applies as much as possible to the process for determining whether or not the posture of the viewing user matches the trigger posture.

The delay monitoring unit 11e monitors the transmission delay of the motion data transmitted from the moving image playback device 10 to the moving image distribution device 20 by the transmitting unit 11c. For example, the delay monitoring unit 11e adds a time stamp to an actual packet containing motion data before transmission, and adds a type stamp added at the time of transmission and when the actual packet is received by the video distribution device 20. It is possible to obtain the transmission time of the actual packet on the transmission line between the moving image reproduction device 10 and the moving image distribution device 20 by using the time stamp. The delay monitoring unit 11e can determine that a delay has occurred in the transmission line when the transmission time exceeds a predetermined reference time or is longer than the reference time. After determining that a delay has occurred once in the transmission line, the delay monitoring unit 11e eliminates the delay when the transmission time becomes shorter than the predetermined reference time or becomes less than or equal to the predetermined reference time. Can be determined. When the video distribution device 20 is a server and the video playback device 10 is a client, data transmission from the video playback device 10 to the video distribution device 20 is performed using an uplink. In this case, the delay monitoring unit 11e monitors whether or not a delay has occurred in the uplink of the transmission line between the moving image playback device 10 and the moving image distribution device 20. The delay monitoring unit 11e may measure the transmission time of the actual packet as described above, or may measure the transmission time using a pseudo packet that does not include motion data. While there is a delay in the transmission of motion data transmitted from the video playback device 10 of one viewing user, there is a delay in the transmission of motion data transmitted from the video playback device 10 of another viewing user. It may be determined that there is no such thing. Therefore, the delay in the transmission line may be determined for each viewing user. Further, the presence or absence of transmission delay may be determined by the moving image distribution device 20 based on the time stamp included in the packet. The moving image playback device 10 may receive the determination result of the transmission delay in the moving image distribution device 20.

The function of the transmission unit 11c will be described again. When the delay monitoring unit 11e determines that a delay has occurred in the transmission line, the transmission unit 11c can transmit the reference posture identification data (reference posture ID) stored in the storage 13. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As described above, the reference attitude identification data may include the start attitude data and the trigger attitude data. The transmission unit 11c can transmit the trigger posture data as the reference posture identification data. In one embodiment, when the classification unit 11d determines that the posture of the viewing user belongs to the trigger posture, the transmission unit 11c uses the trigger posture as the reference posture identification data while the transmission of the motion data is delayed. The identification reference posture ID is transmitted to the moving image distribution device 20. When the transmission unit 11c determines that the viewing user's posture belongs to the trigger posture within a predetermined interval after the classification unit 11d determines that the viewing user's posture belongs to the start posture, the transmission unit 11c causes the motion. While the data transmission is delayed, the reference posture ID that identifies the trigger posture is transmitted to the moving image distribution device 20. By transmitting the reference posture ID when the posture of the viewing user is determined to belong to the trigger posture after the posture of the viewing user is determined to belong to the start posture, the posture belongs to only one of the start posture and the trigger posture. It is possible to prevent or suppress the viewing user from unintentionally transmitting the reference posture ID as compared with the case where the reference posture ID is transmitted in response to the determination. “While a delay is occurring in the transmission of motion data” means a period from when the delay monitoring unit 11e determines that a delay has occurred in the transmission line to when it is determined that the delay has been eliminated. You may. The transmission unit 11c does not have to transmit the reference posture ID that identifies the start posture to the moving image distribution device 20.

Next, the functions realized by the computer processor 21 will be described more specifically. The computer processor 21 functions as a moving image generation unit 21a, a moving image distribution unit 21b, and an animation generation unit 21c by executing a computer-readable instruction included in the distribution program. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the moving image distribution system 1. At least a part of the functions realized by the computer processor 21 may be realized by, for example, the computer processor 11 of the moving image reproducing device 10 or a computer processor provided in the other moving image distribution system 1.

The video distribution device 20 can distribute various types of moving images. In the following, it is assumed that the moving image distribution device 20 distributes a moving image including an animation of a character object generated based on the movement of an actor.

The moving image generation unit 21a detects the movement of the actor by a motion sensor mounted on the actor. The moving image generation unit 21a can generate an animation of a character that moves in synchronization with the body movement of the actor detected by the motion sensor. The moving image generation unit 21a may acquire face motion data which is a digital representation of the face movement of the actor. In this case, the moving image generation unit 21a can generate an animation of a character whose facial expression changes in synchronization with the movement of the actor's face. The moving image generation unit 21a can construct a virtual space using the object data 23a, and can generate a moving image including the virtual space and the animation of the character corresponding to the actor. The moving image generation unit 21a can synthesize the voice of the actor acquired from the microphone with the generated moving image. The generation of moving images, including animations of characters that move in synchronization with the movements of the actor's body and facial expressions, is disclosed in detail in the specification of Japanese Patent No. 644615.

When the moving image generation unit 21a receives a participation request for the avatar to participate in the moving image from the viewing user who is watching the moving image, the moving image generation unit 21a can generate the moving image so as to include the avatar of the viewing user. The participation request from the viewing user may include a user ID that identifies the viewing user. The moving image generation unit 21a identifies the avatar ID that identifies the avatar of the viewing user based on the user ID included in the participation request, and is based on the part information stored in association with the avatar ID in the storage 23. It is possible to generate an avatar object that represents the avatar of the viewing user.

The video distribution unit 21b distributes the video generated by the video generation unit 21a. This moving image is distributed to the moving image reproducing device 10 via the network 50. The received moving image is played back in the moving image reproducing device 10.

FIG. 9a shows a display example of a moving image distributed from the moving image distribution device 20 to the moving image reproducing device 10a and reproduced by the moving image reproducing device 10. The moving image reproduced by the moving image reproducing device 10 may be displayed on the display 14. As shown in the figure, the moving image 60 displayed on the moving image reproducing device 10 includes a character object 51 representing an actor's character, a floor object 54a on which the character object 51 stands, and a screen object 54b defining the rear end of the stage. The avatar objects 56a to 56j indicating the avatar of the viewing user who is viewing the moving image 60 are included. The character object 51 can move in the virtual space in synchronization with the movement of the actor's body. Since the moving image 60 is distributed from the moving image distribution device 20 to a large number of moving image playback devices 10, the image 60 contains a large number of avatars. In FIG. 9a, it is assumed that 10 viewing users are participating in the moving image via the avatar. The number of viewing users who can participate in the video may be more than 10, or at least 10 or less.

The display 14 may display an operation button for accepting a user's operation so as to overlap the moving image 60. In the example shown in FIG. 9a, a gift button 61 for performing gifting, an evaluation button 62 for providing an evaluation, an avatar participation button 63 for applying for participation in the video 60 by the avatar, and the like. Is superimposed and displayed on the moving image 60. The gift button 61, the evaluation button 62, and the avatar participation button 63 are displayed so as to be selectable by the viewing user. Operation buttons other than these may be displayed on the moving image 60. The viewing user who is watching the video 60 gives a desired gift to the distributor who distributes the video 60 or the actor who appears in the video 60 via the character 51 by selecting the gift button 61. Can be done. By selecting the evaluation button 62, the viewing user can transmit the evaluation information indicating that the moving image 60 has been positively evaluated to the moving image distribution device 20. Evaluation information from various viewing users may be aggregated and the aggregated result may be displayed together with the moving image 60. By selecting the avatar participation button 63, the viewing user can send a participation request requesting that his / her own avatar participate in the moving image 60 to the moving image distribution device 20.

The animation generation unit 21c generates an animation of the viewer's avatar participating in the moving image 60. In one embodiment, the animation generation unit 21c generates an animation of the viewing user's avatar based on the viewing user's posture feature amount received from the moving image playback device 10. The posture feature amount (for example, three-dimensional skeleton data) received continuously in time represents the movement of the body of the viewing user in chronological order. Therefore, the animation generation unit 21c continuously receives the posture feature amount of the viewing user from the moving image playback device 10, and the avatar moves in synchronization with the body movement of the viewing user based on the posture feature amount. Animations can be generated. When the animation generation unit 21c generates the animation of the avatar, the moving image generation unit 21a generates the moving image 60 so as to include the animation of the avatar.

If a transmission delay occurs in the upstream transmission line from the video playback device 10 to the video distribution device 20, the video distribution device 20 may not be able to continuously receive the posture feature amount of the viewing user in time. is there. If an attempt is made to create an avatar animation based only on the posture feature amount of the viewing user transmitted from the video playback device 10, and a transmission delay occurs in the transmission line, the avatar animation that moves to reflect the movement of the viewing user is generated. It may not be possible to generate it. On the other hand, the animation generation unit 21c in one embodiment does not depend only on the posture feature amount of the viewing user, but animates the avatar based on the reference posture identification data (for example, the reference posture ID) from the video playback device 10. Can be generated. Specifically, when the animation generation unit 21c receives the reference posture identification data from the moving image playback device 10, the animation generation unit 21c identifies the registered animation associated with the received reference posture identification data by referring to the animation management data 23c. It is possible to generate an avatar animation based on the animation definition data associated with the registered animation ID of the specified registered animation. As described above, the reference posture identification data is obtained from the video playback device 10 when the delay monitoring unit 11e of the video playback device 10 determines that there is a delay in the transmission of the motion data including the posture feature amount on the uplink. It is transmitted to the video distribution device 20. Therefore, even if the posture feature amount cannot be received while the motion data transmission is delayed, the animation generation unit 21c is sent from the moving image playback device 10 while the delay is occurring. An avatar animation can be generated based on the reference posture identification data.

The generation of avatar animation will be described. The avatar animation generated based on the posture feature amount of the viewing user (for example, three-dimensional skeleton data) expresses the movement of the avatar reflecting the movement of the viewing user. For example, it is assumed that the viewing user performs an operation of lifting the right hand diagonally upward from the position where the right hand is placed on the waist. In this case, the animation of the avatar of the viewing user is generated based on the posture feature amount of the viewing user. By including this animation in the video, the avatar performs the action of lifting the right hand (the part corresponding to the right hand of the avatar) diagonally upward from the position where the right hand (the part corresponding to the right hand of the avatar) is placed on the waist in the video, similar to the movement of the viewing user. ..

On the other hand, if there is a delay in the transmission line, the generation of the avatar animation based on the attitude features may be interrupted. For example, when a delay occurs in the transmission line, the transmission of the posture feature amount by the transmission unit 11c may be interrupted, and the animation generation unit 21c may generate an avatar animation based on the posture feature amount accordingly. May also be interrupted. As described above, the moving image distribution device 20 can determine the presence or absence of a delay based on the time stamp included in the packet received from the moving image playing device 10. When a delay occurs in the transmission line, it becomes difficult for the moving image distribution device 20 to receive the posture feature amount having a large amount of data in a timely manner. Even in such a case, if the animation of the avatar based on the posture feature amount is continued, the movement of the avatar in the moving image may become unnatural. Such unnatural movement of the avatar is not desirable because it deteriorates the quality of the delivered video. When a delay occurs in the transmission line, it is possible to prevent the avatar from making an unnatural movement by interrupting the generation of the avatar animation based on the posture feature amount. Even if the generation of the avatar animation based on the posture feature amount is interrupted, the animation generation unit 21c identifies the avatar registration animation ID based on the reference posture identification data from the moving image playback device 10, and the specified registration animation. An avatar animation can be generated based on the animation definition data associated with the ID. If the viewing user does not take a posture that matches the reference posture while the delay is occurring in the transmission line, the moving image distribution device 20 does not receive the reference posture identification data. In this case, the animation generation unit 21c does not have to generate the animation of the avatar of the viewing user. When the animation generation unit 21c does not generate an animation of the avatar, the avatar is stationary in the moving image. In another embodiment, when the moving image distribution device 20 does not receive the reference posture identification data while the delay is occurring in the transmission line, the movement is performed according to the basic operation defined for the avatar. You may create an animation of the avatar. The basic movement of the avatar refers to a predetermined movement such as shaking the hand up and down, shaking the hand left and right, and jumping. This basic operation may be set in common to a plurality of viewing users. The basic operation of the avatar is generated by the animation generation unit 21c in that it can be performed by the avatar without receiving information (three-dimensional skeleton data and reference posture identification data) regarding the movement of the avatar from the moving image playback device 10. It's different from animation.

When the animation of the avatar is generated by the animation generation unit 21c, the moving image generation unit 21a generates a moving image including the animation, and the moving image distribution unit 21b distributes the moving image including the animation of the avatar to the moving image reproducing device 10. When the moving image 60 shown in FIG. 9a includes the animation of the avatar object 56a, the moving image 60 including the animation of the avatar object 56a is displayed. For example, when an animation in which the avatar object 56a raises the right hand is generated, the avatar object 56a moves to raise the right hand in the moving image 60 as shown in FIG. 9b.

As described above, the video distribution device 20 may request the video playback device 10 to determine an additional reference posture that triggers the inclusion of the user animation for which registration is required in the video. Although some aspects of the method for determining the additional reference posture are as described above, another aspect for determining the additional reference posture will be described with reference to FIG. In the following example, the feature points of the user's image are extracted, and the additional reference posture is determined by using the root mean square of the posture feature amount at the feature points. First, the moving image playback device 10 instructs the user to perform the movement to be registered as a user animation, and images a user performing the movement to be registered based on the instruction at a predetermined frame rate to capture a plurality of frames. Obtain a captured image including. The moving image playback device 10 obtains posture feature quantities for each of the extracted plurality of feature points. Since the extraction of the feature points in each of the plurality of frames of the captured image captured by the user has already been described with reference to FIG. 7, no repetitive description will be given.

FIG. 8 shows a table including posture features in each of the 24 frames from the 0th frame to the 23rd frame. In the table of FIG. 8, each posture feature amount of the feature points P1 to P6 is shown in the "P1" column to the "P6" column, respectively. This posture feature amount is normalized to the range of 0 to 1.0. The moving image playback device 10 calculates the root mean square (RMS) of the posture feature amounts of the feature points P1 to P6 for each frame. The orientation feature quantity at each of the N feature points P ₁ ~P _N x _i (where, i takes values of 1 to N.) To the, the posture feature quantity of the feature point P1~P _N RMS (x ) Is expressed by the following formula.

FIG. 8 shows the RMS (x) calculated for each frame. In addition to the RMS (x), the moving image playback device 10 shows the average value of the posture feature amount at each feature point and the difference between the RMS (x) of the frame and the RMS (x) of the previous frame for each frame. The RMS difference may be calculated. The moving image playback device 10 may obtain a positive / negative flag indicating the positive / negative of the RMS difference and an inversion flag indicating whether or not the positive / negative of the RMS difference is reversed for each frame. For example, the RMS difference in the second frame is the difference between the RMS (x ₂ ) in the second frame and the RMS (x ₁ ) in the first frame. The positive / negative flag is, for example, "1" when the RMS difference is positive.
It becomes "0" when the RMS difference is negative. The inversion flag is, for example, "1" when the sign of the RMS difference is reversed, and is "0" when the sign of the RMS difference is not reversed (that is, the sign is the same as that of all frames). FIG. 8 includes the mean value, the RMS difference, the positive / negative flag, and the inversion flag. In FIG. 8, since the positive and negative of the RMS difference with the immediately preceding frame are reversed in the 4th frame, the 10th frame, the 16th frame, the 17th frame, the 19th frame, and the 21st frame, the inversion flag is set to ". 1 "is set. Since there is no frame preceding the 0th frame, the RMS difference is blank for the 0th frame.

In the frame in which the inversion flag is "1", the sign of the RMS difference is reversed from the previous frame, so it is presumed that the user has made a large movement in the frame. The large movements that can be detected by the inversion flag are often periodic movements, such as waving or blinking. Therefore, in one embodiment of the present invention, the frame set to "1" of the inversion flag, that is, the frame in which the sign of the RMS difference of the posture feature amount is reversed from the previous frame is set as the start frame, and the RMS is started from this start frame. It becomes easy to pay attention to the interval until the sign of the difference is inverted again (then until the inversion flag becomes "1"). The posture of the user in this start frame can be set as the start posture. Further, the frame in which the inversion flag is set to "1" next to the start frame can be set as the trigger frame, and the posture of the user in this trigger frame can be set as the trigger posture. In this way, the user's movement and posture can be analyzed by focusing on the section from when the inversion flag becomes "1" to when the inversion flag becomes "1". In the example shown in FIG. 8, the inversion flag is set to "1" for the first time in the fourth frame, and the inversion flag is set to "1" for the second time in the tenth frame. Therefore, it is possible to extract a periodic operation with the user's posture in the 4th frame as the start posture and the user's posture in the 10th frame as the trigger posture and the trigger posture at a low calculation cost. The start posture data indicating the start posture and the trigger posture data indicating the trigger posture are calculated from the root mean square error (RMSE), the root mean square error rate (RMSPTE), or the captured image. Or can be estimated.

In the example shown in FIG. 8, the inversion flag is set to "1" in both the 16th frame and the 17th frame. Since the time interval between consecutive frames is extremely short, it may not be appropriate to set the user's posture in each of these adjacent frames as the start posture and the trigger posture. Therefore, the lower limit number of frames, which is the lower limit of the number of frames that should exist between the start frame and the trigger frame, may be set. The lower limit number of frames may be, for example, 3 frames or more in consideration of the compression of communication packets, the difference from neighboring frames, and other factors. A lower limit of the time interval between the start frame and the trigger frame may be set. The lower limit of the time interval between the start frame and the trigger frame is, for example, 1 second or more, 2 seconds or more, 3 seconds or more, 4 in consideration of the operating frequency, the operation of the target user, and other factors. It may be a lower limit of seconds or more, 5 seconds or more, or other than these. Assuming that the lower limit frame number is 3, there are 5 frames between the 4th frame and the 10th frame that exceed the lower limit frame number in FIG. 8, so that the 10th frame is treated as a trigger frame. Can be done. If the inversion flag is set to "1" in the 7th frame, the 7th frame is not used as the trigger frame because there are only 2 frames less than the lower limit number of frames between the 4th frame and the 7th frame. Next, the tenth frame in which the inversion flag becomes "1" can be used as the trigger frame. However, since the RMSPE from the 10th frame to the 16th frame to be the next target frame is approximately 1 or less, it is not necessary to determine the 10th frame as noise and process it as a trigger frame.

As described above, by determining the additional reference postures (start posture and trigger posture) using the root mean square of the posture features at the user's feature points, the user animation can be registered without presenting candidate frames or sample videos. In addition, the start posture and the trigger posture corresponding to the user animation can be determined.

Next, with reference to FIG. 10, a part of the processing included in the moving image reproduction method according to one aspect will be described. FIG. 10 is a flow chart showing a partial flow of processing in the moving image reproduction method according to the embodiment. In the moving image reproduction method of FIG. 10, it is assumed that the viewing user is playing the moving image by the moving image reproducing device 10 and has his / her own avatar participate in the moving image. That is, the video being viewed by the viewing user includes its own avatar. The process shown in FIG. 10 relates to the transmission of the posture feature amount and the reference posture identification data to the video distribution device 20 performed during viewing of the video.

First, in step S11, data regarding the posture feature amount of the viewing user who is viewing the moving image is acquired. Further, in step S11, the acquired posture feature amount is immediately transmitted to the moving image distribution device 20. In step S11, motion data including the posture feature amount and other data may be transmitted to the moving image distribution device 20. The posture feature amount of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user. Posture features may be acquired at predetermined sampling time intervals. The posture feature amount of the viewing user is acquired by, for example, the posture data acquisition unit 11b or the sensor unit 15. The posture feature amount is transmitted by, for example, the transmission unit 11c described above.

Next, in step S12, it is determined whether or not the posture of the viewing user belongs to a predetermined reference posture based on the posture feature amount of the viewing user acquired in step S11. This determination is made, for example, by the classification unit 11d described above. If the posture of the viewing user does not belong to the reference posture, the process returns to step S11, and the acquisition of the posture feature amount is continued in step S11. If it is determined that the posture of the viewing user belongs to any of the reference postures, the process proceeds to step S13.

In step S13, it is determined whether or not there is a delay in the transmission of motion data in the transmission line between the moving image reproduction device 10 and the moving image distribution device 20. This determination is performed, for example, by the delay monitoring unit 11e described above. If it is determined that no delay has occurred in the transmission line, the process returns to step S11, and the acquisition of the posture feature amount is continued in step S11. If it is determined that a delay has occurred in the transmission line, the process proceeds to step S14.

In step S14, the reference posture identification data for identifying the reference posture determined to belong to the posture of the viewing user in step S12 is transmitted to the moving image distribution device 20. The reference posture identification data is transmitted by, for example, the transmission unit 11c described above. If it is determined in step S12 that the posture of the viewing user belongs to the start posture and then further belongs to the trigger posture, the trigger posture data that identifies this trigger posture is transmitted.

In parallel with the processes in each of the above steps S11 to S14, the moving image is continuously distributed from the moving image distribution device 20 to the moving image reproducing device 10. During the distribution of the moving image, the processes of steps S11 to S14 are repeated. When it is determined in step S13 that no delay has occurred in the transmission line, the delivered moving image includes an avatar animation generated based on the posture feature amount of the viewing user. On the contrary, when it is determined in step S13 that a delay has occurred in the transmission line, the avatar specified based on the reference posture identification data transmitted to the video distribution device 20 is registered in the distributed video. Animation is included.

Step S13 may be executed before step S11 or between step S11 and step S12. In addition to this, the order of processing shown in FIG. 10 may be changed as appropriate. Further, in addition to the processes of steps S11 to S14, processes not explicitly shown in FIG. 10 may be performed.

Next, other embodiments of the present invention will be described with reference to FIGS. 11 to 13. FIG. 11 shows a block diagram of the moving image distribution system 101 according to another embodiment of the present invention. The video distribution system 101 includes a video playback device 110 and a video distribution device 120. The video playback device 110 of the video distribution system 101 is different from the video playback device 10 of the video distribution system 1 in that the computer processor 11 functions as the facial feature amount acquisition unit 11f and the storage 13 stores the reference facial expression management data 13b. different. The video distribution device 120 of the video distribution system 101 is different from the video distribution device 20 of the video distribution system 1 in that the storage 23 stores the registered facial expression management data 23d. Hereinafter, the differences between the embodiment of FIG. 11 and the embodiment of FIG. 1 will be described. The matters common to the embodiment of FIG. 1 in the embodiment of FIG. 11 will not be described.

The face feature amount acquisition unit 11f acquires a face feature amount representing the facial features of the viewing user who views the moving image using the moving image playback device 110. Specifically, the facial feature amount acquisition unit 11f is a feature that represents facial features from an image including the face of the viewing user acquired by the sensor unit 15 according to, for example, HOG, SIFT, SURF, or a known algorithm other than these. Calculate the amount. For example, the facial feature amount calculated by SIFT is calculated as a 128-dimensional feature amount. As described above, the facial feature amount has a large amount of data as well as the posture feature amount acquired by the posture data acquisition unit 11b.

FIG. 12 shows an example of reference facial expression management data. As shown in FIG. 12, the storage 13 stores the reference facial expression identification data indicating the reference facial expression in association with the reference facial expression identification data for identifying the reference facial expression. The reference facial expression identification data is, for example, a reference facial expression ID that identifies the reference facial expression. The reference facial expression ID is, for example, an identification code represented by several bits. The number of bits of the reference facial expression ID can be determined according to the number of reference facial expressions used in the moving image distribution system 1. The reference facial expression ID is represented by, for example, data having an amount of information of 10 bits or less. The reference facial expression ID may be represented by data having an amount of information of 5 bits or less, 4 bits or less, 3 bits or less, and 2 bits or less. Therefore, the amount of the reference facial expression identification data (reference facial expression ID) is significantly smaller than the amount of facial features.

The reference facial expression identification data is data indicating a reference facial expression. One or more reference facial expressions may be set in the video distribution system 1. The reference facial expression is, for example, a facial expression with the eyes wide open facing the front. The reference facial expression identification data is data that describes the reference facial expression, and has the same data structure as the face feature amount calculated by the face feature amount acquisition unit 11f. When the face feature amount calculated by the face feature amount acquisition unit 11f is the SIFT feature amount calculated by the SIFT algorithm, the reference expression identification data describes the reference expression in the same data format as the SIFT feature amount. is there.

The classification unit 11d determines whether or not the facial expression of the viewing user belongs to a predetermined reference facial expression based on the facial feature amount of the viewing user by a classifier that classifies the facial expression of the viewing user. In one embodiment, the classification unit 11d can display an image representing a reference facial expression on the display 14 in order to obtain teacher data, and prompt the viewing user to make a facial expression according to the image representing the reference facial expression. The content reproduction device 110 acquires the facial feature amount of the facial expression created by the viewing user in response to the display of the image representing the reference facial expression by the facial feature amount acquisition unit 11f. The classification unit 11d can create a classifier by learning the facial expression features of the facial expression created by the viewing user with respect to the image representing the reference facial expression as teacher data.

The transmission unit 11c transmits motion data including a face feature amount representing the facial features of the viewing user acquired by the face feature amount acquisition unit 11f to the moving image distribution device 120. When the delay monitoring unit 11e determines that the transmission of motion data is delayed in the transmission line, the transmission unit 11c distributes the reference facial expression identification data (reference facial expression ID) stored in the storage 13 as a moving image. It can be transmitted to the device 120. When the delay monitoring unit 11e determines that a delay has occurred in the transmission line, the reference facial expression identification data may be transmitted to the moving image distribution device 120 instead of the facial feature amount.

In the video distribution device 20, the animation generation unit 21c generates an animation of the viewer's avatar participating in the video 60. In one embodiment, the animation generation unit 21c generates an animation of the viewing user's avatar based on the amount of facial features of the viewing user received from the moving image playback device 110. The animation generation unit 21c continuously receives the facial feature amount of the viewing user from the moving image playback device 110, and synchronizes with the facial movement (change of facial expression) of the viewing user based on the facial feature amount. It is possible to generate an avatar animation that changes the facial expression.

The animation generation unit 21c in one embodiment refers to the registered facial expression management data 23d, and based on the reference facial expression identification data (for example, the reference facial expression ID) from the moving image playback device 110, the animation of the avatar including the movement of the facial expression. Can be generated. FIG. 13 shows an example of the registered facial expression management data 23d according to the embodiment of FIG. As shown in FIG. 13, the registered facial expression management data 23d displays the reference facial expression identification data (registered facial expression ID), the registered facial expression identification data for identifying the registered facial expression (registered facial expression ID), and the movement of the avatar's facial expression. It has animation definition data for specifying the including animation. This animation definition data is data that defines the movement of the facial expression of the avatar. The animation definition data included in the registered facial expression management data 23d may be data that describes the positions of the feature points of the avatar's face in chronological order. When the animation generation unit 21c receives the reference facial expression identification data from the moving image playback device 110, the animation generation unit 21c identifies the registered facial expression identification data associated with the received reference facial expression identification data by referring to the animation management data 23d. It is possible to generate an avatar animation including facial expression movements based on the animation definition data associated with the specified registered facial expression identification data.

Next, another embodiment of the present invention will be described with reference to FIG. FIG. 14 shows a block diagram of the moving image distribution system 201 according to another embodiment of the present invention. Generally speaking, in the video distribution system 1 shown in FIG. 1, the posture feature amount representing the posture of the viewing user is transmitted in real time, whereas in the video distribution system 201 shown in FIG. 14, the posture feature amount is transmitted in real time. The two are different in that the posture feature amount of the viewing user is transmitted only when the viewing user makes a characteristic movement. The video distribution system 201 includes a video playback device 210 and a video distribution device 220. In the moving image playback device 210 of the moving image distribution system 201, the computer processor 11 functions as a determination unit 11g and a transmission unit 11h. The moving image playback device 210 does not have to have the reference facial expression management data 13b, and the moving image distribution device 220 does not have to have the animation management data 23c. The matters common to the embodiment of FIG. 1 in the embodiment of FIG. 14 will not be described.

The determination unit 11g determines whether or not the user of the moving image reproduction device 210 has performed a characteristic movement. Specifically, the posture data acquisition unit 11b calculates the posture feature amount of the user's feature point of the moving image playback device 210 at a predetermined frame rate, and the determination unit 11g is based on the RMS difference of the posture feature amount of this frame. It is determined whether or not the user has performed a characteristic movement. The calculation of the posture feature amount for each frame and the calculation of the RMS difference have already been described with reference to FIG. In one embodiment, the determination unit 11g determines that the user has made a characteristic movement in the frame in which the positive and negative of the RMS difference are reversed. As described above, when the sign of the RMS difference is reversed from that of the previous frame in a certain frame, it is presumed that the user made a characteristic movement accompanied by a reciprocating movement such as waving or nodding in the frame. To. For example, when the posture data acquisition unit 11b calculates the posture feature amount for 24 frames shown in FIG. 8, the fourth frame, the tenth frame, the 16th frame, in which the inversion flag is set to "1", It is determined that the user has made a characteristic movement in the 17th frame, the 19th frame, and the 21st frame.

In one embodiment, the determination unit 11g determines that the user has made a large movement in a frame in which the RMSPE is greater than a predetermined threshold. This threshold can be, for example, 1.0. The threshold value used by the determination unit 11g can be changed as appropriate. In the example of FIG. 8, the RMSPE is larger than the threshold value of 1.0 in the 4th to 7th frames. Therefore, the determination unit 11g can determine that the user has made a large movement in the 4th to 8th frames.

The transmission unit 11h selects a frame determined by the determination unit 11g that the user has performed a characteristic movement, and transmits the posture feature amount in the selected frame to the moving image distribution device 220. The frame selected by the transmission unit 11h may be referred to as a "selected frame". The selection frame may be only a frame determined by the determination unit 11g that the user has performed a characteristic movement, or a frame determined by the determination unit 11g that the user has performed a characteristic movement, and one subsequent frame. Alternatively, it may be a plurality of frames. In this way, the transmission unit 11h is configured so that the posture feature amount in the selected frame is transmitted to the moving image distribution device 220, while the posture feature amount in the frames other than the selected frame is not transmitted to the moving image distribution device 220.

In the video distribution device 220 that has received the posture feature amount transmitted by the transmission unit 11h, the animation generation unit 21c generates an animation of the user's avatar of the video playback device 210 based on the received posture feature amount. The method of generating the avatar animation based on the posture feature amount is the same as the method of generating the animation in the moving image distribution system 1. In the video distribution system 201, there is a period in which the posture feature amount is not transmitted from the video playback device 210 to the video distribution device 220. Specifically, the posture feature amount is not transmitted to the moving image distribution device 220 in frames other than the selected frame. During the period when the posture feature amount is not transmitted, the animation generation unit 21c may create an animation of the avatar so as to perform a movement according to a basic motion defined for the avatar. The basic movements of the avatar are as described, and are predetermined movements such as waving hands.

In the video distribution system 201, since the posture feature amount is transmitted from the video playback device 210 to the video distribution device 220 only in the selected frame, the amount of data to be transmitted is smaller than the mode in which the calculated posture feature amount is transmitted in real time. Can be reduced.

Next, another embodiment of the present invention will be described with reference to FIG. FIG. 15 shows a block diagram of the moving image distribution system 301 according to another embodiment of the present invention. The moving image distribution system 1 is different from the moving image distribution system 1 in that the moving image reproducing device generates a moving image based on data necessary for generating a moving image such as a posture feature amount. Specifically, the video distribution system 301 includes a posture data acquisition device 310, a video distribution device 320, and a video playback device 330. The posture data acquisition device 310, the video distribution device 320, and the video playback device 330 are connected via the network 50. In the video distribution system 301, it is assumed that the video playback device 320 generates a video including an avatar animation generated based on the movement of the user of the posture data acquisition device 310, and the generated video is played back. .. That is, the user of the posture data acquisition device 310 can play a moving image including an avatar that moves based on his / her own movement on the moving image reproducing device 320 used by the viewing user. The user of the posture data acquisition device 310 can interact with the user of the moving image playback device 320 via the avatar. Regarding the present embodiment, the user of the posture data acquisition device 310 may be simply referred to as a “delivery user”. Although the posture data acquisition device 310, the moving image distribution device 320, and the moving image reproducing device 330 are shown one by one in FIG. 15 for explanation, the moving image distribution system 301 may include a plurality of these devices.

The posture data acquisition device 310 acquires a posture feature amount representing the posture characteristics of the user of the posture data acquisition device 310. The posture feature amount representing the posture feature of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user, that is, three-dimensional vector data indicating the position and orientation of the bones of the viewing user. The posture data acquisition device 310 may include the sensor unit 15 described above. The posture data acquisition device 310 can generate three-dimensional skeleton data of the viewing user based on the detection data detected by the sensor unit 15. The posture data acquisition device 310 transmits the acquired posture feature amount to the moving image distribution device 320. The posture data acquisition device 310 may transmit voice data representing the voice of the distribution user to the video distribution device 320.

The video distribution device 320 includes a computer processor 21, a communication I / F 22, and a storage 23, similarly to the video distribution device 20 described above. The video distribution device 320 differs from the video distribution device 20 in the functions realized by the computer processor 21 and the data stored in the storage 23. Therefore, in the following, the functions realized by the computer processor 21 of the moving image distribution device 320 and the data stored in the storage 23 will be described.

As shown in the figure, the moving image distribution system 301 storage 23 can store object data 23a, avatar data 23b, reference posture management data 323a, and various information necessary for generating and distributing moving images other than the above. The object data 23a and the avatar data 23b may be the same as the object data 23a and the avatar data 23b stored in the moving image distribution system 1. The reference posture management data 323a may be the same as the reference posture management data 13a in the moving image distribution system 1. That is, as shown in FIG. 3, the reference posture management data 323a includes the reference posture identification data for identifying the reference posture, and the start posture data and the trigger posture data stored in association with the reference posture identification data. Can include.

The computer processor 21 functions as a posture feature acquisition unit 321a, a transmission unit 321b, a classification unit 321c, and a delay monitoring unit 321d by executing a computer-readable instruction. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the moving image distribution system 301.

The posture feature amount acquisition unit 321a acquires the posture feature amount representing the user's posture of the posture data acquisition device 310 from the posture data acquisition device 310.

The transmission unit 321b transmits the posture feature amount representing the posture of the user of the posture data acquisition device 310 acquired from the posture data acquisition device 310 to the moving image playback device 330. When the transmission unit 321b receives the posture feature amount from the posture data acquisition device 310, it may immediately transmit it to the moving image distribution device 320. In other words, the transmission unit 321b can transmit the posture feature amount of the distribution user to the video distribution device 320 in real time. The transmission unit 321b may transmit a packet including this posture feature amount to the moving image playback device 320. In addition to the posture feature amount, the transmission unit 321b can transmit the data necessary for generating the moving image in the moving image reproducing device 330 to the moving image reproducing device 330. For example, the transmission unit 321b can transmit at least a part of the object data 23a and the avatar data 23b to the moving image playback device 330. The object data 23a and the avatar data 23b may be stored in advance in the moving image reproduction device 330 instead of being transmitted to the moving image reproduction device 330 by the transmission unit 321b. In this case, it is not necessary to transmit the object data 23a and the avatar data 23b from the video distribution device 320 to the video playback device 330. Further, the transmission unit 321b may transmit the voice data representing the voice of the distribution user acquired from the generation data acquisition device 310 by the video distribution device 320 to the video playback device 330.

The classification unit 321c uses a classifier that classifies the posture of the distribution user, and whether the posture of the viewing user belongs to a predetermined reference posture based on the posture feature amount (for example, three-dimensional skeleton data) of the viewing user. Judge whether or not. This classifier is, for example, a linear classifier. The classification unit 321c may realize substantially the same function as the classification unit 11d of the moving image distribution system 1. For example, when the reference posture includes the start posture and the trigger posture, the classification unit 321c can determine whether or not the posture of the distribution user belongs to the reference posture and whether or not the posture belongs to the trigger posture. ..

The delay monitoring unit 321d monitors the transmission delay of the motion data transmitted from the video distribution device 320 to the video playback device 330 by the transmission unit 321c. The delay monitoring unit 321d may realize substantially the same function as the delay monitoring unit 11e of the moving image distribution system 1. For example, the delay monitoring unit 321d adds a time stamp to an actual packet containing motion data before transmission, and adds a type stamp added at the time of transmission and when this actual packet is received by the moving image playback device 330. It is possible to obtain the transmission time of the actual packet on the transmission line between the moving image distribution device 320 and the moving image reproducing device 330 by using the time stamp. Whether or not a transmission delay has occurred may be determined by the moving image player 330 based on the time stamp included in the packet. The moving image playback device 320 may receive the determination result of the transmission delay in the moving image distribution device 330.

The function of the transmission unit 321b will be described again. When the delay monitoring unit 321d determines that a delay has occurred in the transmission line, the transmission unit 321b transmits the reference posture identification data (reference posture ID) stored in the storage 23 to the moving image playback device 330. be able to. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As described above, the reference attitude identification data may include the start attitude data and the trigger attitude data. The function of the transmission unit 321b executed when the transmission delay occurs is the same as the function executed by the transmission unit 11c when the transmission delay occurs.

The moving image reproducing device 330 includes a computer processor 11, a communication I / F 12, and a storage 13 like the moving image reproducing device 10 described above. The moving image playback device 330 differs from the moving image playback device 10 in terms of the functions realized by the computer processor 11 and the data stored in the storage 23. Therefore, in the following, the functions realized by the computer processor 21 of the moving image reproduction device 330 and the data stored in the storage 23 will be described.

Animation management data 313a is stored in the storage 13 of the moving image playback device 330. The animation management data 313a may be the same as the animation management data 13a stored in the moving image distribution system 1. For example, as shown in FIG. 6, the animation management data 313a includes reference posture identification data, registered animation identification data (registered animation ID) for identifying registered animation, and animation definition for specifying avatar animation. With data.

The computer processor 11 functions as an animation generation unit 331a, a moving image generation unit 331b, and a moving image reproduction unit 331c by executing a computer-readable instruction. At least a part of the functions realized by the computer processor 11 may be realized by a computer processor other than the computer processor 11 of the moving image distribution system 301.

The animation generation unit 331a may realize substantially the same function as the animation generation unit 21c of the moving image distribution system 1. For example, the animation generation unit 331c can generate an animation of the avatar of the distribution user of the posture data acquisition device 310. In one embodiment, the animation generation unit 331c generates an animation of the avatar of the distribution user based on the posture feature amount of the distribution user and the avatar data 23b received from the video distribution device 320. The animation generation unit 331c continuously receives the posture feature amount of the viewing user from the video distribution device 320, and based on the posture feature amount, produces an animation of the avatar that moves in synchronization with the body movement of the viewing user. Can be generated.

The moving image generation unit 331b can construct a virtual space using the object data 23a acquired from the moving image distribution device 320, and can generate a moving image in which the avatar generated by the animation generation unit 331a moves in the virtual space. The audio of the distribution user can be combined with this video.

The moving image reproduction unit 331a reproduces the moving image generated by the moving image generation unit 331b. As a result, the moving image including the animation of the distribution user's avatar is displayed on the display 14 of the moving image reproducing device 330.

A process realized by the animation generation unit 331a when a transmission delay occurs in the transmission line for transmitting data from the moving image distribution device 320 to the moving image reproducing device 330 will be described. If a transmission delay occurs in the transmission line from the video distribution device 320 to the video playback device 330, it may not be possible to generate a smoothly moving avatar animation that reflects the movement of the distribution user. The animation generation unit 331a can generate an animation of the avatar based on the reference posture identification data (for example, the reference posture ID) from the moving image distribution device 320. Specifically, when the animation generation unit 331a receives the reference posture identification data from the moving image playback device 10, the animation management unit 331a identifies the registered animation associated with the received reference posture identification data by referring to the animation management data 313a. It is possible to generate an avatar animation based on the animation definition data associated with the registered animation ID of the specified registered animation. Therefore, even if the posture feature amount cannot be received while the motion data transmission is delayed, the animation generation unit 331a is sent from the moving image distribution device 320 while the delay is occurring. An avatar animation can be generated based on the reference posture identification data.

In one embodiment, the posture data acquisition device 310 may be configured to realize the function of the moving image reproduction device 320, and the moving image reproduction device 320 may be configured to be able to realize the function of the posture data acquisition device 310. As a result, users of both devices can communicate with each other bidirectionally via the avatar.

Next, a modified example of the above-mentioned video distribution system will be described with reference to FIG. In the above embodiment, when the posture of the viewing user is determined to belong to the reference posture by the classification unit 11d, the posture of the user is determined by transmitting the reference posture ID that identifies the reference posture instead of the posture feature amount. The amount of data to be represented is reduced. In the example shown in FIG. 16, instead of the reference posture ID, the index (subscript) of the feature array determined as follows is used. That is, when the transmission delay occurs, the index of the feature array is transmitted instead of the posture feature amount representing the posture of the user. The index of the feature array is data of about several bits as described below, and can be represented by an amount of information in the range of 10 bits or less like the reference posture ID.

In the moving image distribution system 1, the feature arrangement is determined as follows. First, as described with reference to FIGS. 7 and 8, the moving image playback device 10 captures the user of the moving image playback device 10 at a predetermined frame rate to acquire a captured image, and in each frame of the captured image. Extract a plurality of feature points related to the user. For example, as shown in FIG. 7, six feature points P1 to P6 are extracted. The moving image playback device 10 obtains an image vector relating to the movement of the user for each of the extracted plurality of feature points. The image vector at each of the feature points P1 to P6 is, for example, the depth at each feature point. The image vector of each feature point P1 to P6 may be normalized to the range of 0 to 1.0 and expressed as a float array. The image vectors of the feature points P1 to P6 acquired in this way are shown in FIG. Although FIG. 16 shows image vectors for 10 frames for the sake of brevity, the image vectors are actually acquired for a sufficient amount of frames for statistical analysis.

Next, for each of the feature points P1 to P6, the sum Σ of the elements of the image vector in the predetermined section is calculated. In the table of FIG. 16, the sum of the elements of the image vector in the section of 26 frames from the 0th frame to the 25th frame is shown in the row indicated by “Σ”. Next, the sum of the elements of the image vectors of the feature points P1 to P6 is sorted in descending order. In FIG. 16, this sort result is shown in the line labeled "RANK". In the example of FIG. 16, the items are sorted in the order of P4, P3, P6, P2, P1, and P5. This sorted array is expressed as {4,3,6,2,1,5}.

The moving image playback device 10 calculates a representative vector showing a characteristic posture or movement of the user by learning a sample of an image vector related to the movement of the user. The representative vector is calculated using, for example, the Lloyd algorithm and other known algorithms. That is, the representative vector indicates the characteristic posture of the user. Therefore, the posture of the user can be specified based on this representative vector. The moving image playback device stores the calculated representative vector together with the index. The moving image playback device 10 may acquire a representative vector calculated by a device other than its own device. In the present embodiment, four representative vectors A to D are calculated, and the elements of each representative vector are sorted in descending order to obtain an array of representative vectors. The array in which each element of the representative vectors A to D is sorted in descending order is {1,3,6,2,4,5}, {4,3,6,2,1,5} {2,3, respectively. It is assumed that 6,4,5,1} {1,2,3,4,5,6} is represented.

The moving image playback device 10 compares the input array {4,3,6,2,1,5} obtained by sorting the image vectors obtained from the captured image with the array of each representative vector, and compares the input array { Select the array of representative vectors closest to 4,3,6,2,1,5}. In the example shown in FIG. 16, the input array {4,3,6,2,1,5} is an array of the representative vector B out of an array of four representative vectors {4,3,6,2,1, Since it matches 5}, {4,3,6,2,1,5} is selected as the feature sequence. The video playback device 10
The index of the feature array selected in this way is transmitted to the moving image distribution device 20.

The moving image distribution device 20 stores a code book in which the representative vectors A to D and their arrays are associated with each other. The video distribution device 20 refers to the codebook and generates an animation of the user's avatar so as to take a posture corresponding to the representative vector B corresponding to the feature array acquired from the video playback device 10.

In the above example, the feature array is transmitted from the moving image playback device 10 to the moving image distribution device 20 for each unit interval of 26 frames. This unit interval can be adjusted as appropriate.

As described above, it has been described that the video distribution system 1 transmits the index of the feature array instead of the reference posture ID, and generates the avatar animation based on the index of the feature array when the transmission is delayed. Similarly, in the video distribution system 301, a feature array can be used instead of the reference posture ID. The feature array can also be used in place of the reference facial expression ID in the moving image distribution system 101.

The action and effect produced by the above embodiments will be described. In the above aspect, while the delay occurs in the transmission line, the reference posture identification data that can specify the information about the posture of the viewing user with a small amount of data instead of the large amount of posture features is the moving image. It is transmitted from the playback device 10 to the moving image distribution device 20. As a result, when a delay occurs in the transmission line, information regarding the posture of the viewing user can be transmitted to the moving image distribution device 20 by the reference posture identification data of a small amount of data.

In one aspect, a registration animation can be registered for the viewing user's avatar, and while a delay occurs in the transmission line, a registration animation corresponding to the reference posture specified based on the reference posture identification data is created. The including video will be delivered. As a result, even when it is difficult to transmit the posture feature amount representing the posture of the viewing user to the video distribution device 20 due to the delay in the transmission path from the video playback device 10 to the video distribution device 20, the viewing user Posture-related registration animations can be included in the video.

In the above aspect, the moving image playback device 330 generates an animation of an avatar that moves based on the movement of the distribution user based on the posture feature amount of the distribution user received from the video distribution device 320. While the delay is occurring in the transmission line, the reference posture identification data, not the posture feature amount, is transmitted from the video distribution device 320 to the video playback device 310. As a result, when a delay occurs in the transmission line, the avatar animation can be generated in the moving image playback device 330 by the reference posture identification data that can represent the movement of the distribution user with a small amount of data.

In one aspect, a user animation generated based on the user's movement can be registered as a registration animation. As a result, it is possible to increase the variation of the movement of the avatar when the posture feature amount cannot be used.

In one aspect, while the delay occurs in the transmission line, the reference facial expression identification data that can specify the information about the posture of the viewing user with a small amount of data instead of the large amount of facial features is the video playback device. It is transmitted from 10 to the moving image distribution device 20. As a result, when a delay occurs in the transmission line, information on the facial expression of the viewing user can be transmitted to the moving image distribution device 20 by the reference facial expression identification data of a small amount of data.

In the processing procedure described in the present specification, particularly the processing procedure described using the flow chart, a part of the steps (steps) constituting the processing procedure is omitted, as a step constituting the processing procedure. It is possible to add steps that are not specified and / or to change the order of the steps, and the processing procedure in which such omissions, additions, and changes in order are made does not deviate from the gist of the present invention. It is included in the scope of the invention.

The inventions described in the claims at the time of filing the original application of the present application are added below.
[1]
Equipped with one or more computer processors
The one or more computer processors may execute computer-readable instructions.
Play the video received from the video distribution device and play it.
Motion data including a posture feature amount representing the posture feature of the user is transmitted to the video distribution device, and the motion data is transmitted to the video distribution device.
A video including an animation of the user's avatar generated based on the posture feature amount is received from the video distribution device, and the video is received.
It is determined whether or not the user's posture belongs to a predetermined reference posture, and the user's posture is determined.
While the transmission of the motion data to the moving image distribution device is delayed, the reference posture determined to belong to the user's posture is identified, and the reference posture identification of a data amount smaller than the posture feature amount is specified. Sending data to the video distribution device,
Video playback device.
[2]
The classifier determines whether or not the posture of the user matches the reference posture based on the evaluation function using the posture feature amount as a variable.
The moving image playback device according to [1].
[3]
Whether or not the posture of the user belongs to the reference posture is determined by a classifier that classifies the posture of the user based on the posture feature amount.
The classifier is created by presenting an image representing the reference posture to the user and learning the posture feature amount representing the posture taken by the user with respect to the presented image as teacher data.
The moving image playback device according to [1].
[4]
One or more registered animations are registered for the user's avatar.
While the delay is occurring, a moving image including the registered animation corresponding to the reference posture identified based on the reference posture identification data is received.
The moving image playback device according to any one of [1] to [3].
[5]
A user animation of the avatar is generated based on the posture feature amount acquired in time series.
Registering the user animation as the registration animation,
Generate a sample video containing the user animation
The reference posture corresponding to the user animation is determined based on the image of the reference frame selected from the plurality of frames constituting the sample moving image.
The moving image playback device according to [4].
[6]
A user animation of the avatar is generated based on the posture feature amount acquired in time series.
Registering the user animation as the registration animation,
The posture feature amount for the user is calculated at each of the plurality of feature points for the user based on a predetermined frame rate.
In the first frame, the first RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
In the second frame after the first frame in chronological order, the second RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
In the third frame after the second frame in chronological order, the third RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
When the positive / negative of the first RMS difference, which is the difference between the second RMS and the first RMS, and the positive / negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the posture feature in the third frame. The reference posture corresponding to the user animation is determined based on the quantity.
The moving image playback device according to [4].
[7]
A user animation of the avatar is generated based on the posture feature amount acquired in time series.
Registering the user animation as the registration animation,
The first evaluation data including the posture feature amount acquired in time series based on the movement of the user with respect to the user animation, and the posture acquired in time series based on other movements of the user with respect to the user animation. The reference posture corresponding to the user animation is determined by comparing with the second evaluation data including the feature amount.
The moving image playback device according to [4].
[8]
The posture feature amount includes bone data representing the position and orientation of the user's bone as a three-dimensional vector.
The moving image playback device according to any one of [1] to [7].
[9]
The motion data includes a facial feature amount representing the facial features of the user.
The one or more computer processors
It is determined whether or not the user's facial expression belongs to a predetermined reference facial expression by another classifier that classifies the user's facial expression based on the facial feature amount.
While the delay is occurring, the reference facial expression determined to belong to the user's facial expression is identified, and the reference facial expression identification data having a data amount smaller than the facial feature amount is transmitted to the moving image distribution device.
The moving image playback device according to any one of [1] to [8].
[10]
The reference posture identification data is transmitted in place of the motion data while the delay is occurring.
The moving image playback device according to any one of [1] to [9].
[11]
The motion data is transmitted in real time,
The moving image playback device according to any one of [1] to [10].
[12]
A video distribution system that includes one or more computer processors and distributes a video including a user's avatar to the user's video playback device.
The one or more computer processors may execute computer-readable instructions.
Motion data including a posture feature amount representing the posture of the user is received from the moving image playback device via a transmission line, and the motion data is received.
The animation of the user's avatar generated based on the posture feature amount is included in the moving image and distributed.
When the posture of the user is determined to belong to the reference posture by the classifier that classifies the posture of the user based on the posture feature amount, the transmission of the motion data to the moving image distribution device is delayed. In the meantime, the reference posture is identified, and the reference posture identification data having a data amount smaller than the posture feature amount is received from the moving image playback device.
The user's avatar registration animation generated based on the reference posture identification data is included in the moving image and distributed.
Video distribution system.
[13]
A method of playing video that is executed by one or more computer processors executing computer-readable instructions.
The process of playing the video received from the video distribution device and
A process of transmitting motion data including a posture feature amount representing a user's posture to the video distribution device, and
A step of receiving a moving image including an animation of the user's avatar generated based on the posture feature amount from the moving image distribution device, and
A step of determining whether or not the user's posture belongs to a predetermined reference posture by a classifier that classifies the user's posture based on the posture feature amount.
A step of determining whether or not a delay has occurred in the transmission of the motion data to the moving image distribution device, and
A step of identifying the reference posture determined to belong to the user's posture while the delay is occurring, and transmitting reference posture identification data having a data amount smaller than the posture feature amount to the moving image distribution device. ,
Video playback method with.
[14]
A video player equipped with one or more computer processors.
The one or more computer processors may execute computer-readable instructions.
Based on a predetermined frame rate, the posture feature amount related to the user is calculated at each of the plurality of feature points related to the user of the moving image playback device.
In the first frame, the first RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
In the second frame after the first frame in chronological order, the second RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
The RMS difference, which is the difference between the second RMS and the first RMS, is calculated.
When the RMS difference is larger than a predetermined threshold value, the posture feature amount in the second frame is transmitted to the moving image distribution device.
A video including an animation of the user's avatar generated based on the posture feature amount is received from the video distribution device, and the video is received.
Play the video,
Video playback device.
[15]
A video player equipped with one or more computer processors.
The one or more computer processors may execute computer-readable instructions.
Based on a predetermined frame rate, the posture feature amount related to the user is calculated at each of the plurality of feature points related to the user of the moving image playback device.
In the first frame, the first RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
In the second frame after the first frame in chronological order, the second RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
In the third frame after the second frame in chronological order, the third RMS, which is the root mean square of the posture feature amount at each of the plurality of feature points, is calculated.
When the positive / negative of the first RMS difference, which is the difference between the second RMS and the first RMS, and the positive / negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the posture feature in the third frame. Send the amount,
A video including an animation of the user's avatar generated based on the posture feature amount is received from the video distribution device, and the video is received.
Play the video,
Video playback device.
[16]
Equipped with one or more computer processors
The one or more computer processors may execute computer-readable instructions.
Motion data including a posture feature amount representing the posture feature of the user is transmitted to the video playback device, and the motion data is transmitted.
It is determined whether or not the user's posture belongs to a predetermined reference posture, and the user's posture is determined.
While the transmission of the motion data to the moving image playback device is delayed, the reference posture determined to belong to the user's posture is identified, and the reference posture identification of a data amount smaller than the posture feature amount is specified. Sending the data to the video playback device,
Video distribution device.
[17]
Equipped with one or more computer processors and storage for storing registered animations,
The one or more computer processors may execute computer-readable instructions.
Receives motion data including a posture feature amount representing the posture feature of the user received from the video distribution device, and receives the motion data.
Generate a video containing the user's avatar animation generated based on the motion data.
While the transmission of the motion data from the moving image distribution system is delayed, the reference posture determined to belong to the user's posture is identified, and the reference posture identification data having a data amount smaller than the posture feature amount is used. Generates a video containing the registered animation corresponding to the reference posture based on
Video playback device.

1, 101, 201, 301 Video distribution system 10, 110, 210, 330 Video playback device 11a Video playback unit 11b Posture data acquisition unit 11c Transmission unit 11d Classification unit 11e Delay monitoring unit 11f Facial feature amount acquisition unit 20, 120, 220 , 320 Video distribution device 21a Video generation unit 21b Video distribution unit 21c Animation generation unit 310 Posture data acquisition device

Claims (12)

Equipped with one or more computer processors
The one or more computer processors may execute computer-readable instructions.
Play the video received from the video distribution device and play it.
Motion data including a facial feature amount representing a user's facial feature is transmitted to the video distribution device, and the motion data is transmitted to the video distribution device.
A video including an animation of the user's avatar generated based on the facial features is received from the video distribution device, and the video is received.
It is determined whether or not the user's facial expression matches a predetermined reference facial expression, and the user's facial expression is determined.
While the transmission of the motion data to the moving image distribution device is delayed, the reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression is transmitted to the moving image distribution device.
The amount of the reference facial expression identification data is smaller than the amount of the facial feature.
Video playback device. Whether or not the user's facial expression matches the reference facial expression is determined based on the evaluation function with the facial feature amount as a variable.
The moving image playback device according to claim 1. Whether or not the user's facial expression matches the reference facial expression is determined by a classifier that classifies the user's facial expression based on the facial feature amount.
The classifier is created by presenting an image representing the reference facial expression to the user and learning the facial feature amount representing the facial expression created by the user with respect to the presented image as teacher data.
The moving image playback device according to claim 1. One or more registered animations are registered for the user's avatar.
While the delay is occurring, the moving image including the registered animation corresponding to the reference facial expression identified based on the reference facial expression identification data is received.
The moving image playback device according to any one of claims 1 to 3. A user animation of the avatar is generated based on the facial features acquired in chronological order.
Registering the user animation as the registration animation,
Generate a sample video containing the user animation
The reference facial expression corresponding to the user animation is determined based on the image of the reference frame selected from the plurality of frames constituting the sample moving image.
The moving image playback device according to claim 4. A user animation of the avatar is generated based on the facial features acquired in chronological order.
Registering the user animation as the registration animation,
The facial feature amount for the user is calculated at each of the plurality of feature points for the user based on a predetermined frame rate.
In the first frame, the first RMS, which is the root mean square of the facial features at each of the plurality of feature points, is calculated.
In the second frame after the first frame in chronological order, the second RMS, which is the root mean square of the facial features at each of the plurality of feature points, is calculated.
In the third frame after the second frame in chronological order, the third RMS, which is the root mean square of the facial features at each of the plurality of feature points, is calculated.
When the positive / negative of the first RMS difference, which is the difference between the second RMS and the first RMS, and the positive / negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the facial expression feature in the third frame. The reference facial expression corresponding to the user animation is determined based on the amount.
The moving image playback device according to claim 4. A user animation of the avatar is generated based on the facial features acquired in chronological order.
Registering the user animation as the registration animation,
The first evaluation data including the facial feature amount acquired in time series based on the facial expression of the user with respect to the user animation, and the face acquired in time series based on other facial expressions of the user with respect to the user animation. The reference facial expression corresponding to the user animation is determined by comparing with the second evaluation data including the feature amount.
The moving image playback device according to claim 4. The motion data is transmitted in real time,
The moving image playback device according to any one of claims 1 to 7. A video distribution system that includes one or more computer processors and distributes a video including a user's avatar to the user's video playback device.
The one or more computer processors may execute computer-readable instructions.
Motion data including a facial feature amount representing the user's facial expression is received from the moving image playback device via a transmission path, and the motion data is received.
The animation of the user's avatar generated based on the facial feature amount is included in the moving image and distributed.
While the transmission of the motion data from the moving image playback device matching the user's facial expression is delayed, the reference facial expression identification data for identifying the reference facial expression matching the user's facial expression is obtained from the moving image playback device. Receive and
The registered animation of the user's avatar generated based on the reference facial expression identification data is included in the moving image and distributed.
The amount of the reference facial expression identification data is smaller than the amount of the facial feature.
Video distribution system. A method of playing video that is executed by one or more computer processors executing computer-readable instructions.
The process of playing the video received from the video distribution device and
A process of transmitting motion data including a facial feature amount representing a user's facial expression to the video distribution device, and
A step of receiving a moving image including an animation of the user's avatar generated based on the facial feature amount from the moving image distribution device, and
A step of determining whether or not the user's facial expression matches a predetermined reference facial expression by a classifier that classifies the user's facial expression based on the facial feature amount.
A step of determining whether or not a delay has occurred in the transmission of the motion data to the moving image distribution device, and
A step of transmitting reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression to the moving image distribution device while the delay is occurring.
With
The amount of the reference facial expression identification data is smaller than the amount of the facial feature.
Video playback method. Equipped with one or more computer processors
The one or more computer processors may execute computer-readable instructions.
Motion data including facial features representing the features of the user's facial expressions is transmitted to the video playback device,
It is determined whether or not the user's facial expression matches a predetermined reference facial expression, and the user's facial expression is determined.
While the transmission of the motion data to the moving image playback device is delayed, the reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression is transmitted to the moving image playback device.
The amount of the reference facial expression identification data is smaller than the amount of the facial feature.
Video distribution device. Equipped with one or more computer processors and storage for storing registered animations,
The one or more computer processors may execute computer-readable instructions.
Receives motion data including facial features that represent the features of the user's facial expressions from the video distribution device,
Generate a video containing the user's avatar animation generated based on the motion data.
The said reference facial expression corresponding to the reference facial expression based on the reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression while the transmission of the motion data from the moving image distribution device is delayed. Generate a video containing the registered animation and
The amount of the reference facial expression identification data is smaller than the amount of the facial feature.
Video playback device.