JP6731532B1 - Video playback device, video playback method, and video delivery system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an avatar animation in which a motion of a viewing user is reflected in real time, and include it in a moving image. A moving image reproducing apparatus reproduces a moving image received from a moving image distribution apparatus by executing a computer-readable instruction, and stores motion data including a posture feature amount indicating a characteristic of a user's posture in the moving image distribution apparatus. A user's posture is predetermined by a classifier that transmits and receives a video including an animation of the user's avatar generated based on the posture feature amount from the video distribution device, and classifies the user's posture based on the posture feature amount. It is determined whether the user's posture is determined to belong to the reference posture while delaying the transmission of the motion data to the moving image distribution device, and the reference feature is determined to be more than the posture feature amount. The reference attitude identification data having a small data amount is transmitted to the moving image distribution device. [Selection diagram] Fig. 10

Description

The disclosure in the present specification mainly relates to a moving image reproducing device, a moving image reproducing method, and a moving image distribution system.

A moving image distribution system is known in which a viewing user who views a moving image can participate in the moving image using an avatar. For example, Japanese Unexamined Patent Application Publication No. 2012-120098 (Patent Document 1) describes that an avatar of a viewing user is included in a distributed moving image.

JP 2012-120098 A

In a video distribution service that distributes videos, it is possible to increase the engagement of the viewing users by reflecting the feedback from the viewing users on the videos being distributed. For example, Patent Document 1 describes that a comment from a viewing user is displayed in association with the avatar of the viewing user. In the system of Patent Document 1, the viewing user provides feedback to the distributed video by a method of posting a comment, and this feedback is reflected in the distributed video.

If it is possible to include not only a comment from the viewing user but also an avatar animation that moves according to the operation of the viewing user in the distribution content, it is considered that the engagement of the viewing user can be further enhanced. However, in order to move the avatar according to the action of the viewing user, the motion data indicating the action of the viewing user is transmitted in real time to the moving image generating device that generates the moving image from the moving image playback device of the viewing user. Is required. In order for the avatar to perform a motion that reflects the motion of the user, the motion data indicating the motion of the user needs to include bone data indicating the position and orientation of the bone of the user. The position and orientation of the bone are represented by a three-dimensional vector. Bone data of 20 or more bones may be required to represent the movement of the human body.

As described above, since the amount of bone data is large, it is difficult to transmit motion data including bone data indicating the posture of the viewing user without delay, depending on the bandwidth of the transmission path and the degree of congestion. In particular, the uplink from the user equipment has a smaller transmission capacity than the downlink used for content distribution, and therefore delay is likely to occur in the transmission of bone data using the uplink. As a result, it is difficult to generate an avatar animation that reflects the action of the viewing user in real time based on the bone data transmitted from the viewing user, and to include the generated animation in the moving image being distributed.

Not only when the viewing user participates in the video being delivered by the avatar, but also in the system where the users communicate with each other through the avatar that moves according to their own movement, the bone data of each user is transmitted to the device of the other user. There is a need. Therefore, in a system in which users communicate with each other via an avatar that moves according to their own movements, when a delay occurs in the transmission path, it is necessary to generate an animation of the avatar that moves smoothly reflecting the actions of each user. Is difficult

It is an object of the present disclosure to provide a technical improvement that solves or alleviates at least some of the above-mentioned problems of the prior art.

One of the more specific objects of the present invention is to be able to generate an avatar animation related to a user's posture based on information having a smaller amount of data than before.

Other objectives of the disclosure of the present specification will become apparent by referring to the entire specification. The invention disclosed in this specification may solve the problems grasped from the description of the modes for carrying out the invention of this specification instead of or in addition to the above problems. Good.

A moving image reproducing apparatus according to one aspect includes one or more computer processors, and the one or more computer processors execute a computer-readable instruction to reproduce the moving image received from the moving image distribution apparatus, and to reproduce the moving image. Motion data including a posture characteristic amount representing a posture characteristic is transmitted to the moving image distribution device, and a moving image including an animation of the user's avatar generated based on the posture characteristic amount is received from the moving image distribution device, A classifier that classifies the posture of the user based on the posture characteristic amount determines whether or not the posture of the user belongs to a predetermined reference posture, and there is a delay in transmission of the motion data to the video distribution device. While occurring, the reference posture determined to belong to the user's posture is identified, and reference posture identification data having a data amount smaller than the posture feature amount is transmitted to the moving image distribution device.

In the video playback device according to one aspect, the classifier determines whether or not the posture of the user matches the reference posture, based on an evaluation function with the posture feature amount as a variable.

The moving image reproducing apparatus according to one aspect presents the image representing the reference posture to the user, and learns the posture feature amount representing the posture taken by the user with respect to the presented image as teacher data to perform the classification. Create a container.

In the moving image reproducing device according to one aspect, one or a plurality of registered animations are registered for the avatar of the user. The moving image reproducing apparatus according to an aspect receives a moving image including the registered animation corresponding to the reference posture specified based on the reference posture identification data while the delay is occurring.

A moving image reproducing apparatus according to an aspect generates a user animation of the avatar based on the posture feature amount acquired in time series, registers the user animation as the registered animation, and generates a sample moving image including the user animation. Then, the reference posture corresponding to the user animation is determined based on the image of the reference frame selected from the plurality of frames forming the sample moving image.

The moving image reproducing apparatus according to one aspect calculates the posture feature amount regarding the user at each of the plurality of feature points regarding the user based on a predetermined frame rate, and the posture at each of the plurality of feature points in the first frame. A first RMS that is a root mean square of the feature amount is calculated, and a second RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in the second frame that is later than the first frame in time series. And a third RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in a third frame that is later in time series than the second frame, and the second RMS and the When the positive/negative of the first RMS difference, which is the difference from 1RMS, and the positive/negative of the second RMS difference, which is the difference between the third RMS and the second RMS, are reversed, the user is based on the posture feature amount in the third frame. The reference posture corresponding to the animation is determined.

The moving image reproduction device in one aspect is based on first evaluation data including the posture feature amount acquired in time series based on the movement of the user with respect to the user animation, and other movements of the user with respect to the user animation. The reference posture corresponding to the user animation is determined by comparing the second evaluation data including the posture feature amount acquired in time series.

In one aspect, the posture feature amount includes bone data that represents the position and orientation of the bone of the user with a three-dimensional vector.

In one aspect, the motion data includes a facial feature amount representing a facial feature of the user. The moving image reproducing apparatus in one aspect determines whether or not the facial expression of the user belongs to a predetermined reference facial expression by another classifier that classifies the facial expression of the user based on the facial feature amount, and the delay is During generation, the reference facial expression determined to belong to the facial expression of the user is identified, and reference facial expression identification data having a data amount smaller than the facial feature amount is transmitted to the moving image distribution device.

In one aspect, the reference posture identification data is transmitted in place of the motion data while the delay is occurring.

In one aspect, the motion data is transmitted in real time.

A moving image distribution system according to one aspect includes one or a plurality of computer processors, and distributes a moving image including a user's avatar to the user's moving image playback device. The one or more computer processors execute a computer-readable instruction to receive motion data including a posture characteristic amount indicating the posture of the user from the moving image reproducing apparatus via a transmission path, and the posture characteristic It is determined that the user's posture belongs to a reference posture by a classifier that distributes the animation of the user's avatar generated based on the amount in the moving image and classifies the user's posture based on the posture feature amount. In this case, while the transmission of the motion data to the moving image distribution device is delayed, the reference posture is identified, and the reference posture identification data having a data amount smaller than the posture feature amount is used as the moving image reproducing device. From the reference posture identification data, and the distribution animation is generated by including the user's avatar registration animation generated based on the reference posture identification data in the moving image.

The video playback method according to one aspect is performed by one or more computer processors executing computer-readable instructions. The method includes a step of reproducing a moving image received from a moving image distribution device, a step of transmitting motion data including a posture characteristic amount representing a posture of a user to the moving image distribution device, and a step of changing the posture characteristic amount from the moving image distribution device. Receiving a moving image including an animation of the avatar of the user generated based on the above, and the posture of the user belongs to a predetermined reference posture by a classifier that classifies the posture of the user based on the posture feature amount And determining the reference posture determined to belong to the posture of the user while delaying the transmission of the motion data to the moving image distribution apparatus. And a step of transmitting reference posture identification data having a smaller data amount to the moving image distribution device.

A moving image reproducing apparatus according to an aspect of the present invention calculates a posture characteristic amount regarding the user at each of a plurality of characteristic points regarding the user of the moving image reproducing apparatus based on a predetermined frame rate, and the plurality of characteristic points in a first frame. A first RMS, which is a root mean square of the posture feature amount at each of the points, is calculated, and a square of the posture feature amount at each of the plurality of feature points in a second frame that is later in time series than the first frame. A second RMS that is an average square root is calculated, an RMS difference that is a difference between the second RMS and the first RMS is calculated, and the second frame is transmitted to the video distribution device when the RMS difference is larger than a predetermined threshold. And transmitting the posture feature amount in, the moving image receiving apparatus receives the moving image including the animation of the avatar of the user generated based on the posture characteristic amount, and reproduces the moving image.

A moving picture reproducing apparatus according to an aspect of the present invention calculates a posture characteristic amount regarding the user at each of a plurality of characteristic points regarding the user of the moving picture reproducing apparatus based on a predetermined frame rate, and the plurality of characteristic points in a first frame. A first RMS, which is a root mean square of the posture feature amount at each of the points, is calculated, and a square of the posture feature amount at each of the plurality of feature points in a second frame that is chronologically later than the first frame. Calculating a second RMS that is a mean square root, and calculating a third RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in a third frame that is later in time series than the second frame, The posture characteristic in the third frame when the positive/negative of the first RMS difference which is the difference between the second RMS and the first RMS and the positive/negative of the second RMS difference which is the difference between the third RMS and the second RMS are reversed. The amount of video data is transmitted, the video including the animation of the avatar of the user generated based on the posture feature amount is received from the video distribution device, and the video is played.

According to the embodiment, it is possible to generate an avatar animation related to a user's posture based on information having a smaller data amount than before.

FIG. 1 is a block diagram showing a moving image distribution system according to an embodiment. It is a figure which shows notionally a three-dimensional skeleton model. It is a figure explaining the reference attitude|position management data memorize|stored in the moving image distribution system of FIG. It is a figure which shows notionally the three-dimensional skeleton model corresponding to a starting posture. It is a figure which shows notionally the three-dimensional skeleton model corresponding to a trigger posture. It is a figure explaining the avatar data memorize|stored in the moving image distribution system of FIG. It is a figure explaining the animation management data memorize|stored in the moving image distribution system of FIG. It is a figure which shows the frame which comprises a picked-up image typically. 9 is a table showing an example of feature amounts and RMS thereof at a plurality of feature points in each frame. FIG. 5 is a diagram showing an example of a moving image displayed on the moving image reproducing apparatus 10 in one embodiment. FIG. 5 is a diagram showing an example of a moving image displayed on the moving image reproducing apparatus 10 in one embodiment. It is a flowchart which shows the one part flow of a process in the moving image reproduction method in one Embodiment. It is a block diagram which shows the moving image distribution system by other embodiment. FIG. 12 is a diagram illustrating reference facial expression management data stored in the moving image distribution system shown in FIG. 11. FIG. 12 is a diagram illustrating registered facial expression management data stored in the moving image distribution system shown in FIG. 11. It is a block diagram which shows the moving image distribution system by other embodiment. It is a block diagram which shows the moving image distribution system by other embodiment. 7 is a table showing an example of feature amounts at a plurality of feature points in each frame.

Various embodiments of the present invention will be described below with reference to the drawings. The same or similar components are denoted by the same reference symbols in the drawings.

A moving image distribution system according to an embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing a moving image distribution system 1 according to one embodiment. FIGS. 2, 4a, and 4b are diagrams conceptually showing a three-dimensional skeleton model, and FIGS. FIG. 6 is a diagram for explaining information stored in the moving image distribution system 1.

The moving image distribution system 1 includes a moving image reproduction device 10 and a moving image distribution device 20. The video playback device 10 and the video distribution device 20 are connected to each other via a network 50 so that they can communicate with each other. The moving image distribution system 1 may include a storage 60. A viewing user who is a user of the moving picture reproducing apparatus 10 can watch the moving picture distributed from the moving picture distributing apparatus 20 by the moving picture reproducing apparatus 10. The video distribution system 1 may include two or more video playback devices. The viewing user who views the video distributed from the video distribution device 20 on the video playback device 10 can display his or her avatar on the video. In other words, the viewing user can participate in the video distributed via his/her avatar. The viewing user can display his or her avatar on the distributed video and interact with the distribution user (or the character in the distributed video) of the video or another viewing user via the avatar.

First, the moving image reproducing apparatus 10 will be described. The video playback device 10 is an information processing device such as a smartphone. The moving image reproducing apparatus 10 may be a mobile phone, a tablet terminal, a personal computer, an electronic book reader, a wearable computer, a game console, and various information processing apparatuses other than these which can reproduce moving images.

The video playback device 10 is used by a viewing user for viewing a video and other purposes. The moving image reproducing apparatus 10 includes a computer processor 11, a communication I/F 12, a storage 13 that stores various information, a display 14 that displays a reproduced moving image, and a sensor unit 15. The moving image distribution device 10 may include other components such as a sound collecting microphone. The video playback device 10 plays the video delivered from the video delivery device 20.

The computer processor 11 is an arithmetic unit that loads an operating system and various programs that realize various functions from a storage 13 or another storage into a memory, and executes instructions included in the loaded programs. The computer processor 11 is, for example, a CPU, MPU, DSP, GPU, various arithmetic devices other than these, or a combination thereof. The computer processor 11 may be realized by an integrated circuit such as an ASIC, PLD, FPGA, MCU or the like. Although the computer processor 11 is illustrated as a single component in FIG. 1, the computer processor 11 may be a collection of a plurality of physically separate computer processors. In the present specification, the program described as being executed by the computer processor 11 or the instructions included in the program may be executed by a single computer processor or may be executed by a plurality of computer processors in a distributed manner. Good. Further, the program executed by the computer processor 11 or the instructions included in the program may be executed by a plurality of virtual computer processors. The functions implemented by the computer processor 11 will be described later.

The communication I/F 12 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof. The video playback device 10 can send and receive data to and from other devices via the communication I/F 12.

The storage 13 is a storage device accessed by the computer processor 11. The storage 13 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various storage devices other than the above that can store data. Various programs may be stored in the storage 13. At least a part of the programs and various data that can be stored in the storage 13 may be stored in a storage physically separate from the moving image playback apparatus 10 (for example, the storage 60).

The display 14 has a display panel and a touch panel. The display panel is a liquid crystal panel, an organic EL panel, an inorganic EL panel, or any other display panel capable of displaying an image other than these. The touch panel is configured to be able to detect a touch operation (contact operation) by the player. The touch panel can detect various touch operations such as tapping, double tapping, and dragging by the player. The touch panel may include an electrostatic capacitance type proximity sensor and may be configured to be able to detect a non-contact operation by the player. The moving image distributed from the moving image distribution device 20 is displayed on the display 14.

The sensor unit 15 includes one or a plurality of sensing devices that detect the posture of the user of the video playback device 10. The sensing device may include an RGB camera, a depth sensor, and other devices that can detect the posture of the user. The sensor unit 15 may include a computer processor. The computer processor of the sensor unit 15 may generate three-dimensional skeleton data representing the posture of the user by analyzing the data acquired by the sensing device. The computer processor included in the sensor unit 15 may generate the three-dimensional skeleton data by executing dedicated software. The three-dimensional skeleton data representing the user's posture detected by the sensor unit 15 is an example of the “posture feature amount” described in the claims. The sensor unit 15 may generate three-dimensional skeleton data by using the Light Coding technology like kinect (trademark) provided by Microsoft Corporation. That is, even if the sensor unit 15 irradiates a target such as a user with an infrared random pattern, acquires the depth of the target by analyzing the image, and analyzes the depth to generate three-dimensional skeleton data. Good. The feature amount representing the user's posture detected using the Light Coding technique is not limited to the three-dimensional skeleton data, and other feature amounts representing the user's posture may be used.

The three-dimensional skeleton data is data for representing a human body or a part of a human body by a three-dimensional skeleton model. The three-dimensional skeleton model models a skeleton of the human body or a part of the human body by a plurality of bones corresponding to the axis of the bone of the human body and a plurality of joints connecting the bones. The three-dimensional skeleton model will be further described with reference to FIG. FIG. 2 is an explanatory diagram for explaining the concept of the three-dimensional skeleton model. In the example shown in FIG. 2, the three-dimensional skeleton model 100 includes eight bones and joints that connect these bones. The bone of the three-dimensional skeleton model is represented as a line segment connecting two points on the three-dimensional coordinate. Therefore, each bone is represented by three-dimensional vectors V1 to V8 that represent effective line segments corresponding to each bone in the three-dimensional space (XYZ space). The 3D skeleton model forming the 3D skeleton model includes a 3D vector representing each of the bones included in the 3D model. FIG. 2 shows a three-dimensional skeleton model of the upper half of the human body, but the three-dimensional skeleton model may be a model of the entire skeleton of the human body, or a part other than the upper half of the human body ( For example, it may be a model of an arm, a lower body, or the like. Further, in the example of FIG. 2, the upper half of the human body is represented by eight bones, but the upper half of the human body may be represented by more or less than eight bones. A three-dimensional skeletal model that includes a greater number of bones may include bones that represent finger bones. Thereby, the movement of the finger can also be detected.

The information stored in the storage 13 will be described. In the illustrated embodiment, the storage 13 stores the reference posture management data 13a and other various information necessary for using the services provided by the moving image distribution apparatus 20. In the moving image distribution system 1 according to the embodiment, the reference posture of the user is set in advance. In the moving image distribution system 1, only one reference posture may be set, or a plurality of reference postures may be set. Each of the one or more reference poses may be a set of poses to identify a series of motions of the user. For example, the reference posture includes a start posture that indicates a posture of the user at the start of a series of actions associated with the reference posture, and a trigger posture that indicates a specific posture after the start posture is taken in the series of actions. May be included. At least one of the start posture and the trigger posture may be plural.

An example of the reference attitude management data 13a is shown in FIG. In the example shown in FIG. 3, the reference posture includes the start posture and the trigger posture. Therefore, in the storage 13, the start posture data indicating the start posture included in the reference posture and the trigger posture indicating the trigger posture included in the reference posture are associated with the reference posture identification data for identifying the reference posture. The data and can be stored.

The reference posture identification data is, for example, a reference posture ID that identifies the reference posture. The reference posture ID is an identification code represented by several bits, for example. The number of bits of the reference posture ID can be determined according to the number of reference postures used in the moving image distribution system 1. The reference posture ID is represented by data having an information amount of 10 bits or less, for example. The reference attitude ID may be represented by data of 5 bits or less, 4 bits or less, 3 bits or less, or 2 bits or less. Therefore, the reference posture identification data (reference posture ID) has a significantly smaller data amount than the three-dimensional skeleton data.

The start attitude data is data indicating the start attitude. The start posture data is, for example, three-dimensional skeleton data indicating the start posture. An example of a starting posture is shown in Figure 4a. The starting posture in the illustrated example is a posture in which the right palm is projected forward so that the right fist is at the same height as the right shoulder. The starting posture data may be three-dimensional skeleton data obtained by modeling the starting posture. The start posture data may include three-dimensional vector data indicating the position and orientation of each bone in the start posture. The three-dimensional skeleton data indicating the starting posture may include eight bones as schematically shown in FIG. 4a. The vectors T1 to T8 representing the eight bones respectively correspond to the bone vectors V1 to V8 included in the three-dimensional skeleton data of the viewing user generated based on the detection data of the sensor unit 15. When data other than the three-dimensional skeleton data is used as the posture feature amount, the start posture data is data that represents the start posture among the used posture feature amounts.

The trigger attitude data is data indicating the trigger attitude. The trigger posture data is, for example, three-dimensional skeleton data indicating the trigger posture. An example of a trigger pose is shown in Figure 4b. The trigger posture in the illustrated example is a posture in which the right arm is pushed forward obliquely upward. The trigger posture data may be three-dimensional skeleton data obtained by modeling the trigger posture. The three-dimensional skeleton data indicating the trigger posture may include three-dimensional vector data indicating the position and orientation of each bone. The three-dimensional skeleton data indicating the trigger posture may include eight bones as schematically shown in FIG. 4b. The eight bone vectors T1 to T8 respectively correspond to the bone vectors V1 to V8 included in the 3D skeleton data of the viewing user generated based on the detection data of the sensor unit 15. When data other than the three-dimensional skeleton data is used as the posture feature amount, the trigger posture data is data representing the trigger posture in the used posture feature amount.

Next, the moving image distribution device 20 will be described. The moving image distribution device 20 is, for example, a moving image distribution server that distributes moving images to the moving image reproduction device 10 via the downlink of the network 50. The moving image distribution apparatus 20 includes a computer processor 21, a communication I/F 22, and a storage 23 that stores various types of information. The moving image distribution apparatus 20 may include components other than the above, such as a sound collection microphone.

The computer processor 21 is an arithmetic unit that loads an operating system and various programs that realize various functions from a storage 23 or another storage into a memory and executes an instruction included in the loaded program. The above description of the computer processor 11 also applies to the computer processor 21 as much as possible. The functions implemented by the computer processor 21 will be described later.

The communication I/F 22 is implemented as hardware, firmware, communication software such as a TCP/IP driver or a PPP driver, or a combination thereof. The video distribution device 20 can send and receive data to and from other devices via the communication I/F 22.

The storage 23 is a storage device accessed by the computer processor 21. The storage 23 is, for example, a magnetic disk, an optical disk, a semiconductor memory, or various storage devices other than the above that can store data. Various programs may be stored in the storage 23. At least a part of the programs and various data that can be stored in the storage 23 may be stored in a storage physically separate from the moving image distribution apparatus 20 (for example, the storage 60).

The storage 23 may store the object data 23a, the avatar data 23b, the animation management data 23c, and various kinds of information necessary for generating and distributing moving images other than the above.

The object data 23a may include asset data for constructing a virtual space forming a moving image. The object data 23a is data for drawing the background of the virtual space forming the moving image, data for drawing various objects displayed in the moving image, and for drawing various objects displayed in the moving image other than these. Contains data. The object data 23a may include object position information indicating the position of the object in the virtual space. In addition to the above, the object data 23a may include data indicating a gift object displayed in a moving image based on a display request from the viewing user of the moving image reproducing apparatus 10. The gift object may include an effect object, a normal object, and a decoration object. The viewing user can purchase the desired gift object. The details of the gift object displayed in the moving image are described in the specification of Japanese Patent No. 6446154. Also in the moving image distribution system 1 of the present application, the gift object can be displayed in the moving image similarly to the description in the specification of Japanese Patent No. 6446154.

An example of the avatar data 23b is shown in FIG. As shown in FIG. 5, the avatar data 23b can include avatar identification information of an avatar used by the viewing user in the video distribution system 1 and avatar display information for displaying the avatar in the video. .. In other words, the storage 23 can store the avatar display information for displaying the avatar in association with the avatar identification information of the avatar. The avatar identification information is, for example, an avatar ID that identifies an avatar. The user of the video playback device 10 can set his or her own avatar in the video delivery system 1. In order to manage the avatar for each user, the avatar ID may be stored in the storage 23 in association with the user ID that identifies the user. The avatar is displayed in the moving image as an image simulating a human being or an animal, for example. The avatar display information is information used to display the avatar in the moving image. The avatar information includes, for example, part information indicating images of heads, hairstyles, facial parts (eyes, nose, mouth, etc.), torso, clothes, accessories, items, and parts other than these, which form avatars. Skin information for specifying the appearance of other avatars is included. The user can register his or her avatar by selecting a desired part image. The avatar display information may include 2D display information for displaying the avatar in 2D on the moving image and 3D display information for displaying the avatar in 3D on the moving image. The 3D display information includes part information indicating an image of a part for stereoscopically displaying an avatar in a moving image, bone data for expressing a three-dimensional movement of the avatar, and avatars other than these three-dimensionally. It includes known information used to display.

As described above, the moving image distributed from the moving image distribution device 20 may include the animation of the avatar of the user of the moving image reproduction device 10. The animation of the avatar of a certain user may be generated so as to reflect the posture and movement of the user in real time, based on the posture feature amount indicating the posture of the user, as described later.

A registered animation that is registered in advance may be adopted as the avatar animation. The registration animation is not generated in real time during the distribution of the video so as to follow the movement of the viewing user, but is registered or defined in advance before the distribution of the video or before the animation of the avatar needs to be displayed. It is an animation. An example of the animation management data 23c for managing the registered animation is shown in FIG. As illustrated, the animation management data 23c has reference posture identification data, registered animation identification data (registered animation ID) for identifying a registered animation, and animation definition data for identifying an avatar animation. In the storage 23, registered animation identification data and animation definition data are stored in association with the reference posture identification data. The reference posture identification data may be the reference posture ID as described above. When a plurality of reference posture IDs are used in the moving image distribution system 1, a plurality of registered animations may be registered according to the number of the reference posture IDs. One registration animation selected from the plurality of registration animations may be included in the moving image. The animation definition data may be data describing time-series bone data indicating the position and orientation of the avatar bone.

The registration animation may be generated based on a request from the viewing user. Hereinafter, for convenience of description, a registered animation registered based on a request from a viewing user is referred to as a “user animation”. When the viewing user issues a user animation registration request, the video playback device 10 can prompt the viewing user to perform a motion corresponding to the animation desired to be registered. The moving image reproduction apparatus 10 acquires the posture feature amount (for example, three-dimensional skeleton data) of the posture taken by the viewing user in response to the display of the image representing the reference posture for a predetermined time. The acquisition of the posture characteristic amount is performed by the posture data acquisition unit 11b or the sensor unit 15 at predetermined sampling time intervals. In this way, the animation definition data indicating the movement corresponding to the animation that the viewing user desires to register is obtained. This animation definition data may have the same data format as the animation definition data of the registered animation stored as the animation management data 23c. The video playback device 10 transmits the animation definition data defining the user animation thus obtained to the video distribution device 20, and registers this user animation in the video distribution device 20 as a new registration animation.

Upon receiving the animation definition data defining the user animation, the video distribution device 20 stores the user animation in the storage 23 as a part of the animation management data 23c. Specifically, the video distribution device 20 issues a registration animation ID that identifies the user animation for which registration is requested, and stores the animation definition data received from the video reproduction device 10 in association with the registration animation ID. .. In addition, the video distribution device 20 requests the video playback device 10 to determine a reference posture that serves as a trigger for including the user animation whose registration is requested in the video.

In response to a request from the video distribution device 20, the video playback device 10 determines a reference posture that serves as a trigger for causing its own avatar to perform a motion according to a newly registered user animation. In this specification, the reference posture that is a trigger for moving the avatar according to the newly registered user animation may be referred to as an additional reference posture. The additional reference posture is determined by various methods. For example, a sample moving image including a newly registered user animation is generated, and one or more candidate frames are selected from a plurality of frames included in this sample moving image to the viewing user who has requested registration of the user animation. Then, the additional reference posture can be determined based on the image of the avatar included in the selected frame. The viewing user can select a favorite frame from a plurality of frames. The viewing user may select, for example, a frame that includes an avatar that is in a memorable posture, a frame that includes an avatar that has a characteristic posture, or a frame other than these. it can. When two frames are selected from the plurality of frames, the posture of the avatar included in the preceding frame of the two frames in time series is determined as the start posture, and the start posture is determined in time series. The posture of the avatar included in a certain frame may be determined as the trigger posture.

Another method of determining the additional reference posture will be described. The video playback device 10 instructs the viewing user who has made the user animation registration request to perform a plurality of movements corresponding to the user animation. This instruction can be given by voice or screen display. The moving image reproducing apparatus 10 acquires the posture feature amount (for example, three-dimensional skeleton data) corresponding to the movement performed by the viewing user in response to this instruction. Specifically, three-dimensional skeleton data representing the posture of the viewing user moving in response to the user animation is acquired at a predetermined sampling interval in a predetermined measurement period. As a result, two sets of three-dimensional skeleton data representing time series the posture of the viewing user who has performed the movement corresponding to the user animation from the start of measurement to the end of measurement are obtained. Next, among the two sets of three-dimensional skeleton data, the three-dimensional skeleton data acquired at the same timing after the measurement is started are compared with each other, and the additional reference posture is determined based on the comparison result. For example, of the two sets of three-dimensional skeleton data, the total value of the angles formed by the corresponding bone vectors of the three-dimensional skeleton data acquired at the same timing after the start of measurement is calculated, and the total value of the angles is the smallest. The posture corresponding to the three-dimensional skeleton data (whichever of the two sets of skeleton data may be adopted) can be set as the additional reference posture.

Next, the function of the video playback device 10 will be described. The functions of the video playback device 10 are realized by the computer processor 11 executing computer-readable instructions included in the program. The computer processor 11 executes the computer-readable instructions included in the program to function as the moving image reproduction unit 11a, the posture data acquisition unit 11b, the transmission unit 11c, the classification unit 11d, and the delay monitoring unit 11e. At least a part of the functions implemented by the computer processor 11 may be implemented by a computer processor other than the computer processor 11. At least a part of the functions implemented by the computer processor 11 may be implemented by the computer processor 21 of the video distribution device 20 or a computer processor provided in the other video distribution system 1.

The moving image reproduction unit 11a reproduces the moving image distributed from the moving image distribution device 20. The reproduced moving image is displayed on the display 14.

The attitude data acquisition unit 11b acquires an attitude feature amount that represents the characteristics of the attitude of a user who views a video using the video playback device 10 (sometimes referred to as a “viewing user”). The posture feature amount representing the characteristic of the posture of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user, that is, three-dimensional vector data indicating the position and orientation of the bone of the viewing user. The posture data acquisition unit 11b generates three-dimensional skeleton data of the viewing user based on the detection data detected by the sensor unit 15, for example. The 3D skeleton data of the viewing user may be generated at predetermined sampling time intervals. When the sensor unit 15 includes a processor and the processor of the sensor unit 15 generates three-dimensional skeleton data, the posture data acquisition unit 11b does not have to be executed as a function of the processor 11.

The posture feature amount acquired by the posture data acquisition unit 11b may be a feature amount other than the three-dimensional skeleton data. For example, as described above, a target such as a user may be irradiated with a random pattern of infrared rays by using the Light Coding technique, the depth of the target may be acquired by analyzing the image, and the depth may be used as the posture feature amount. The detection value detected by the sensor unit 15 and various values calculated based on this detection value can be used as the posture feature amount.

The posture data acquisition unit 11b may obtain the posture feature amount as follows from a captured image including a plurality of frames obtained by capturing the movement of the user of the video playback device 10 at a predetermined frame rate. Specifically, the posture data acquisition unit 11b extracts a plurality of feature points related to the user in each frame of the captured image. Positions suitable for expressing the posture and movement of the user are extracted as feature points. A plurality of feature points may be referred to as a feature point group. FIG. 7 schematically shows a frame f1 which is one of a plurality of frames forming a captured image captured by the moving image reproducing apparatus 10. As illustrated, the frame f1 includes the captured image U1 of the user. P1 to P6 shown in FIG. 7 indicate the extracted feature points, respectively. The number and position of the characteristic points are not limited to those shown in FIG. 7. For example, when a random dot pattern is projected onto the body or face of the user of the moving image reproducing apparatus 10 by an infrared laser and the user on which the random dot pattern is projected is imaged by a camera capable of capturing an infrared region, the random dot pattern is used. The whole becomes a feature point group.

The posture data acquisition unit 11b can obtain the posture feature amount (image vector) for each of the plurality of extracted feature points. The posture feature amount at each of the feature points P1 to P6 may be the depth of each feature point or may be a three-dimensional coordinate. When three-dimensional coordinates are used as the posture feature amount, a three-dimensional world coordinate system is set in the captured image, and the relative amount normalized based on the captured image and the world coordinate system is set as the posture feature amount. It The normalized posture feature amount is represented by a relative amount of 0 to 1.0, for example. The posture feature amount of each of the feature points P1 to P6 in the frame f1 is obtained in the frame f1 and the posture feature amount of each of the feature points P1 to P6 obtained in the frame (for example, immediately before) in time series before the frame f1. The change amount may be represented by the difference between the calculated feature points P1 to P6 and the posture feature amount. When it is necessary to distinguish the two when the change amount of the posture feature amount between frames is used as another posture feature amount, the posture feature amount in a certain frame is called “intra-frame feature amount” and The posture feature amount represented by the change amount of the feature amount may be referred to as an “interframe feature amount”. Unless otherwise specified or unless otherwise understood in the context, the term "posture feature amount" includes both "intra-frame feature amount" and "inter-frame feature amount". The posture feature amount of each of the feature points P1 to P6 may be normalized in the range of 0 to 1.0 and expressed as a float array. In this case, the posture feature amount at each of the feature points P1 to P6 becomes an element of the array.

The transmission unit 11c transmits, to the video distribution device 20, the posture feature amount representing the posture of the viewing user acquired by the posture data acquisition unit 11b or the sensor unit 15. When the transmitting unit 11c receives the posture characteristic amount from the posture data acquiring unit 11b or the sensor unit 15, the transmitting unit 11c immediately transmits the posture characteristic amount to the video distribution device 20. In other words, the transmission unit 11c can transmit the posture feature amount of the viewing user to the video distribution device 20 in real time. As described above, the posture feature amount is acquired at a predetermined sampling time interval or a predetermined frame rate. Thereby, the posture feature amount can be generated for each predetermined sampling time interval or each frame rate. Therefore, the posture feature amount of the viewing user continuously acquired over a predetermined time interval can represent the body movement of the viewing user in time series as digital data. The posture feature amount indicating the posture of the viewing user may be transmitted to the video distribution device 20 together with data other than the posture feature amount. In the present specification, data representing the posture or facial expression of the viewing user or data having a correlation with the posture or facial expression of the viewing user may be collectively referred to as “motion data” of the viewing user. The motion data of the viewing user is transmitted from the video playback device 10 to the video distribution device 20 via the transmission path including the network 50. The motion data may be transmitted in packets. That is, the transmission unit 11c may transmit a packet including motion data to the video distribution device 20.

The classifying unit 11d according to the embodiment uses a classifier that classifies the posture of the viewing user, and a reference in which the posture of the viewing user is predetermined based on the posture feature amount of the viewing user (for example, three-dimensional skeleton data). It is determined whether or not it belongs to the posture. This classifier is, for example, a linear classifier. When the reference posture includes the start posture and the trigger posture, the classification unit 11d can respectively determine whether the posture of the viewing user belongs to the reference posture and whether the posture of the viewing user belongs to the trigger posture. The classification unit 11d determines whether or not the posture of the viewing user belongs to the start posture of the reference postures at the time t1, and when it is determined that the posture of the user at the time t1 belongs to the start posture, Subsequently, it may be determined whether or not the posture of the viewing user at time t2 after time t1 belongs to the trigger posture.

In one embodiment, the classification unit 11d displays an image representing the reference attitude on the display 14 in order to obtain the teacher data, and displays audio or screen to the viewing user so as to take the attitude according to the image representing the reference attitude. Can be instructed through. The moving image reproducing apparatus 10 sequentially acquires the posture feature amount of the movement performed by the viewing user in response to this instruction, by the posture data acquisition unit 11b or the sensor unit 15. The classifying unit 11d can create a classifier by learning the posture feature amount of the posture taken by the viewing user as teacher data in response to the image representing the reference posture.

The classification unit 11d in one embodiment determines whether or not the posture of the viewing user indicated by the posture feature amount from the posture data acquisition unit 11b or the sensor unit 15 matches a predetermined reference posture, for example. Can be determined as a variable based on the evaluation function. When it is determined that the orientation of the viewing user matches the reference orientation based on the evaluation function, the orientation of the viewing user belongs to the reference orientation. Whether or not the posture of the viewing user matches the start posture is determined by the posture data acquisition unit 11b or the sensor unit 15 and represents the posture feature amount indicating the posture of the viewing user and the posture feature amount indicating the start posture (start posture data). Can be determined based on When the posture feature amount is three-dimensional skeleton data, the vectors V1 to V8 of each bone included in the three-dimensional skeleton data of the viewing user and the vectors T1 to T1 of each bone corresponding to the vectors V1 to V8 of the start posture data. The smaller the total value of the angles formed with T8, the higher the degree of similarity between the posture of the viewing user detected by the sensor unit 15 and the start posture. That is, there is a negative correlation between the size of the angle θ formed by the three-dimensional vector included in the three-dimensional skeleton data representing the posture of the viewing user and the corresponding vector of the three-dimensional skeleton model representing the reference posture and the similarity of the posture. There is. Therefore, paying attention to this point, a new evaluation function f for normalizing the magnitude of the angle θ defined by the two three-dimensional skeleton models is defined. An example of the evaluation function f is shown in the following formula (1).

In the above equation, (Ti·Vi) represents the inner product of the corresponding vectors, (‖Ti‖‖Vi‖) represents the product of the magnitudes of the respective vectors, and k represents the vector that constitutes the three-dimensional skeleton model. Indicates the number of elements in. That is, according to the evaluation function f shown in the above equation (1), the average value of the cosine (=cos θ) of the angle θ formed by the corresponding vectors of the two three-dimensional skeleton models can be obtained. Here, when the corresponding vectors of the two 3D skeleton models completely match (angle θ=0), cos θ=1, and the directions of the corresponding vectors of the two 3D skeleton models are opposite. Since (angle θ=180°) is cos θ=−1, the range of the value of the evaluation function f is −1≦f≦+1. In this case, for example, if the range of the value of the evaluation function f (−1≦f≦+1) is linearly assigned to the percentage value (0 to 100%), the similarity can be expressed as a percentage. This predetermined threshold can be set to 90%, for example. The determination of the similarity between two pieces of three-dimensional skeleton data is also disclosed in Japanese Patent Laid-Open No. 2013-37454.

When the degree of similarity between the posture feature amount representing the posture of the viewing user and the posture feature amount representing the start posture calculated as described above is equal to or greater than a predetermined threshold, the classification unit 11d sets the posture of the viewing user to the start posture. It may be determined that they match.

Whenever the posture data acquisition unit 11b or the sensor unit 15 acquires the posture feature amount of the viewing user, the classification unit 11d determines whether the posture of the viewing user indicated by the posture feature amount belongs to the start posture. Good. In this case, it is determined whether the posture of the viewing user matches the start posture at the same frequency as the sampling rate for acquiring the posture feature amount of the viewing user.

Whether or not the posture of the viewing user matches the trigger posture can be similarly determined. That is, the classification unit 11d calculates the degree of similarity between the posture feature amount indicating the posture of the viewing user and the posture feature amount indicating the trigger posture at time t2, and when the degree of similarity is equal to or higher than a predetermined threshold, It may be determined that the posture matches the trigger posture. The above description for determining whether or not the posture of the viewing user matches the starting posture applies to the process for determining whether or not the posture of the viewing user matches the trigger posture as much as possible.

The delay monitoring unit 11e monitors the transmission delay of the motion data transmitted from the video playback device 10 to the video distribution device 20 by the transmission unit 11c. For example, the delay monitoring unit 11e adds a time stamp to a real packet including motion data before transmission, adds the time stamp added at the time of transmission, and the time stamp added when the real packet is received by the video distribution device 20. It is possible to obtain the transmission time of the actual packet on the transmission path between the moving image reproduction apparatus 10 and the moving image distribution apparatus 20 by using the time stamps. The delay monitoring unit 11e can determine that a delay has occurred in the transmission path when the transmission time is longer than or equal to a predetermined reference time or longer than the reference time. The delay monitoring unit 11e eliminates the delay when the transmission time becomes shorter than a predetermined reference time or becomes less than or equal to the predetermined reference time after determining that the delay has once occurred in the transmission path. Can be determined. When the video distribution device 20 is a server and the video reproduction device 10 is a client, data transmission from the video reproduction device 10 to the video distribution device 20 is performed using the uplink. In this case, the delay monitoring unit 11e monitors whether or not there is a delay in the upstream line of the transmission path between the video playback device 10 and the video distribution device 20. The delay monitoring unit 11e may measure the transmission time of the actual packet as described above, or may measure the transmission time using a pseudo packet that does not include motion data. While the transmission of the motion data transmitted from the video playback device 10 of a certain viewing user is delayed, the transmission of the motion data transmitted from the video playback device 10 of another viewing user is delayed. It is possible that it is determined that there is not. Therefore, the delay on the transmission path may be determined for each viewing user. Further, whether or not a transmission delay has occurred may be determined by the moving image distribution apparatus 20 based on the time stamp included in the packet. The video playback device 10 may receive the determination result of the transmission delay in the video distribution device 20.

The function of the transmission unit 11c will be described again. The transmission unit 11c can transmit the reference attitude identification data (reference attitude ID) stored in the storage 13 when the delay monitoring unit 11e determines that a delay has occurred in the transmission path. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As described above, the reference attitude identification data may include start attitude data and trigger attitude data. The transmitter 11c can transmit the trigger posture data as the reference posture identification data. In one embodiment, when the classification unit 11d determines that the posture of the viewing user belongs to the trigger posture, the transmission unit 11c determines the trigger posture as the reference posture identification data while the transmission of the motion data is delayed. The reference posture ID to be identified is transmitted to the moving image distribution device 20. When the transmitting unit 11c determines that the posture of the viewing user belongs to the trigger posture within a predetermined interval after the classification unit 11d determines that the posture of the viewing user belongs to the start posture, the transmitting unit 11c performs the motion. While the data transmission is delayed, the reference posture ID for identifying the trigger posture is transmitted to the video distribution device 20. If the posture of the viewing user is determined to belong to the start posture and then the posture of the viewing user is also determined to belong to the trigger posture, the reference posture ID is transmitted to belong to only one of the start posture and the trigger posture. It is possible to prevent or suppress the viewing user from unintentionally transmitting the reference posture ID as compared with the case where the reference posture ID is transmitted in response to the determination. "While a delay occurs in transmission of motion data" means a period from when the delay monitoring unit 11e determines that a delay has occurred in the transmission path to when it is determined that the delay has been eliminated. May be. The transmission unit 11c does not have to transmit the reference posture ID for identifying the start posture to the video distribution device 20.

Next, the function realized by the computer processor 21 will be described more specifically. The computer processor 21 functions as a moving image generation unit 21a, a moving image distribution unit 21b, and an animation generation unit 21c by executing computer-readable instructions included in the distribution program. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the moving image distribution system 1. At least a part of the functions implemented by the computer processor 21 may be implemented by, for example, the computer processor 11 of the video playback device 10 or a computer processor included in the other video delivery system 1.

The moving picture distribution device 20 can distribute various kinds of moving pictures. In the following, it is assumed that the video distribution device 20 distributes a video including an animation of a character object generated based on the movement of an actor.

The moving image generation unit 21a detects the movement of the actor by a motion sensor attached to the actor. The moving image generation unit 21a can generate an animation of a character that moves in synchronization with the movement of the actor's body detected by the motion sensor. The moving image generation unit 21a may acquire face motion data that is a digital expression of the movement of the face of the actor. In this case, the moving image generator 21a can generate an animation of a character whose facial expression changes in synchronization with the movement of the face of the actor. The moving image generation unit 21a can construct a virtual space using the object data 23a, and can generate a moving image including the virtual space and the animation of the character corresponding to the actor. The moving image generation unit 21a can synthesize the generated moving image with the voice of the actor acquired from the microphone. The generation of a moving image including animation of a character that moves in synchronization with the movement of the actor's body or facial expression is disclosed in detail in the specification of Japanese Patent No. 644615.

Upon receiving a participation request for allowing an avatar to participate in the video from the viewing user who is watching the video, the video generation unit 21a can generate the video including the avatar of the viewing user. The participation request from the viewing user may include a user ID that identifies the viewing user. The moving image generation unit 21a identifies an avatar ID for identifying the avatar of the viewing user based on the user ID included in the participation request, and based on the parts information stored in the storage 23 in association with the avatar ID. Thus, it is possible to generate an avatar object representing the avatar of the viewing user.

The moving image distribution unit 21b distributes the moving image generated by the moving image generation unit 21a. This video is distributed to the video playback device 10 via the network 50. The received moving image is reproduced by the moving image reproducing device 10.

FIG. 9a shows a display example of a moving image that is distributed from the moving image distribution device 20 to the moving image reproduction device 10a and is reproduced by this moving image reproduction device 10. The moving image reproduced by the moving image reproducing device 10 may be displayed on the display 14. As shown in the figure, the moving image 60 displayed on the moving image reproducing apparatus 10 includes a character object 51 representing an actor character, a floor object 54a on which the character object 51 stands, and a screen object 54b defining the rear end of the stage. The avatar objects 56a to 56j indicating the avatars of the viewing user who is viewing the moving image 60 are included. The character object 51 can move in the virtual space in synchronization with the movement of the actor's body. Since the moving image 60 is distributed from the moving image distribution device 20 to a large number of moving image reproducing devices 10, the image 60 includes a large number of avatars. In FIG. 9a, it is assumed that 10 viewing users are participating in the video via avatars. The number of viewing users who can participate in the video may be greater than 10, or at least greater than 10.

On the display 14, an operation button for accepting a user operation may be displayed so as to overlap with the moving image 60. In the example shown in FIG. 9a, a gift button 61 for giving a gift, an evaluation button 62 for providing an evaluation, an avatar participation button 63 for applying for participation in the video 60 by an avatar, Is superimposed on the moving image 60. The gift button 61, the evaluation button 62, and the avatar participation button 63 are displayed so that the viewing user can select them. Operation buttons other than these may be displayed on the moving image 60. The viewing user who is viewing the video 60 presents a desired gift to the distributor who is delivering the video 60 or the actor who appears in the video 60 via the character 51 by selecting the gift button 61. You can By selecting the evaluation button 62, the viewing user can transmit evaluation information indicating that the video 60 is positively evaluated to the video distribution device 20. The evaluation information from various viewing users may be totaled, and the totaled result may be displayed together with the moving image 60. By selecting the avatar participation button 63, the viewing user can send a participation request to the video distribution device 20 requesting that his or her avatar participate in the video 60.

The animation generation unit 21c generates an animation of an avatar of a viewer participating in the moving image 60. In one embodiment, the animation generation unit 21c generates an animation of an avatar of the viewing user based on the posture feature amount of the viewing user received from the video playback device 10. The posture feature amount (for example, three-dimensional skeleton data) received continuously in time represents the movement of the body of the viewing user in time series. Therefore, the animation generation unit 21c continuously receives the posture feature amount of the viewing user from the moving image reproducing apparatus 10, and thereby the avatar that moves in synchronization with the movement of the body of the viewing user based on the posture feature amount. Animations can be generated. When the animation generation unit 21c generates the avatar animation, the moving image generation unit 21a generates the moving image 60 so as to include the avatar animation.

When a transmission delay occurs in the upstream transmission path from the video playback device 10 to the video distribution device 20, the video distribution device 20 may not be able to continuously receive the posture feature amount of the viewing user in time. is there. When an avatar animation is created based only on the posture feature amount of the viewing user transmitted from the video playback device 10, if a transmission delay occurs in the transmission path, the animation of the avatar that moves reflecting the movement of the viewing user is generated. It may not be able to be generated. On the other hand, the animation generation unit 21c according to the embodiment does not depend only on the posture feature amount of the viewing user, but creates the avatar animation based on the reference posture identification data (for example, the reference posture ID) from the video playback device 10. Can be generated. Specifically, when the animation generation unit 21c receives the reference posture identification data from the moving image reproduction device 10, the animation generation unit 21c refers to the animation management data 23c to identify the registered animation associated with the received reference posture identification data. It is possible to generate the avatar animation based on the animation definition data associated with the registered animation ID of the specified registered animation. As described above, the reference posture identification data is transmitted from the video playback device 10 when the delay monitoring unit 11e of the video playback device 10 determines that the transmission of the motion data including the posture feature amount in the uplink is delayed. It is transmitted to the video distribution device 20. Therefore, even if the posture feature amount cannot be received while the motion data transmission is delayed, the animation generation unit 21c is sent from the video playback device 10 while the delay is generated. An avatar animation can be generated based on the reference posture identification data.

Generation of avatar animation will be described. The avatar animation generated based on the posture feature amount of the viewing user (for example, three-dimensional skeleton data) expresses the movement of the avatar that reflects the movement of the viewing user. For example, it is assumed that the viewing user performs an operation of lifting the right hand on the waist and obliquely upward. In this case, the avatar animation of the viewing user is generated based on the posture feature amount of the viewing user. By including this animation in the video, the avatar performs an action of lifting the right hand (a part corresponding to the right hand of the avatar) diagonally upward from the position where the right hand (the part corresponding to the right hand of the avatar) is applied to the waist in the video. ..

On the other hand, when a delay occurs in the transmission path, the generation of the avatar animation based on the posture feature amount may be interrupted. For example, when a delay occurs in the transmission path, the transmission of the posture feature amount by the transmission unit 11c may be interrupted, and accordingly, the animation generation unit 21c generates the avatar animation based on the posture feature amount. May also be interrupted. As described above, the video distribution device 20 can determine the presence or absence of delay based on the time stamp included in the packet received from the video reproduction device 10. When the transmission path is delayed, it becomes difficult for the moving image distribution apparatus 20 to receive the posture feature amount having a large amount of data in a timely manner. Even in such a case, if the animation of the avatar based on the posture feature amount is continued, the movement of the avatar in the moving image may become unnatural. Such an unnatural movement of the avatar deteriorates the quality of the moving image to be distributed, which is not desirable. When a delay occurs in the transmission path, by interrupting the generation of the avatar animation based on the posture feature amount, it is possible to prevent the avatar from performing an unnatural movement. Even if the generation of the avatar animation based on the posture feature amount is interrupted, the animation generation unit 21c specifies the registered animation ID of the avatar based on the reference posture identification data from the video playback device 10, and the specified registered animation. An avatar animation can be generated based on the animation definition data associated with the ID. If the viewing user does not take a posture that matches the reference posture while the delay occurs in the transmission path, the moving image distribution apparatus 20 does not receive the reference posture identification data. In this case, the animation generation unit 21c does not have to generate the animation of the avatar of the viewing user. When the animation generation unit 21c does not generate an avatar animation, the avatar is still in the moving image. In another embodiment, when the moving image distribution apparatus 20 does not receive the reference attitude identification data while the delay is occurring in the transmission path, the moving image distribution apparatus 20 performs a motion according to the basic operation defined for the avatar. You may create avatar animations. The basic action of the avatar refers to a predetermined action such as shaking the hand up and down, shaking the hand left and right, and jumping. This basic operation may be set commonly to a plurality of viewing users. The basic motion of the avatar is generated by the animation generation unit 21c in that the basic motion of the avatar can be performed by the avatar without receiving the information (three-dimensional skeleton data or reference posture identification data) about the motion of the avatar from the video playback device 10. It's different from animation.

When the animation generating unit 21c generates the avatar animation, the moving image generating unit 21a generates a moving image including the animation, and the moving image including the avatar animation is distributed to the moving image reproducing apparatus 10 by the moving image distribution unit 21b. If the animation 60 shown in FIG. 9a includes the animation of the avatar object 56a, the animation 60 including the animation of the avatar object 56a is displayed. For example, when the animation in which the avatar object 56a raises the right hand is generated, the avatar object 56a performs the movement of raising the right hand in the moving image 60, as shown in FIG. 9b.

As described above, the video distribution device 20 may request the video playback device 10 to determine the additional reference posture that is a trigger for including the user animation whose registration is requested in the video. Although some aspects of the method of determining the additional reference posture have already been described, another aspect of determining the additional reference posture will be described with further reference to FIG. 8. In the following example, the feature point of the image of the user is extracted, and the additional reference posture is determined using the root mean square of the posture feature amount at the feature point. First, the moving image reproducing apparatus 10 instructs a user to perform a motion to be registered as a user animation, and captures a plurality of frames by capturing an image of the user performing the motion to register based on the instruction at a predetermined frame rate. A captured image including the acquired image is obtained. The moving image reproducing apparatus 10 obtains the posture feature amount for each of the plurality of extracted feature points. The extraction of the feature points in each of the plurality of frames of the captured image of the user has already been described with reference to FIG. 7, and thus will not be repeatedly described.

FIG. 8 shows a table including posture feature values in each of 24 frames from the 0th frame to the 23rd frame. In the table of FIG. 8, the posture feature amounts of the feature points P1 to P6 are shown in the “P1” column to the “P6” column, respectively. This posture feature amount is normalized to the range of 0 to 1.0. The moving image reproducing apparatus 10 calculates the root mean square (RMS) of the posture feature amount of each of the feature points P1 to P6 for each frame. The orientation feature quantity at each of the N feature points P ₁ ~P _N x _i (where, i takes values of 1 to N.) To the, the posture feature quantity of the feature point P1~P _N RMS (x ) Is represented by the following formula.

FIG. 8 shows RMS(x) calculated for each frame. In addition to RMS(x), the moving image reproducing apparatus 10 shows the average value of the posture feature amount at each feature point and the difference between the RMS(x) of the frame and the RMS(x) of the previous frame for each frame. The RMS difference may be calculated. The moving image reproducing apparatus 10 may obtain, for each frame, a positive/negative flag indicating whether the RMS difference is positive or negative, and an inversion flag indicating whether the positive or negative of the RMS difference is reversed. For example, the RMS difference in the second frame is the difference between RMS(x ₂ ) in the second frame and RMS(x ₁ ) in the first frame. The positive/negative flag is “1” when the RMS difference is positive, and is “0” when the RMS difference is negative, for example. The inversion flag is, for example, “1” when the sign of the RMS difference is reversed, and is “0” when the sign of the RMS difference is not reversed (that is, the sign is the same as in all frames). FIG. 8 includes the average value, the RMS difference, the positive/negative flag, and the inversion flag. In FIG. 8, the positive/negative of the RMS difference is reversed in the fourth frame, the tenth frame, the sixteenth frame, the seventeenth frame, the nineteenth frame, and the twenty-first frame from the immediately preceding frame. 1” is set. Since there is no frame preceding the 0th frame, the RMS difference is blank for the 0th frame.

In the frame in which the inversion flag is "1", the sign of the RMS difference is reversed from the previous frame, so it is estimated that the user has made a large motion in the frame. The large movement that can be detected by the reversal flag is often a periodic movement such as waving a hand or blinking. Therefore, in one embodiment of the present invention, a frame in which the reversal flag is set to “1”, that is, a frame in which the sign of the RMS difference of the posture feature amount is the reverse of the previous frame is set as the start frame, and the RMS is started from this start frame. It becomes easy to pay attention to the interval until the sign of the difference is inverted again (until the inversion flag becomes “1” next time). The posture of the user in this start frame can be set as the start posture. Further, the frame in which the inversion flag is set to “1” next to the start frame can be set as the trigger frame, and the posture of the user in this trigger frame can be set as the trigger posture. In this way, the motion and posture of the user can be analyzed by paying attention to the section from when the reversal flag becomes "1" to when the next reversal flag becomes "1". In the example shown in FIG. 8, the inversion flag becomes “1” for the first time in the fourth frame, and the inversion flag becomes “1” for the second time in the tenth frame. Therefore, it is possible to extract a periodical motion in which the posture of the user in the fourth frame is the starting posture and the posture of the user in the tenth frame is the trigger posture and the extraction of the trigger posture at low calculation cost. The start posture data indicating the start posture and the trigger posture data indicating the trigger posture are calculated from root mean square error (RMSE: Root Mean Squared Error), root mean square error ratio (RMSPTE: Root Mean Squared Error), or an imaged image. Or can be estimated.

In the example shown in FIG. 8, the inversion flag is set to “1” in both the 16th frame and the 17th frame. Since the time interval between consecutive frames is extremely short, it may not be appropriate to set the user's posture in each of these adjacent frames as the start posture and the trigger posture. Therefore, a lower limit frame number, which is the lower limit of the number of frames that should exist between the start frame and the trigger frame, may be set. The lower limit number of frames may be, for example, 3 frames or more in consideration of compression of communication packets, difference from neighboring frames, and other factors. A lower bound on the time interval between the start frame and the trigger frame may be defined. The lower limit of the time interval between the start frame and the trigger frame is, for example, 1 second or more, 2 seconds or more, 3 seconds or more, 4 in consideration of the operating frequency, the target user's operation, and other factors. It may be a second or longer, 5 seconds or longer, or a lower limit other than these. If the lower limit number of frames is set to 3 and there are 5 frames exceeding the lower limit number of frames between the 4th frame and the 10th frame in FIG. 8, treat the 10th frame as a trigger frame. You can If the inversion flag is set to "1" in the seventh frame, there are only two frames of the lower limit number of frames or less between the fourth frame and the seventh frame, and therefore the seventh frame is not used as a trigger frame. Then, the tenth frame in which the inversion flag becomes "1" next can be set as the trigger frame. However, since the RMSPE from the 10th frame to the 16th frame, which is the next target frame, is approximately 1 or less, it is not necessary to judge the 10th frame as noise and process it as a trigger frame.

As described above, by defining the additional reference postures (starting posture and triggering posture) by using the root mean square of the posture feature amount at the user's feature point, the user animation is registered without presenting the candidate frame or the sample moving image. Also, the start posture and the trigger posture corresponding to the user animation can be determined.

Next, with reference to FIG. 10, a part of the processing included in the moving image reproducing method according to one aspect will be described. FIG. 10 is a flowchart showing a part of the flow of processing in the moving image reproduction method according to the embodiment. In the moving image reproducing method of FIG. 10, it is assumed that the viewing user is reproducing the moving image by the moving image reproducing apparatus 10 and has his/her avatar participate in the moving image. That is, the video that the viewing user is viewing includes his or her avatar. The process illustrated in FIG. 10 relates to transmission of the posture feature amount and the reference posture identification data to the moving image distribution device 20 while the moving image is being viewed.

First, in step S11, data regarding the posture feature amount of the viewing user who is viewing the moving image is acquired. In addition, in step S11, the acquired posture feature amount is immediately transmitted to the video distribution device 20. In step S11, motion data including the posture feature amount and other data may be transmitted to the moving image distribution device 20. The posture feature amount of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user. The posture feature amount may be acquired at predetermined sampling time intervals. Acquisition of the posture feature amount of the viewing user is performed by, for example, the posture data acquisition unit 11b or the sensor unit 15 described above. The transmission of the posture feature amount is performed by, for example, the transmission unit 11c described above.

Next, in step S12, it is determined whether or not the posture of the viewing user belongs to a predetermined reference posture, based on the posture feature amount of the viewing user acquired in step S11. This determination is performed, for example, by the classification unit 11d described above. When the posture of the viewing user does not belong to the reference posture, the process returns to step S11, and the posture feature amount is continuously acquired in step S11. When it is determined that the orientation of the viewing user belongs to any of the reference orientations, the process proceeds to step S13.

In step S13, it is determined whether or not there is a delay in the transmission of motion data on the transmission path between the moving image reproduction device 10 and the moving image distribution device 20. This determination is made, for example, by the delay monitoring unit 11e. When it is determined that the delay has not occurred in the transmission path, the process returns to step S11, and the posture feature amount is continuously acquired in step S11. If it is determined that a delay has occurred in the transmission path, the process proceeds to step S14.

In step S14, reference posture identification data for identifying the reference posture determined to include the posture of the viewing user in step S12 is transmitted to the video distribution device 20. The transmission of the reference attitude identification data is performed by, for example, the transmission unit 11c described above. When it is determined in step S12 that the posture of the viewing user belongs to the start posture and then it is determined that the posture of the viewing user further belongs to the trigger posture, trigger posture data for identifying this trigger posture is transmitted.

In parallel with the processing in each of the above steps S11 to S14, the moving image is continuously distributed from the moving image distribution device 20 to the moving image reproduction device 10. During the distribution of the moving image, the processing of steps S11 to S14 is repeated. When it is determined in step S13 that the delay has not occurred in the transmission path, the distributed moving image includes the animation of the avatar generated based on the posture feature amount of the viewing user. On the contrary, when it is determined in step S13 that a delay has occurred in the transmission path, the avatar registered in the moving image to be distributed is specified based on the reference attitude identification data transmitted to the moving image distribution device 20. Animations are included.

Step S13 may be executed before step S11 or between steps S11 and S12. Besides this, the order of the processes shown in FIG. 10 may be changed as appropriate. Further, in addition to the processing of steps S11 to S14, processing not explicitly described in FIG. 10 may be performed.

Next, another embodiment of the present invention will be described with reference to FIGS. 11 to 13. FIG. 11 shows a block diagram of a moving image distribution system 101 according to another embodiment of the present invention. The moving image distribution system 101 includes a moving image reproduction device 110 and a moving image distribution device 120. The moving image reproducing apparatus 110 of the moving image distribution system 101 is the same as the moving image reproducing apparatus 10 of the moving image distribution system 1 in that the computer processor 11 functions as the facial feature amount acquisition unit 11f and the storage 13 stores the reference facial expression management data 13b. different. The moving picture distribution apparatus 120 of the moving picture distribution system 101 is different from the moving picture distribution apparatus 20 of the moving picture distribution system 1 in that the storage 23 stores the registered facial expression management data 23d. In the following, differences between the embodiment of FIG. 11 and the embodiment of FIG. 1 will be described. Descriptions of matters in the embodiment of FIG. 11 that are common to the embodiment of FIG. 1 are omitted.

The facial feature amount acquisition unit 11f acquires the facial feature amount representing the facial feature of the viewing user who views the video using the video playback device 110. Specifically, the facial feature amount acquisition unit 11f uses the image including the face of the viewing user acquired by the sensor unit 15 to represent the facial feature according to, for example, HOG, SIFT, SURF, or a known algorithm other than these. Calculate the amount. For example, the facial feature amount calculated by SIFT is calculated as a 128-dimensional feature amount. As described above, the facial feature amount has a large data amount similarly to the posture feature amount acquired by the posture data acquisition unit 11b.

FIG. 12 shows an example of reference facial expression management data. As shown in FIG. 12, the storage 13 stores reference facial expression identification data indicating the reference facial expression in association with the reference facial expression identification data for identifying the reference facial expression. The reference facial expression identification data is, for example, a reference facial expression ID that identifies the reference facial expression. The reference facial expression ID is an identification code represented by several bits, for example. The number of bits of the reference facial expression ID can be determined according to the number of reference facial expressions used in the moving image distribution system 1. The reference facial expression ID is represented by data having an information amount of 10 bits or less, for example. The reference facial expression ID may be represented by data of 5 bits or less, 4 bits or less, 3 bits or less, or 2 bits or less. Therefore, the reference facial expression identification data (reference facial expression ID) has a significantly smaller data amount than the facial feature amount.

The reference facial expression identification data is data indicating a reference facial expression. One or more reference facial expressions may be set in the moving image distribution system 1. The reference facial expression is, for example, an expression in which the eyes are wide open with the front facing. The reference facial expression identification data is data that describes the reference facial expression and has the same data structure as the facial feature amount calculated by the facial feature amount acquisition unit 11f. When the face feature amount calculated by the face feature amount acquisition unit 11f is the SIFT feature amount calculated by the SIFT algorithm, the reference facial expression identification data is the reference facial expression described in the same data format as the SIFT feature amount. is there.

The classifying unit 11d uses a classifier that classifies the facial expression of the viewing user, and determines whether the facial expression of the viewing user belongs to a predetermined reference facial expression, based on the facial feature amount of the viewing user. In one embodiment, the classifying unit 11d may display an image representing the reference facial expression on the display 14 to obtain the teacher data, and prompt the viewing user who has the facial expression according to the image representing the reference facial expression. The content reproduction apparatus 110 acquires the facial feature amount of the facial expression created by the viewing user in response to the display of the image representing the reference facial expression by the facial feature amount acquisition unit 11f. The classifying unit 11d can create a classifier by learning the facial expression feature amount of the facial expression created by the viewing user as the teacher data for the image representing the reference facial expression.

The transmission unit 11c transmits, to the video distribution device 120, the motion data including the facial feature amount representing the facial feature of the viewing user acquired by the facial feature amount acquisition unit 11f. The transmission unit 11c distributes the reference facial expression identification data (reference facial expression ID) stored in the storage 13 to the moving image when the delay monitoring unit 11e determines that the transmission of the motion data is delayed on the transmission path. It can be transmitted to the device 120. When the delay monitoring unit 11e determines that a delay has occurred in the transmission path, the reference facial expression identification data may be transmitted to the video distribution device 120 instead of the facial feature amount.

In the moving image distribution device 20, the animation generation unit 21c generates an animation of an avatar of a viewer participating in the moving image 60. In one embodiment, the animation generation unit 21c generates an animation of an avatar of the viewing user based on the facial feature amount of the viewing user received from the video playback device 110. The animation generation unit 21c continuously receives the facial feature amount of the viewing user from the moving image reproduction apparatus 110, and in synchronization with the facial movement (change of facial expression) of the viewing user based on the facial feature amount. It is possible to generate avatar animations that change facial expressions.

The animation generation unit 21c in one embodiment refers to the registered facial expression management data 23d, and based on the standard facial expression identification data (for example, standard facial expression ID) from the video playback device 110, the animation of the avatar including the movement of the facial expression. Can be generated. FIG. 13 shows an example of the registered facial expression management data 23d in the embodiment of FIG. As shown in FIG. 13, the registered facial expression management data 23d includes the reference facial expression identification data (registered facial expression ID), the registered facial expression identification data (registered facial expression ID) for identifying the registered facial expression, and the movement of the avatar's facial expression. And animation definition data for specifying the included animation. This animation definition data is data that defines the movement of the avatar's facial expression. The animation definition data included in the registered facial expression management data 23d may be data that describes the positions of the feature points of the avatar's face in time series. When the animation generation unit 21c receives the reference facial expression identification data from the video playback device 110, the animation generation unit 21c identifies the registered facial expression identification data associated with the received reference facial expression identification data by referring to the animation management data 23d. It is possible to generate an avatar animation including facial movement based on the animation definition data associated with the identified registered facial expression identification data.

Next, another embodiment of the present invention will be described with reference to FIG. FIG. 14 shows a block diagram of a moving image distribution system 201 according to another embodiment of the present invention. Generally speaking, in the moving image distribution system 1 shown in FIG. 1, the posture feature amount representing the posture of the viewing user is transmitted in real time, whereas in the moving image distribution system 201 shown in FIG. The two are different in that the posture feature amount of the viewing user is transmitted only when the viewing user makes a characteristic motion. The moving image distribution system 201 includes a moving image reproduction device 210 and a moving image distribution device 220. In the video playback device 210 of the video distribution system 201, the computer processor 11 functions as the determination unit 11g and the transmission unit 11h. The video playback device 210 does not need to have the reference facial expression management data 13b, and the video delivery device 220 does not have to have the animation management data 23c. Descriptions of matters common to the embodiment of FIG. 1 in the embodiment of FIG. 14 will be omitted.

The determination unit 11g determines whether or not the user of the video playback device 210 has performed a characteristic motion. Specifically, the posture data acquisition unit 11b calculates the posture feature amount of the feature point of the user of the video playback device 210 at a predetermined frame rate, and the determination unit 11g determines the posture feature amount of the frame based on the RMS difference. It is determined whether the user has performed a characteristic movement. The calculation of the posture feature amount and the RMS difference for each frame has already been described with reference to FIG. In one embodiment, the determination unit 11g determines that the user has performed a characteristic motion in a frame in which the positive and negative signs of the RMS difference are reversed. As described above, when the sign of the RMS difference in a certain frame is reversed from that in the previous frame, it is estimated that the user has performed a characteristic motion involving reciprocating motion such as waving his hand or making a large nod in that frame. It For example, when the posture data acquisition unit 11b calculates the posture feature amount for 24 frames shown in FIG. 8, the fourth frame, the tenth frame, the sixteenth frame in which the inversion flag is set to “1”, It is determined that the user has performed a characteristic motion in the 17th frame, the 19th frame, and the 21st frame.

In one embodiment, the determination unit 11g determines that the user has made a large motion in a frame in which RMSPE is larger than a predetermined threshold value. This threshold can be set to 1.0, for example. The threshold used by the determination unit 11g can be changed as appropriate. In the example of FIG. 8, RMSPE is greater than the threshold value of 1.0 in the fourth to seventh frames. Therefore, the determination unit 11g can determine that the user has made a large movement in the fourth frame to the eighth frame.

The transmission unit 11h selects a frame in which the determination unit 11g has determined that the user has performed a characteristic motion, and transmits the posture feature amount in the selected frame to the video distribution device 220. The frame selected by the transmitter 11h may be called a "selected frame". The selected frame may be only a frame determined by the determination unit 11g to have the user perform a characteristic motion, or a frame determined by the determination unit 11g to be the user's characteristic motion and a subsequent frame. Alternatively, it may be a plurality of frames. As described above, the transmission unit 11h is configured to transmit the posture feature amount in the selected frame to the moving image distribution device 220, but not to transmit the posture feature amount in frames other than the selected frame to the moving image distribution device 220.

In the video distribution device 220 that has received the posture feature amount transmitted by the transmission unit 11h, the animation generation unit 21c generates an animation of the avatar of the user of the video reproduction device 210 based on the received posture feature amount. The avatar animation generation method based on the posture feature amount is the same as the generation method in the moving image distribution system 1. In the moving image distribution system 201, there is a period in which the posture feature amount is not transmitted from the moving image reproducing device 210 to the moving image distributing device 220. Specifically, the posture feature amount is not transmitted to the moving image distribution apparatus 220 in the frames other than the selected frame. During the period in which the posture feature amount is not transmitted, the animation generation unit 21c may create an animation of the avatar so that the avatar performs a motion according to the basic motion defined for the avatar. The basic operation of the avatar is as described, and it is a predetermined operation such as a hand shaking operation.

In the moving image distribution system 201, the posture characteristic amount is transmitted from the moving image reproducing device 210 to the moving image distribution device 220 only in the selected frame. Therefore, the transmitted amount of data is smaller than that of transmitting the calculated posture characteristic amount in real time. Can be reduced.

Next, another embodiment of the present invention will be described with reference to FIG. FIG. 15 shows a block diagram of a moving image distribution system 301 according to another embodiment of the present invention. The moving picture distribution system 1 differs from the moving picture distribution system 1 in that the moving picture reproducing apparatus generates a moving picture based on data necessary for generating a moving picture such as a posture feature amount. Specifically, the moving image distribution system 301 includes an attitude data acquisition device 310, a moving image distribution device 320, and a moving image reproduction device 330. The posture data acquisition device 310, the moving image distribution device 320, and the moving image reproduction device 330 are connected via the network 50. In the moving image distribution system 301, it is assumed that the moving image reproducing device 320 generates a moving image including an avatar animation generated based on the movement of the user of the posture data acquisition device 310, and reproduces the generated moving image. .. That is, the user of the posture data acquisition device 310 can reproduce a moving image including an avatar that moves based on his/her own movement in the moving image reproduction device 320 used by the viewing user. The user of the posture data acquisition device 310 can interact with the user of the video playback device 320 via an avatar. In the present embodiment, the user of the posture data acquisition device 310 may be simply referred to as a “delivery user”. For the sake of explanation, FIG. 15 illustrates the posture data acquisition device 310, the moving image distribution device 320, and the moving image reproduction device 330 one by one, but the moving image distribution system 301 may include a plurality of these devices.

The posture data acquisition device 310 acquires a posture feature amount representing the feature of the posture of the user of the posture data acquisition device 310. The posture feature amount representing the characteristic of the posture of the viewing user may be three-dimensional skeleton data representing the posture of the viewing user, that is, three-dimensional vector data indicating the position and orientation of the bone of the viewing user. The posture data acquisition device 310 may include the sensor unit 15 described above. The posture data acquisition device 310 can generate three-dimensional skeleton data of the viewing user based on the detection data detected by the sensor unit 15. The attitude data acquisition device 310 transmits the acquired attitude feature amount to the moving image distribution device 320. The posture data acquisition device 310 may transmit audio data representing the audio of the distribution user to the video distribution device 320.

The moving image distribution device 320 includes a computer processor 21, a communication I/F 22, and a storage 23, similar to the moving image distribution device 20 described above. The moving image distribution apparatus 320 differs from the moving image distribution apparatus 20 in the functions realized by the computer processor 21 and the data stored in the storage 23. Therefore, in the following, the functions realized in the computer processor 21 of the moving image distribution apparatus 320 and the data stored in the storage 23 will be described.

As illustrated, the moving image distribution system 301 storage 23 can store object data 23a, avatar data 23b, reference attitude management data 323a, and various information necessary for generating and distributing moving images other than the above. The object data 23a and the avatar data 23b may be the same as the object data 23a and the avatar data 23b stored in the moving image distribution system 1. The reference attitude management data 323a may be the same as the reference attitude management data 13a in the moving image distribution system 1. That is, as shown in FIG. 3, the reference attitude management data 323a includes reference attitude identification data for identifying a reference attitude and start attitude data and trigger attitude data stored in association with the reference attitude identification data. Can be included.

The computer processor 21 executes a computer-readable instruction to function as the posture feature amount acquisition unit 321a, the transmission unit 321b, the classification unit 321c, and the delay monitoring unit 321d. At least a part of the functions realized by the computer processor 21 may be realized by a computer processor other than the computer processor 21 of the moving image distribution system 301.

The posture feature amount acquisition unit 321 a acquires a posture feature amount representing the posture of the user of the posture data acquisition device 310 from the posture data acquisition device 310.

The transmission unit 321b transmits, to the video playback device 330, the posture feature amount representing the posture of the user of the posture data acquisition device 310 acquired from the posture data acquisition device 310. The transmission unit 321b may immediately transmit the posture feature amount to the video distribution device 320 upon receiving the posture feature amount from the posture data acquisition device 310. In other words, the transmission unit 321b can transmit the posture feature amount of the distribution user to the video distribution device 320 in real time. The transmitting unit 321b may transmit a packet including the posture feature amount to the moving image reproducing device 320. The transmitting unit 321b can transmit, to the moving image reproducing apparatus 330, data necessary for generating a moving image in the moving image reproducing apparatus 330, in addition to the posture feature amount. For example, the transmission unit 321b can transmit at least a part of the object data 23a and the avatar data 23b to the video playback device 330. The object data 23a and the avatar data 23b may be stored in the moving image reproducing apparatus 330 in advance, instead of being transmitted to the moving image reproducing apparatus 330 by the transmitting unit 321b. In this case, it is not necessary to transmit the object data 23a and the avatar data 23b from the moving image distribution device 320 to the moving image reproduction device 330. Further, the transmission unit 321b may transmit the audio data representing the voice of the distribution user, which is acquired by the moving image distribution apparatus 320 from the generated data acquisition apparatus 310, to the moving image reproduction apparatus 330.

The classification unit 321c is a classifier that classifies the posture of the distribution user, and whether the posture of the viewing user belongs to a predetermined reference posture based on the posture feature amount of the viewing user (for example, three-dimensional skeleton data). Determine whether or not. This classifier is, for example, a linear classifier. The classification unit 321c may realize substantially the same function as the classification unit 11d of the moving image distribution system 1. For example, when the reference posture includes the start posture and the trigger posture, the classification unit 321c can respectively determine whether the posture of the delivery user belongs to the reference posture and whether the posture of the delivery user belongs to the trigger posture. ..

The delay monitoring unit 321d monitors the transmission delay of the motion data transmitted from the video distribution device 320 to the video playback device 330 by the transmission unit 321c. The delay monitoring unit 321d may realize substantially the same function as the delay monitoring unit 11e of the moving image distribution system 1. For example, the delay monitoring unit 321d adds a time stamp to a real packet including motion data before transmission, adds the time stamp added at the time of transmission, and the time stamp added when the real packet is received by the video playback device 330. It is possible to obtain the transmission time of the actual packet on the transmission path between the moving picture distribution device 320 and the moving picture reproduction device 330 by using the time stamp. Whether or not a transmission delay has occurred may be determined by the video playback device 330 based on the time stamp included in the packet. The video playback device 320 may receive the determination result of the transmission delay in the video distribution device 330.

The function of the transmission unit 321b will be described again. The transmission unit 321b transmits the reference posture identification data (reference posture ID) stored in the storage 23 to the video playback device 330 when the delay monitoring unit 321d determines that a delay has occurred in the transmission path. be able to. The reference posture identification data may be transmitted instead of the motion data including the posture feature amount of the viewing user. As described above, the reference attitude identification data may include start attitude data and trigger attitude data. The function of the transmission unit 321b executed when this transmission delay occurs is similar to the function executed by the transmission unit 11c when the transmission delay occurs.

The moving image reproducing apparatus 330 includes the computer processor 11, the communication I/F 12, and the storage 13, similarly to the moving image reproducing apparatus 10 described above. The moving image reproducing apparatus 330 differs from the moving image reproducing apparatus 10 in the functions realized by the computer processor 11 and the data stored in the storage 23. Therefore, in the following, the functions realized in the computer processor 21 of the video playback device 330 and the data stored in the storage 23 will be described.

Animation management data 313a is stored in the storage 13 of the video playback device 330. The animation management data 313a may be the same as the animation management data 13a stored in the moving image distribution system 1. For example, as shown in FIG. 6, the animation management data 313a includes reference posture identification data, registered animation identification data (registered animation ID) that identifies a registered animation, and an animation definition for identifying an avatar animation. And data.

The computer processor 11 functions as an animation generation unit 331a, a moving image generation unit 331b, and a moving image reproduction unit 331c by executing a computer-readable instruction. At least a part of the functions realized by the computer processor 11 may be realized by a computer processor other than the computer processor 11 of the moving image distribution system 301.

The animation generation unit 331a may realize substantially the same function as the animation generation unit 21c of the moving image distribution system 1. For example, the animation generation unit 331c can generate an animation of the avatar of the distribution user of the posture data acquisition device 310. In one embodiment, the animation generation unit 331c generates an animation of the distribution user's avatar based on the distribution user's posture feature amount and the avatar data 23b received from the video distribution device 320. The animation generation unit 331c continuously receives the posture feature amount of the viewing user from the moving image distribution device 320, and thus, based on the posture feature amount, an animation of an avatar that moves in synchronization with the movement of the body of the viewing user. Can be generated.

The moving image generation unit 331b can construct a virtual space by using the object data 23a acquired from the moving image distribution device 320, and can generate a moving image in which the avatar generated by the animation generation unit 331a moves in this virtual space. The voice of the distribution user can be synthesized with this moving image.

The moving image reproducing unit 331a reproduces the moving image generated by the moving image generating unit 331b. Thereby, the moving image including the animation of the avatar of the distribution user is displayed on the display 14 of the moving image reproducing apparatus 330.

A process implemented by the animation generation unit 331a when a transmission delay occurs in the transmission path that transmits data from the moving image distribution apparatus 320 to the moving image reproduction apparatus 330 will be described. If a transmission delay occurs in the transmission path from the video distribution device 320 to the video reproduction device 330, it may not be possible to generate an avatar animation that smoothly moves reflecting the motion of the distribution user. The animation generation unit 331a can generate an avatar animation based on the reference posture identification data (for example, the reference posture ID) from the video distribution device 320. Specifically, when the animation generation unit 331a receives the reference posture identification data from the video playback device 10, the animation generation unit 331a refers to the animation management data 313a to identify the registered animation associated with the received reference posture identification data. It is possible to generate an avatar animation based on the animation definition data associated with the registered animation ID of the specified registered animation. Therefore, even if the posture feature quantity cannot be received while the motion data transmission is delayed, the animation generation unit 331a sends it from the video distribution device 320 while the delay is generated. An avatar animation can be generated based on the reference posture identification data.

In one embodiment, the posture data acquisition device 310 may be configured to implement the function of the video playback device 320, and the video playback device 320 may be configured to implement the function of the posture data acquisition device 310. This allows users of both devices to communicate with each other bidirectionally via the avatar.

Next, with reference to FIG. 16, a modification of the above-described video distribution system will be described. In the above embodiment, when the classification unit 11d determines that the posture of the viewing user belongs to the reference posture, the posture of the user is determined by transmitting the reference posture ID for identifying the reference posture instead of the posture feature amount. The amount of data represented is reduced. In the example shown in FIG. 16, the index (subscript) of the feature array determined as follows is used instead of the reference posture ID. That is, when the transmission delay occurs, the index of the feature array is transmitted instead of the posture feature amount indicating the posture of the user. The index of the feature array is data of about several bits as described below, and can be represented by the amount of information in the range of 10 bits or less like the reference posture ID.

In the moving image distribution system 1, the characteristic array is determined as follows. First, as described with reference to FIGS. 7 and 8, the moving image reproducing apparatus 10 images the user of the moving image reproducing apparatus 10 at a predetermined frame rate to acquire a captured image, and in each frame of the captured image. A plurality of feature points related to the user are extracted. For example, as shown in FIG. 7, six characteristic points P1 to P6 are extracted. The moving image reproducing apparatus 10 obtains an image vector related to the user's motion for each of the extracted plurality of feature points. The image vector at each of the feature points P1 to P6 is, for example, the depth at each feature point. The image vector of each of the feature points P1 to P6 may be normalized as a range of 0 to 1.0 and expressed as a float array. The image vectors of the characteristic points P1 to P6 thus obtained are shown in FIG. Although FIG. 16 shows image vectors for 10 frames for simplification of description, the image vectors are actually acquired for a sufficient number of frames for statistical analysis.

Next, for each of the feature points P1 to P6, the sum Σ of the image vector elements in the predetermined section is calculated. In the table of FIG. 16, the sum of the elements of the image vector in the section of the 0th frame to the 25th frame for 26 frames is shown in the row labeled “Σ”. Next, the sum of the image vector elements of the respective feature points P1 to P6 is sorted in descending order. In FIG. 16, the sorting result is shown in the line labeled "RANK". In the example of FIG. 16, P4, P3, P6, P2, P1, and P5 are sorted in this order. The array after this sorting is expressed as {4,3,6,2,1,5}.

The moving image reproducing apparatus 10 calculates a representative vector indicating a characteristic posture or movement of the user by learning a sample of an image vector related to the movement of the user. The representative vector is calculated using, for example, the Lloyd algorithm and other known algorithms. That is, the representative vector indicates the characteristic posture of the user. Therefore, the posture of the user can be specified based on this representative vector. The moving image reproducing device stores the calculated representative vector together with the index. The video playback device 10 may acquire the representative vector calculated by a device other than the own device. In this embodiment, four representative vectors A to D are calculated, and the elements of each representative vector are sorted in descending order to obtain an array of representative vectors. Arrays obtained by sorting the elements of the representative vectors A to D in descending order are {1,3,6,2,4,5}, {4,3,6,2,1,5}{2,3, 6,4,5,1} {1,2,3,4,5,6} is assumed to be represented.

The moving image reproducing apparatus 10 compares the input array {4,3,6,2,1,5} obtained by sorting the image vectors obtained from the captured images with the array of each representative vector, and the input array { 4,3,6,2,1,5} is selected as the closest representative vector array. In the example shown in FIG. 16, the input array {4,3,6,2,1,5} is an array of the representative vector B among the arrays of four representative vectors {4,3,6,2,1, 5}, {4,3,6,2,1,5} is selected as the feature array. The moving image reproducing apparatus 10 transmits the index of the feature array thus selected to the moving image distribution apparatus 20.

The moving image distribution apparatus 20 stores a codebook in which the representative vectors A to D and their arrays are associated with each other. The moving image distribution apparatus 20 refers to the codebook and generates an animation of the avatar of the user so as to take a posture corresponding to the representative vector B corresponding to the feature array acquired from the moving image reproduction apparatus 10.

In the above example, the feature array is transmitted from the video playback device 10 to the video delivery device 20 for each unit section of 26 frames. This unit section can be adjusted appropriately.

As described above, in the moving image distribution system 1, the index of the characteristic array is transmitted instead of the reference posture ID, and the animation of the avatar is generated based on the index of the characteristic array when the transmission is delayed. Similarly, in the moving image distribution system 301, the feature array can be used instead of the reference posture ID. The feature array can be used in place of the reference facial expression ID in the moving image distribution system 101.

The effects of the above embodiment will be described. In the above aspect, while the delay occurs in the transmission path, the reference posture identification data that can specify the information about the posture of the viewing user with a small amount of data instead of a large amount of posture feature amount of a moving image is a moving image. It is transmitted from the reproduction device 10 to the moving image distribution device 20. Accordingly, when a delay occurs in the transmission path, the information regarding the posture of the viewing user can be transmitted to the video distribution device 20 by the reference posture identification data having a small data amount.

In one aspect, a registration animation can be registered for the avatar of the viewing user, and while the delay occurs in the transmission path, the registration animation corresponding to the reference posture specified based on the reference posture identification data can be registered. Video including is delivered. As a result, even if it is difficult to transmit the posture feature amount representing the posture of the viewing user to the moving image distribution device 20 due to the delay in the transmission path from the moving image reproducing device 10 to the moving image distribution device 20, Enrollment animations related to posture can be included in the video.

In the above aspect, the moving image reproducing apparatus 330 generates an animation of an avatar that moves based on the movement of the distribution user based on the posture feature amount of the distribution user received from the moving image distribution apparatus 320. While the transmission path is being delayed, the reference posture identification data, not the posture feature amount, is transmitted from the moving image distribution device 320 to the moving image reproduction device 310. Accordingly, when delay occurs in the transmission path, the avatar animation can be generated in the moving image reproduction apparatus 330 by the reference posture identification data that can represent the movement of the distribution user with a small amount of data.

In one aspect, a user animation generated based on the movement of the user can be registered as a registration animation. As a result, it is possible to increase variations in the movement of the avatar when the posture feature amount cannot be used.

In one aspect, while the delay is occurring in the transmission path, the reference facial expression identification data that can specify the information about the posture of the viewing user with a small amount of data instead of the facial feature amount with a large amount of data is the moving image reproducing apparatus. 10 is transmitted to the video distribution device 20. Thereby, when delay occurs in the transmission path, the information regarding the facial expression of the viewing user can be transmitted to the video distribution device 20 by the reference facial expression identification data having a small data amount.

In the processing procedure described in this specification, particularly in the processing procedure described with reference to the flowchart, some of the steps (steps) constituting the processing procedure are omitted, It is possible to add steps that are not explicitly shown and/or change the order of the steps, and the processing procedure in which such omissions, additions, and changes in order are made without departing from the spirit of the present invention. It is included in the scope of the invention.

1, 101, 201, 301 Video distribution system 10, 110, 210, 330 Video playback device 11a Video playback unit 11b Posture data acquisition unit 11c Transmission unit 11d Classification unit 11e Delay monitoring unit 11f Face feature amount acquisition unit 20, 120, 220 , 320 Video distribution device 21a Video generation unit 21b Video distribution unit 21c Animation generation unit 310 Posture data acquisition device

Claims (15)

Comprises one or more computer processors,
The one or more computer processors execute computer-readable instructions to:
Play the video received from the video distribution device,
Sending motion data including a posture feature amount indicating the feature of the posture of the user to the moving image distribution device,
Receiving a video including an animation of the avatar of the user generated based on the posture feature amount from the video distribution device,
It is determined whether or not the posture of the user matches a predetermined reference posture,
Wherein while the motion data delay for transmission to a video distribution apparatus has occurred, and transmits the reference posture identification data identifying the reference posture is determined to match the attitude of the user to the video distribution device,
The reference posture identification data has a smaller data amount than the posture feature amount,
Video playback device. Whether the posture of the user matches with the reference attitude is determined based on the evaluation function the orientation feature amount as a variable,
The moving picture reproducing apparatus according to claim 1. Whether or not the posture of the user matches the reference posture is determined by a classifier that classifies the posture of the user based on the posture feature amount,
An image representing the reference posture is presented to the user, and the classifier is created by learning a posture feature amount representing a posture taken by the user with respect to the presented image as teacher data.
The moving picture reproducing apparatus according to claim 1. One or more registration animations are registered for the user's avatar,
Receiving a moving image including the registered animation corresponding to the reference posture specified based on the reference posture identification data while the delay is occurring,
The moving image reproducing apparatus according to claim 1. Generate a user animation of the avatar based on the posture feature amount acquired in time series,
Register the user animation as the registered animation,
Generate a sample video containing the user animation,
Determining the reference posture corresponding to the user animation based on an image of a reference frame selected from a plurality of frames forming the sample moving image;
The video playback device according to claim 4. A user animation of the avatar is generated based on the posture feature amount acquired in time series,
Register the user animation as the registered animation,
Calculating the posture feature amount for the user at each of a plurality of feature points for the user based on a predetermined frame rate,
Calculating a first RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in the first frame;
Calculating a second RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in a second frame that is later than the first frame in time series;
Calculating a third RMS that is a root mean square of the posture feature amount at each of the plurality of feature points in a third frame that is time-sequentially later than the second frame,
The posture characteristic in the third frame when the positive/negative of the first RMS difference which is the difference between the second RMS and the first RMS and the positive/negative of the second RMS difference which is the difference between the third RMS and the second RMS are reversed. Determining the reference pose corresponding to the user animation based on an amount,
The video playback device according to claim 4. Generate a user animation of the avatar based on the posture feature amount acquired in time series,
Register the user animation as the registered animation,
First evaluation data including the posture feature amount obtained in time series based on the movement of the user with respect to the user animation, and the posture obtained in time series based on other movements of the user with respect to the user animation Determining the reference posture corresponding to the user animation by comparing with second evaluation data including a feature amount,
The video playback device according to claim 4. The posture feature amount includes bone data that represents the position and orientation of the bone of the user with a three-dimensional vector.
The moving image reproducing apparatus according to claim 1. The motion data includes a facial feature amount representing a facial feature of the user,
The one or more computer processors,
It is determined whether the facial expression of the user matches a predetermined reference facial expression by another classifier that classifies the facial expression of the user based on the facial feature amount,
While the delay is occurring, the reference facial expression identification data for identifying the reference facial expression determined to match the user's facial expression is transmitted to the video distribution device,
The reference facial expression identification data has a smaller data amount than the facial feature amount,
The moving image reproducing apparatus according to claim 1. The reference posture identification data is transmitted instead of the motion data while the delay is occurring,
The moving image reproducing apparatus according to claim 1. The motion data is transmitted in real time,
The video playback device according to claim 1. A moving image distribution system comprising one or a plurality of computer processors, which distributes a moving image including a user's avatar to a moving image reproducing device of the user,
The one or more computer processors execute computer-readable instructions to:
Receiving motion data including a posture feature amount indicating the posture of the user from the moving image reproducing device via a transmission path,
An animation of the avatar of the user generated based on the posture feature amount is included in the moving image and distributed,
While the transmission of the motion data from the moving image reproducing apparatus matching the user's posture is delayed, reference posture identifying data for identifying a reference posture matching the user's posture is provided from the moving image reproducing apparatus. Received,
The user's avatar registration animation generated based on the reference posture identification data is included in the moving image and distributed,
The reference posture identification data has a smaller data amount than the posture feature amount,
Video distribution system. A method of playing back a moving image that is executed by one or more computer processors executing computer readable instructions,
A step of playing a video received from the video distribution device,
A step of transmitting motion data including a posture feature amount indicating a posture of the user to the moving image distribution device,
Receiving a video including an animation of the avatar of the user generated based on the posture feature amount from the video distribution device,
Determining whether or not the posture of the user matches a predetermined reference posture by a classifier that classifies the posture of the user based on the posture characteristic amount;
Determining whether or not there is a delay in the transmission of the motion data to the video distribution device;
A step of transmitting reference posture identification data for identifying the reference posture determined to match the posture of the user to the video distribution device while the delay is occurring;
Equipped with
The reference posture identification data has a smaller data amount than the posture feature amount,
Video playback method. Comprises one or more computer processors,
The one or more computer processors execute computer-readable instructions to:
The motion data including the posture feature amount representing the feature of the user's posture is transmitted to the video playback device,
It is determined whether or not the posture of the user matches a predetermined reference posture,
While a delay occurs in the transmission of the motion data to the video playback device, the reference posture identification data for identifying the reference posture determined to match the user's posture is transmitted to the video playback device,
The reference posture identification data has a smaller data amount than the posture feature amount,
Video distribution device. Comprising one or more computer processors and storage for storing registration animations,
The one or more computer processors execute computer-readable instructions to:
Receives motion data including a posture feature amount representing the feature of the posture of the user from the video distribution device,
Generates a video containing an animation of the user's avatar generated based on the motion data,
While delaying the transmission of the motion data from the video distribution device, the reference attitude corresponding to the reference attitude based on reference attitude identification data for identifying a reference attitude determined to match the attitude of the user is used. Generate a video containing a registration animation,
The reference posture identification data has a smaller data amount than the posture feature amount,
Video playback device.

Images