JP5372823B2 - Video content generation system, metadata construction device, video content generation device, portable terminal, video content distribution device, and computer program - Google Patents

Description

  The present invention relates to a video content generation system, a metadata construction device, a video content generation device, a mobile terminal, a video content distribution device, and a computer program.

  In recent years, as a technique for displaying computer graphics (CG) objects in accordance with music, for example, when a performer plays music, a technique in which a CG model moves according to a predetermined mapping pattern with music has been proposed. Has been.

  In Patent Literature 1, drawing information (viewpoint information, light source information) is reset with respect to the time series of CG objects based on the static attribute or dynamic attribute of music data. The music data is reproduced in synchronization with the CG object display. Further, in the motion creation device described in Patent Document 2, a motion database is constructed as a directed graph in which two frames having similar human body postures in a plurality of motions are connected, and music data is selected from the plurality of motions. The motion having the motion feature component correlated with the beat feature component acquired from the above is selected. In the technique described in Non-Patent Document 1, when a unique data structure is generated in advance and music is input, synchronization can be performed at a higher speed by Dynamic Programming (dynamic programming).

As a music analysis technique, a technique described in Non-Patent Document 2 is known. According to the technique of Non-Patent Document 2, beat intervals and beat structures are acquired by estimating sound generation components, chord changes, sound generation time of percussion instruments, and the like. As a motion analysis method, a technique described in Non-Patent Document 3 is known. According to the technique of Non-Patent Document 3, the beat interval and the beat structure are acquired by estimating the motion beat change and the generation time.
Non-Patent Document 4 discloses a technique for generating new motion data using motion graphs.
Non-Patent Document 5 discloses a technique for searching for an optimum path from a certain starting point by dynamic programming as a path search technique.

JP 2005-56101 A JP 2007-18388 A

Xu Jian, Koichi Takagi, Yasuo Yoneyama, “Dance Generation Method Synchronized with Music Based on Motion Beats and Exciting Information”, 2009 Video Media Processing Symposium (IMPS), I-04-01, October 7, 2009 M. Goto, “An Audio-based Real-time Beat Tracking System for Music With or Without Drum-sounds”, Journal of New Music Research, Vol. 30, no. 2, pp. 159-171, 2001 T. Kim, S.I.Park, S.Y.Shin, “Rhythmic-Motion Synthesis Based on Motion-Beat Analysis”, ACM Transaction on Graphics, Vol. 22, Issue 3, 2003 (SIGGRAPH 2003), pp. 392-401 L. Kovar, M. Gleicher, and F. Pighin, “Motion Graphs”, ACM Transaction on Graphics, Vol. 21, Issue 3, 2002 (SIGGRAPH 2002), pp. 473-482 Cormen, Thomas H .; Leiserson, Charles E., Rivest, Ronald L. (1990). Introduction to Algorithms (2st ed.). MIT Press and McGraw-Hill. ISBN 0-262-03141-8. Pp. 323- 69

  However, in the prior art described in Patent Document 1 described above, when creating a CG animation for a certain piece of music, if the time series of the CG object does not match the music of the music, a CG animation that matches the music is displayed. There is a problem that it is difficult to create. Further, in the prior art described in Patent Document 2, for a vast number of generally available motion data, constructing a motion database as a directed graph connecting two frames having similar human body postures, There is a problem that it is difficult in practice. For this reason, it is desirable to link motion data with respect to motion data selected based on the actual motion creation target music.

  In the prior art described in Non-Patent Document 4, the amount of calculation required for constructing a motion graph and searching for an optimal path is enormous. In addition, when a motion graph constructed without considering the original motion structure is used, the original motion structure may be broken. For example, transitioning between intense and slow movements can result in unnatural or inconsistent movements due to sudden changes in movement.

  In the prior art described in Non-Patent Document 1, there is a problem that the music data and the CG object cannot be synchronized unless music data from the start to the end of the song is input. For this reason, for example, it is difficult to generate a CG animation that matches a streamed song in real time.

  The present invention has been made in consideration of such circumstances, and its purpose is to use a large-scale motion database even if music data from the start to the end of the music to be synchronized with the video content is not input. Thus, an object of the present invention is to provide a video content generation system, a metadata construction device, a video content generation device, a mobile terminal, a video content distribution device, and a computer program that can generate video content that matches music.

The present invention provides a motion graph construction unit that constructs a motion graph including nodes, edges, and edge weights based on feature amounts extracted from motion capture data applicable to video content, and music in each node of the motion graph. A metadata construction unit that calculates metadata indicating the possibility of synchronization with the music content, and music data of the music for which video content generation is to be performed are input in a predetermined amount in order of reproduction, and a music feature amount is acquired from the input music data of the predetermined amount And a node having a certain synchronization possibility in the node group having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data on the motion graph. An optimal path search unit that searches for paths that can be connected to each other, and the path searched by the optimal path search unit A synchronization unit that generates synchronization information that associates the motion data with the predetermined amount of music data, and a video data generation unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information And the metadata constructing unit includes a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to the video content, and an edge indicating connectivity of the node. For each node included in the motion graph to be calculated, the weight calculation means for calculating the weight according to the number of nodes that can be connected after the node, the optimum path search unit is calculated by the weight calculation means a video content generation system comprising that you search the path based on the weights.

  Also, in the video content generation system of the present invention, the weight calculation means has an infinite number of nodes that can be connected after the node, and nodes that can be connected after the node. And a weight of another node is estimated from the two types of nodes.

  In the video content generation system of the present invention, the optimum path search unit is a path that connects nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph. Thus, a path having a weight of the last node equal to or greater than a certain amount is searched for.

  In the video content generation system of the present invention, the optimum path search unit is a path that connects nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph. Thus, only a part of the head is determined.

  In the video content generation system of the present invention, the music analysis unit determines a predetermined amount of music data according to a calculation time of the optimum path search unit and a music data rate.

  The present invention further includes a storage unit that stores the music data of the input video content generation target music, and the music analysis unit determines a predetermined amount of the music data to be used in the next process as a maximum storage capacity of the storage unit. Alternatively, the video amount is the smaller of the storage amount specified by the user and is a data amount that can be processed within the reproduction time of the video data generated by the video data generation unit in the current processing It is a content generation system.

  Further, the present invention is included in a motion graph composed of a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content and an edge indicating connectivity of the node. It is a metadata construction apparatus characterized by having a weight calculation means for calculating a weight corresponding to the number of nodes that can be connected after the node for each node.

  Further, in the metadata construction device of the present invention, the weight calculation means includes a node in which the number of nodes that can be connected after the node is zero, and an infinite number of nodes that can be connected after the node. And a weight of another node is estimated from the two types of nodes.

  Further, the present invention provides a motion graph composed of a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content, and an edge indicating connectivity of the node, For each node included in the motion graph, an acquisition unit that acquires weights according to the number of nodes that can be connected after the node, and music data of the video content generation target music are input in predetermined amounts in the order of playback and input. A music analysis unit that acquires a music feature value from the predetermined amount of music data, and a node having a motion feature value corresponding to the music feature value of the predetermined amount of music data based on the weight on the motion graph An optimum path search unit for searching for a path connecting groups, and a path corresponding to the path searched by the optimum path search unit. A synchronization unit that generates synchronization information that associates motion data with the predetermined amount of music data, and a video data generation unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information. A video content generation device characterized by comprising the video content generation device.

  In the video content generation device of the present invention, the optimum path search unit is a path that connects nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data on the motion graph. Thus, a path having a weight of the last node equal to or greater than a certain amount is searched for.

  In the video content generation device of the present invention, the optimum path search unit is a path that connects nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data on the motion graph. Thus, only a part of the head is determined.

  In the video content generation apparatus of the present invention, the music analysis unit determines a predetermined amount of music data according to a calculation time of the optimum path search unit and a music data rate.

  The present invention further includes a storage unit that stores the music data of the input video content generation target music, and the music analysis unit determines a predetermined amount of the music data to be used in the next process as a maximum storage capacity of the storage unit. Alternatively, the data amount is the smaller of the storage amount designations by the user and can be processed within the reproduction time of the video data generated by the video data generation unit in the current process.

The present invention also provides a motion graph constructing unit that constructs a motion graph including nodes, edges and edge weights based on feature amounts extracted from motion capture data applicable to video content, and each node of the motion graph. A metadata construction unit for calculating metadata indicating the possibility of synchronization with the music, and music data of the video content generation target music is input in a predetermined amount in the reproduction order, and a music feature amount from the input music data of the predetermined amount And a music analysis unit that acquires a predetermined synchronization possibility that a final node of the node group among the node groups having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data on the motion graph. An optimal path search unit that searches as a path that can connect nodes having the group, and the optimal path search unit searched for A synchronization unit that generates synchronization information that associates the motion data corresponding to the predetermined amount of music data with the predetermined amount of music data, and video data that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information A generation unit , wherein the metadata construction unit includes a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to the video content, and an edge indicating connectivity of the node For each node included in the motion graph composed of: a weight calculating unit that calculates a weight according to the number of nodes that can be connected after the node; and the optimum path searching unit includes: on the basis of the calculated weights is a mobile terminal characterized that you search the path.

The present invention also provides a motion graph constructing unit that constructs a motion graph including nodes, edges and edge weights based on feature amounts extracted from motion capture data applicable to video content, and each node of the motion graph. A metadata construction unit for calculating metadata indicating the possibility of synchronization with the music, and music data of the video content generation target music is input in a predetermined amount in the reproduction order, and a music feature amount from the input music data of the predetermined amount And a music analysis unit that acquires a predetermined synchronization possibility that a final node of the node group among the node groups having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data on the motion graph. An optimal path search unit that searches as a path that can connect nodes having the group, and the optimal path search unit searched for A synchronization unit that generates synchronization information that associates the motion data corresponding to the predetermined amount of music data with the predetermined amount of music data, and video data that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information A generation unit; and a distribution unit that distributes the predetermined amount of music data and video data reproduced together with the predetermined amount of music data , wherein the metadata constructing unit is a motion applicable to the video content. For each node included in a motion graph composed of a frame node having the same motion feature amount among the feature amounts extracted from the capture data and an edge indicating the connectivity of the node, the nodes can be connected after the node. A weight calculation unit configured to calculate a weight according to the number of nodes, wherein the optimum path search unit includes the weight calculation unit; On the basis of the calculated weights is a video content delivery device comprising a search child said path.

  Further, the present invention is included in a motion graph composed of a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content and an edge indicating connectivity of the node. This is a computer program for causing a computer to execute a weight calculation step for calculating a weight corresponding to the number of nodes that can be connected after each node for each node.

  Further, the present invention provides a motion graph composed of a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content, and an edge indicating connectivity of the node, For each node included in the motion graph, an acquisition step for acquiring a weight according to the number of nodes that can be connected after the node, and music data of the video content generation target music are input in predetermined amounts in the order of playback and input. A music analysis step for obtaining a music feature amount from the predetermined amount of music data, and a node having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data based on the weight on the motion graph An optimal path searching step for searching for paths connecting groups, and the optimal path searching step. Generating synchronization information for associating the motion data corresponding to the path and the predetermined amount of music data, and generating video data to be reproduced together with the predetermined amount of music data based on the synchronization information A computer program for causing a computer to execute a video data generation step.

  According to the present invention, even if no music data from the start to the end of the music that synchronizes the video content is input, it is possible to generate video content that matches the music using a large-scale motion database. An effect is obtained.

It is a block diagram which shows the structure of the video content production | generation system 1 which concerns on one Embodiment of this invention. It is a definition example of human body skeleton type motion data. It is a block diagram which shows the structure of the motion analysis part 11 shown in FIG. It is a block diagram which shows the structure of the beat extraction part 31 shown in FIG. It is a conceptual diagram of the data division process which concerns on the beat extraction part 31 shown in FIG. It is a conceptual diagram for demonstrating the principal component coordinate connection process which concerns on the beat extraction part 31 shown in FIG. It is a conceptual diagram of the sine approximation process which concerns on the beat extraction part 31 shown in FIG. It is a conceptual diagram which shows the whole flow of the structure method of the motion graph which concerns on one Embodiment of this invention. It is a conceptual diagram which shows the flow of the motion graph production | generation method which concerns on one Embodiment of this invention. It is the figure which showed the predetermined weight for every joint which concerns on one Embodiment of this invention. It is a conceptual diagram of the blending process which concerns on the bidirectional | two-way edge of one Embodiment of this invention. It is a conceptual diagram explaining the blending process which concerns on one Embodiment of this invention. It is the flowchart which showed the process sequence which specifies the loop which concerns on one Embodiment of this invention. It is the conceptual diagram which showed the flow of the music data delivered by the streaming system based on one Embodiment of this invention, and the produced | generated video data. It is the schematic which showed the relationship between the processing time of the synchronous process which concerns on one Embodiment of this invention, and the reproduction | regeneration time of the video content produced | generated by the synchronous process. It is the schematic which showed the relationship between the processing time of the synchronous process which concerns on one Embodiment of this invention, and the reproduction | regeneration time of the video content produced | generated by the synchronous process. It is a conceptual diagram of the process which adjusts the frame rate of the motion which concerns on one Embodiment of this invention. It is the schematic which showed the relationship between the music data for the fixed time reproduction | regeneration used as the object of the synchronous process which concerns on one Embodiment of this invention, and the path | pass selected by the synchronous process.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration of a video content generation system 1 according to an embodiment of the present invention. In FIG. 1, the video content generation system 1 includes a metadata construction device 10 and a video content generation device 20. The metadata construction device 10 and the video content generation device 20 are connected by a communicable network (not shown) and can transmit and receive data to and from each other. The metadata construction device 10 includes a motion analysis unit 11 and a database 12. The video content generation device 20 includes a music data storage unit 21, a music analysis unit 22, a music analysis data storage unit 23, an optimum path search unit 24, a synchronization unit 25, a synchronization information storage unit 26, and video data generation. Unit 27, video content storage unit 28, and playback unit 29.

  The music data of the music for which video content is generated is streamed from the music file 3 to the video content generation device 20.

  The motion database 2 stores a large amount of generally available motion data (motion capture data applicable to video content). The metadata construction device 10 and the video content generation device 20 input motion data from the motion database 2. In this embodiment, human motion data is handled, and human skeleton type motion data defined as illustrated in FIG. 2 is used as human motion data.

  FIG. 2 is a schematic diagram of a definition example of human motion data, which is a definition example of human skeleton type motion data (human body skeleton type motion data). Human body skeleton-type motion data is based on the human skeleton, using bone and bone connection points (joints), with one joint as the root (root), and the structure of bones sequentially connected from the root via the joint (tree) Tree) structure. FIG. 2 shows only a part of the definition of the human body skeleton type motion data. In FIG. 2, a joint 100 is a waist part and is defined as a root. Joint 101 is the elbow portion of the left arm, Joint 102 is the wrist portion of the left arm, Joint 103 is the elbow portion of the right arm, Joint 104 is the wrist portion of the right arm, Joint 105 is the knee portion of the left foot, and Joint 106 is the left foot portion. The ankle part, joint 107 is the right leg knee part, and joint 108 is the right leg ankle part.

  The skeleton type motion data is data that records the movement of each joint of the skeleton type object, and a human body, an animal, a robot, or the like is applicable as the skeleton type object. As the skeleton type motion data, position information, angle information, speed information, acceleration information, and the like of each joint can be used. Here, human body skeleton angle data and acceleration information will be described as an example of human body skeleton type motion data.

Human skeleton-type angle information data represents a series of human movements by a series of multiple poses, and includes basic pose data that represents a person's basic pose and actual human movements. Frame data for each pose representing each pose. The basic pose data includes information such as the position of the root and the position of each joint in the basic pose, and the length of each bone. The basic pose is specified by the basic pose data. The frame data represents the amount of movement from the basic pose for each joint. Here, angle information is used as the movement amount. Each frame data identifies each pose in which each movement amount is added to the basic pose. Thereby, a series of movements of a person is specified by the continuation of each pose specified by each frame data. The human skeleton-type angle information data can be created by a motion capture process from an image obtained by photographing a person's movement with a camera, or can be created manually by key frame animation.
The human body skeleton type acceleration information data represents the acceleration of each joint of a person by continuous frame data for each pose and a plurality of poses. The human skeleton-type acceleration information data can be recorded by an accelerometer, or calculated from video and motion data.

  In the following description according to the present embodiment, the human skeleton type motion data is simply referred to as “motion data”.

  Hereinafter, each part of the metadata construction apparatus 10 included in the video content generation system 1 shown in FIG. 1 will be sequentially described.

[Motion analysis unit]
The motion analysis unit 11 acquires motion data from the motion database 2, analyzes the acquired motion data to acquire motion feature amounts, and stores the acquired motion feature amounts in the database 12. The motion analysis unit 11 targets all motion data stored in the motion database 2. The processing of the motion analysis unit 11 is performed as a preliminary preparation stage before the stage of actually generating the video content.

  FIG. 3 is a block diagram illustrating a configuration of the motion analysis unit 11 illustrated in FIG. 1. In FIG. 3, the motion analysis unit 11 includes a beat extraction unit 31, a beat information storage unit 32, a climax calculation unit 33, a climax information storage unit 34, and a motion graph generation unit 35. In the present embodiment, the motion graph generation unit 35 operates as a motion graph construction unit and a metadata construction unit.

[Beat extractor]
The beat extraction unit 31 detects the beat time from the input motion data. Here, the beat time of motion data is defined as the time when the direction or intensity of repetitive motion changes. For example, in a movement such as a dance, the timing of hitting is equivalent. The beat extraction unit 31 divides the input motion data into short-time motion data, and detects the beat time by principal component analysis from the divided motion sections.

  Hereinafter, beat time detection processing by the beat extraction unit 31 of the present embodiment will be described. FIG. 4 is a block diagram showing a configuration of the beat extraction unit 31 shown in FIG. In FIG. 4, the beat extraction unit 31 includes a physical quantity conversion unit 311, a data division unit 312, a short-time principal component analysis unit 313, a principal component coordinate connection unit 314, a beat estimation unit 315, and a post-processing unit 316.

[Physical quantity converter]
The physical quantity conversion unit 311 calculates the joint relative position at time t in the input motion data. The joint relative position is a relative position of the joint with respect to the root.

Here, a method for calculating the joint relative position will be described.
First, joint positions are calculated using basic pose data and frame data in the human skeleton-type angle information data. The basic pose data includes information for specifying the basic pose, such as the position of the root and the position of each joint in the basic pose, and the length of each bone. The frame data has information on the amount of movement from the basic pose for each joint. Here, angle information is used as the movement amount. In this case, the position p ^k (t) of the k-th joint at time t is calculated by the equations (1) and (2). p ^k (t) is represented by three-dimensional coordinates. Note that time t is the time of the frame data. In the present embodiment, time t is simply handled as a “frame index”. Thereby, the time t takes values of 0, 1, 2,..., T−1. T is the number of frames included in the motion data.

However, the 0th (i = 0) joint is the root. R _axis ^{i-1, i} (t) is a coordinate rotation matrix between the i-th joint and its parent joint ("i-1" -th joint), and is included in the basic pose data. A local coordinate system is defined for each joint, and the coordinate rotation matrix represents the correspondence of the local coordinate system between joints in a parent-child relationship. R ⁱ (t) is a rotation matrix of the i-th joint in the local coordinate system of the i-th joint, and is angle information included in the frame data. T ⁱ (t) is a transition matrix between the i-th joint and its parent joint, and is included in the basic pose data. The transition matrix represents the bone length between the i-th joint and its parent joint.

Next, the relative position (joint relative position) p ′ ^k (t) of the k-th joint with respect to the root at time t is calculated using equation (3).

Here, p ^root (t) is the position (p ⁰ (t)) of the route (0th joint) at time t.

Thus, the frame “x (t)” at time t is expressed as “x (t) = {p ′ ¹ (t), p ′ ² (t),..., P ′ ^K (t)}”. Is done. K is the number of joints excluding the root.

[Data division part]
The data dividing unit 312 divides the joint relative position data into sections of a certain time. The data division unit 312 performs a data division process on the joint relative position data “p ′ ^k (t)” of each joint. FIG. 5 shows the concept of data division processing. In the data dividing process, the joint relative position data is divided into sections of a fixed time (corresponding to a fixed number of frames). The length of the divided section can be set as appropriate. The length of the divided section is, for example, 60 times the time per frame.

[Short-term principal component analysis section]
The short-time principal component analysis unit 313 performs principal component analysis processing in each section on the joint relative position data divided by the data division unit 312. Here, using the frame “x (t)” at time t, the data “X” in one section is expressed as “X = {x (t1), x (t2),..., X (tN)}”. Where N is the section length (the number of frames included in the section), and X is a matrix of M rows and N columns (where M = 3 × K).

  In the principal component analysis processing, principal component analysis processing is performed on X, and X is converted into a principal component space.

Here, the principal component analysis method will be described.
First, the matrix D of N rows and M columns obtained by subtracting the average value from X is calculated by the equation (4).

  Next, singular value decomposition (Singular Value Decomposition) is performed on the matrix D of N rows and M columns according to the equation (5).

  However, U is a unitary matrix of N rows and N columns. Σ is a diagonal matrix having non-negative diagonal elements of N rows and M columns in descending order, and represents the variance of the coordinates of the principal component space. V is a unitary matrix of M rows and M columns, and is a coefficient (principal component) for the principal component.

  Next, the matrix D of N rows and M columns is converted into the principal component space by the equation (6). The matrix Y with M rows and N columns represents the coordinates of the principal component space.

  The short-time principal component analysis unit 313 stores a matrix (principal component coordinate matrix) Y representing the coordinates of the principal component space and a coefficient matrix (principal component coefficient matrix) V for the principal components in a memory for each section.

  Note that the matrix X representing the coordinates of the original space and the principal component coordinate matrix Y can be converted into each other by the equations (6) and (7).

  Moreover, it can convert by (8) Formula by the upper r main components.

Here, V ^r is a matrix of M rows and r columns composed of upper r rows in the principal component coefficient matrix V. Y ^r is an r-row N-column matrix composed of the upper r columns in the principal component coordinate matrix Y. X ^~ is a matrix of reconstructed M rows and N columns.

  Note that it is also possible to perform principal component analysis processing on only some degrees of freedom of the original space. For example, when the beat can be expressed only by the movement of the foot, the equations (4) and (5) are applied to the M ′ × N matrix X ′ generated only from the joint relative position data regarding the foot. And the principal component analysis processing is performed by the equation (6).

[Principal component coordinate connection part]
The principal component coordinate connection unit 314 connects the coordinates of the principal components of each section along a time series. In the principal component coordinate coupling process performed by the principal component coordinate coupling unit 314, the coordinates of the principal component are adjusted so that the coordinates of the principal component are smoothly coupled at the boundary between two consecutive sections.

  In FIG. 6, the conceptual diagram for demonstrating the principal component coordinate connection process which concerns on this embodiment is shown. In the present embodiment, principal component coordinate connection processing is performed in order from the top section in time series. In FIG. 6, the principal component coordinate connection process up to a certain section (previous section) has been completed. And the principal component coordinate connection process for connecting the next area (this area) is performed with respect to the previous area. In this principal component coordinate connection process, the principal component coordinates of the current section are adjusted so that the principal component coordinates of the current section are smoothly connected to the principal component coordinates of the previous section. In this principal component coordinate adjustment processing, sign inversion or coordinate shift is performed on the principal component coordinates (original coordinates) in the current section selected in the principal component selection step.

  Here, the principal component coordinate connection process will be described.

Principal component coordinate connection Step S11: relative Y ₁ (original coordinates of the first principal component) This section of the main components of the coordinates, the principal component coefficient matrix V of this section, to obtain the coefficients V ₁ for the first principal component. Further, the coefficient V ₁ ^pre for the first principal component is obtained from the principal component coefficient matrix V of the previous section stored in the memory.

Principal component coordinate connection step S12: Based on the relationship between the coefficient V ₁ ^pre respect to the first principal component of the front section and the coefficient V ₁ for a first principal component of the skilled interval, the first principal component of the original according to those sections It is determined whether or not the coordinates of the coordinates are reversed. This sign inversion determination is made by equation (9). (9) of the determination by the formula result, in the case of sign inversion, performs sign inversion on the original coordinate Y ₁ of the first principal component of this section, reference numerals with respect to the main component coefficient matrix V of this section Invert. On the other hand, (9) of the determination by the formula result, code if not reversed, both principal component coefficient matrix V of the first principal component source coordinates Y ₁ and those sections of those sections, the main component coordinate coupling step the raw values The processing result of S12 is used.

However, Y ₁ is selected principal component in this section coordinate (original coordinate of the first principal component). V is a principal component coefficient matrix of this section. V ₁ is a coefficient for the first principal component in the current section. V ₁ ^pre is a coefficient for the first principal component related to the previous section. (V ₁ · V _k ^pre ) is an inner product of V ₁ and V ₁ ^pre . Y ₁ ′ is a processing result of the principal component coordinate connection step S12 for the principal component coordinates (original coordinates of the first principal component) Y ₁ selected in the current section. V ′ is the processing result of the principal component coordinate connection step S12 for the principal component coefficient matrix V in the current section.

Principal component coordinate connection step S13: Coordinate shift is performed on the principal component coordinate Y ₁ ′ of the processing result of the principal component coordinate connection step S12. Specifically, the coordinate shift is performed by the equation (10). In this case, the coordinate Y ₁ ^pre (tN) of the first principal component in the tN frame of the previous section is acquired from the principal component coordinate matrix Y of the previous section.

However, _{Y 1} '(t1), the main component coordinate _{Y 1} of the processing result of step S12' of the coordinates of the t1 frame. Y ₁ ″ (t2) is the coordinate of the t2 frame among the coordinates Y ₁ ″ of the calculation result of the first calculation formula of Formula (10).
The coordinate Y ₁ ″ (t1) of the t1 frame is replaced with Y ₁ ^opt (t1) with respect to the coordinate Y ₁ ″ of the calculation result of the first calculation expression of the equation (10). This replaced coordinate Y ₁ ″ is the coordinate shift result coordinate.

Principal component coordinate connection step S14: In this section, the coordinate Y ₁ ^opt (t1) of the processing result of the principal component coordinate connection step S13 is reflected on the coordinate Y ₁ ′ of the processing result of the principal component coordinate connection step S12. Thereby, the principal component coordinates of the current section are smoothly connected to the principal component coordinates of the previous section.

  The principal component coordinate connection unit 314 performs the above-described principal component coordinate connection processing from the first section to the last section. Thereby, principal component coordinates “y (t), t = 0, 1, 2,..., T−1” of all sections after connection are obtained. T is the number of frames included in the motion data.

[Beat estimation unit]
The beat estimation unit 315 calculates the extreme value b (j) from the principal component coordinates y (t) of all the sections after the connection calculated by the principal component coordinate connection unit 314. The extreme value b (j) of this calculation result corresponds to the beat. The beat set B is expressed by equation (11).

  Here, J is the number of beats.

The beat set can be calculated by a method other than the method described above.
For example, the beat estimation unit 315 calculates the autocorrelation value from the principal component coordinates of all the sections after the connection calculated by the principal component coordinate connection unit 314, and corresponds the extreme value b (j) of the autocorrelation value to the beat. Can be calculated as
In addition, the beat estimation unit 315 calculates an autocorrelation value of the inner product (according to the equation (9)) calculated from the principal component coefficients of the adjacent section after the connection by the principal component coordinate connection unit 314, and the autocorrelation value The extreme value b (j) can be calculated as corresponding to the beat.

[Post-processing section]
The post-processing unit 316 detects the beat time from the beat set B calculated by the beat estimation unit 315.

Here, the beat time detection process will be described.
First, each extreme value in the beat set B is approximated by a sine curve (sinusoid) by the equation (12).

Here, s _j−1 (t) is an approximate sine value in a section from the (j−1) th extreme value b (j−1) to the jth extreme value b (j). t is the time corresponding to the frame, and is “t = 0, 1, 2,..., T−1”. T is the number of frames included in the motion data.

FIG. 7 shows a conceptual diagram of the sine approximation process according to equation (12). In FIG. 7, a section a1 (section in the case of j = 2) from the first extreme value b (1) to the second extreme value b (2) is approximated by s ₁ (t). Similarly, a section a2 (section when j = 3) from the second extreme value b (2) to the third extreme value b (3) is approximated by s ₂ (t), and the third extreme value The section a3 (section in the case of j = 4) from b (3) to the fourth extreme value b (4) is approximated by s ₃ (t), and the fifth extreme value b (4) to the fifth A section a4 (section when j = 5) up to the extreme value b (5) is approximated by s ₄ (t).

Next, Fourier transform is performed on the sine approximation “s _j−1 (t), j = 2, 3,..., J”. In the Fourier transform process, an FFT (Fast Fourier Transform) calculator using a Hann window with a predetermined number of FFT points L is used. Based on the result of the Fourier transform, a frequency (maximum component frequency) fmax having the maximum component is detected from the frequency range related to the Fourier transform. Then, the beat interval TB is calculated by the calculation formula “TB = Fs ÷ fmax”. However, Fs is the number of frames per second.

Next, the maximum between the sine approximation “s _j−1 (t), j = 2, 3,..., J” and the reference value “s ′ (t)” defined by the equation (13) The correlation initial phase is calculated by equation (14).

  Next, a set EB of beat times eb (j) is calculated by the equation (15). However, EJ is the number of beat times eb (j).

  The above is the description of the beat time detection processing by the beat extraction unit 31 of the present embodiment.

  The beat extraction unit 31 calculates a set EB of beat times eb (j) from the motion data by the above-described beat time detection method. Further, the beat extraction unit 31 calculates the tempo of the movement by the equation (16). The tempo of movement is the number of beats per minute. Here, the number of frames per second is 120.

  Where TB is the beat interval (seconds).

  The beat extraction unit 31 stores the set EB and tempo of the beat time eb (j) in the beat information storage unit 32 for each motion data. At this time, the beat information storage unit 32 also stores information indicating the correspondence between the section (principal component analysis section) in which the beat extraction unit 31 performs the principal component analysis process and the beat time eb (j). Thereby, it can be understood to which principal component analysis section a certain beat time belongs.

[Rise calculation part]
The climax calculation unit 33 calculates the climax of the motion by the equation (17) for each principal component analysis section for each motion data.

  However, Σ is a diagonal matrix having non-negative eigenvalues in descending order in the principal component analysis processing of the section, and represents the variance of the coordinates of the principal component space. tr () is the sum of the diagonal components of the diagonal matrix (matrix trace).

  The climax calculation unit 33 stores the climax of each principal component analysis section in the climax information storage unit 34 for each motion data.

[Motion graph generator]
The motion graph generation unit 35 generates a motion graph using the set EB, tempo, and excitement of the beat time eb (j) of each motion data. The motion graph is disclosed in Non-Patent Document 4. The motion graph is composed of a node (vertex) group, an edge (branch) group representing a connection relationship between the nodes, and an edge weight. There are two types of edges: bidirectional and unidirectional.

FIG. 8 is a conceptual diagram showing the overall flow of the motion graph construction method according to this embodiment.
The motion data in the motion database 2 is classified by genre. The genre of motion data in the motion database 2 is set in advance. The genre division is performed according to the characteristics of movement. Information indicating the genre to which the user belongs is added to each motion data. The motion graph generation unit 35 determines the genre of the motion data based on the genre information added to the motion data. In the example of FIG. 8, the motion data in the motion database 2 is classified into n genre databases (genre 1DB to genre nDB).

  The motion graph generation unit 35 further classifies the motion data belonging to the same genre by the value i calculated by the equation (18). In the example of FIG. 8, motion data belonging to genre 2 is classified into m tempo databases (tempo 1DB to tempo mDB).

However, Q _Tempo is the length of the tempo interval. Tempo ^Motion is a tempo related to motion data to be classified. Tempo ^Motion _min is the minimum tempo in the category to be classified.

  The motion graph generation unit 35 generates a motion graph for each tempo database further classified by the calculated value i of equation (18) for motion data belonging to the same genre.

  FIG. 9 is a conceptual diagram showing a flow of a motion graph generation method according to the present embodiment. Hereinafter, a procedure for generating a motion graph of a certain tempo database (tempo iDB) will be described with reference to FIG.

[Beat frame extraction step]
First, in the beat frame extraction step, all frames (beat frames) corresponding to the beat time are extracted from all motion data belonging to the tempo iDB. The set of extracted beat frames is represented as ^{Fi ALL} _B.

[Connectivity calculation step]
Next, in the connectivity calculation step, distances are calculated for all pairs targeted for all beat frames included in the set F ^iALL _B by using the equation (19) or (20). The distance between a certain beat frame F ⁱ _{B and a} certain beat frame F ^j _B is represented as d (F ⁱ _B , F ^j _B ).

Where q _{i, k} is the quaternion of the kth joint of the beat frame F ⁱ _B. w _k is a weight related to the k-th joint. The weight w _k is preset. For example, an example of the weight w _k is as shown in FIG. FIG. 10 is a diagram showing weights predetermined for each joint. In the illustrated example, the weight of Right and Left Hip is 1.0000. The weight of Right and Left Knee is 0.0901. The weight of Right and Left Shoulder is 0.7884. The weight of Right and Left Elbow is 0.0247.

Here, p _{i, k} is a vector of relative positions with respect to the root of the k-th joint of the beat frame F ⁱ _B. That is, p _{i, k} is a vector of the position of the k-th joint of the beat frame F ⁱ _B calculated without considering the position and direction of the route.

  The distance between beat frames can be calculated as a weighted average of differences in physical quantities such as the position, velocity, acceleration, angle, angular velocity, and angular acceleration of each joint constituting a pose in the target beat frame.

Next, in the connectivity calculation step, the connectivity is calculated according to equation (21). The connectivity between a certain beat frame F ⁱ _{B and a} certain beat frame F ^j _B is represented as c (F ⁱ _B , F ^j _B ).

However, d (F ⁱ _B ) is a distance between the previous frame and the rear frame of the beat frame F ⁱ _B (calculated by a calculation formula similar to the formula (19) or the formula (20)). TH is a preset threshold value.

When the connectivity c (F ⁱ _B , F ^j _B ) is 1, it can be determined that the pose of the beat frame F ⁱ _{B and} the pose of the beat frame F ^j _B are similar. When the connectivity c (F ⁱ _B , F ^j _B ) is 0, it cannot be determined that the pose of the beat frame F ⁱ _{B and} the pose of the beat frame F ^j _B are similar.

[Motion graph construction step]
Next, in the motion graph construction step, first, all the beat frames included in the set F ^iALL _B are set as the nodes of the motion graph. Therefore, the initial value of the number of nodes in the motion graph matches the number of beat frames included in the set F ^iALL _B.

Next, when the connectivity c (F ⁱ _B , F ^j _B ) is 1, a bidirectional edge is provided between the node of the beat frame F ⁱ _{B and} the node of the beat frame F ^j _B. When the connectivity c (F ⁱ _B , F ^j _B ) is 0, no bi-directional edge is provided between the node of the beat frame F ⁱ _{B and} the node of the beat frame F ^j _B.

  Next, a unidirectional edge is provided between adjacent beat frames in the same motion data. A unidirectional edge is temporally directed from the node of the previous beat frame to the node of the subsequent beat frame.

Next, a weight for the bidirectional edge is calculated. The weight for the bidirectional edge between the node of the beat frame F ⁱ _{B and} the node of the beat frame F ^j _B is calculated by the equation (22).

  However, rd is calculated by the equation (21).

Next, a weight for a unidirectional edge is calculated. The weight for the unidirectional edge between the node of the beat frame F ⁱ _{B and} the node of the beat frame F ^j _B is determined by one of the following methods (a) and (b).
(A) When the beat frame F ⁱ _B and the beat frame F ^j _B belong to the same principal component analysis section, the rise of the principal component analysis section is used as a weight.
(B) When the beat frame F ⁱ _B and the beat frame F ^j _B belong to different principal component analysis sections, the rise of the principal component analysis section to which the beat frame F ⁱ _B belongs and the principal component analysis section to which the beat frame F ^j _B belongs A value obtained by averaging the swells is used as the weight.

  Next, blending processing is performed on motion data related to nodes (beat frames) at both ends of the bidirectional edge. The blending process is performed for each bidirectional edge direction. Accordingly, two blending processes are performed on one bidirectional edge as shown in FIGS. 11 (1) and (2). FIG. 11 is a conceptual diagram of blending processing related to a bidirectional edge between a node of beat frame i and a node of beat frame j. FIG. 11 (1) shows the blending process in the direction from the node of beat frame i to the node of beat frame j, and FIG. 11 (2) shows the blending in the direction from the node of beat frame j to the node of beat frame i. Represents a process.

  FIG. 12 is a conceptual diagram illustrating the blending process, and corresponds to FIG. 11 (1). Here, with reference to FIG. 12, a blending process in the direction from the node of beat frame i to the node of beat frame j shown in FIG.

  In the blending process, interpolated data (blending) in which the motion data 1 having the beat frame i and the motion data 2 having the beat frame j are mixed to prevent unnatural connection of motion. Motion data) 1_2 is generated. In the present embodiment, the connected portions are interpolated using spherical linear interpolation by quaternions using frames for a fixed time. Specifically, blending motion data 1_2 of a connection section (section length m, where m is a predetermined value) connecting the motion data 1 and the motion data 2, and data 1_m of the last section length m of the motion data 1 are The motion data 2 is generated using data 2_m having the first section length m. At this time, according to the ratio (u / m) of the distance u from the head of the connection section to the section length m of the connection section, the frame i corresponding to the distance u in the data 1_m corresponds to the distance u in the data 2_m. Mix frame j. Specifically, each frame constituting the blending motion data 1_2 is generated by the equations (23) and (24). Note that equation (23) is an equation for one bone.

Where m is the total number (predetermined value) of frames (blending frames) constituting the blending motion data 1_2, u is the order from the top of the blending frame (1 ≦ u ≦ m), and q is the bone of the u-th blending frame. The quaternion, q ⁱ (k, u) is the quaternion of the k-th bone of the u-th frame in frame i, q ^j (k, u) is the quaternion of the k-th bone of the u-th frame in frame j, q ^b (k, u) is a quaternion of the k-th bone of the u-th frame in the blending frame. However, blending is not performed on the route. Equation (24) is a calculation formula for slerp (spherical linear interpolation).

  The blending motion data 1_2 is data of a connection portion between the motion data 1 and the motion data 2.

  Next, node weights (node weights, metadata indicating the possibility of synchronization with music) are calculated. The node weight indicates the number of nodes that can be connected after the self node. Depending on the weight of the node, the node can be classified into three types of TYPE1, TYPE2, and TYPE3.

  A node classified as TYPE1 is a node that can connect nodes indefinitely after its own node. The weight of a node classified as TYPE1 is infinite. For example, a node that can be connected to a node that is included in a loop or a node that is included in a loop can be connected to the node that is included in this loop after its own node. Nodes can be linked.

  Nodes classified as TYPE2 are end nodes and attached nodes. A terminal node is a node that cannot connect nodes after its own node. For example, a node without a bidirectional edge and a unidirectional edge. The weight of the end node is zero. An attached node is a node in which only a one-way edge is connected to a terminal node or an attached node. The weight of the attached node is a value obtained by adding 1 to the weight of the child node (node connected immediately after the own node).

  A node classified as TYPE3 is a node that is not classified as either TYPE1 or TYPE2. Nodes classified as TYPE 3 are not used in the synchronization processing described later.

  For example, among the nodes 901 to 921 illustrated in FIG. 9, nodes classified as TYPE 1 are nodes 901 to 906, 908 to 913, and 915 to 917. The weight of these nodes is infinite. Among the nodes 901 to 921, nodes classified as TYPE 2 are nodes 907, 914, and 918 to 921, respectively. Nodes 907 and 914 are end nodes, and the weight is zero. Nodes 918 to 920 are attached nodes, the weight of node 920 is 1, the weight of node 919 is 2, and the weight of node 918 is 3.

  Here, the process of calculating the weight of the node will be described.

  Node weight calculation step S21: Among the nodes of the motion graph, a node included in the loop and a node capable of connecting nodes included in the loop are specified as a node of TYPE1. Thereafter, the process of the node weight calculation step S22 is performed.

In the node weight calculation step S21, for example, the process shown in FIG. 13 is performed to identify the loop.
(Step S101) The color of all nodes is set to “white”. Thereafter, the process proceeds to step S102.
(Step S102) The color of this node u is set to “gray”. Thereafter, the process proceeds to step S103.
(Step S103) If there is a child node v, the process proceeds to step S106; otherwise, the process proceeds to step S104.
(Step S104) The color of the current node u is set to “black”. Thereafter, the process proceeds to step S105.
(Step S105) This node u is set as a parent node. Thereafter, the process returns to step S103.
(Step S106) If the color of the node v is “white”, the process proceeds to step S107. Otherwise, the process proceeds to step S109.
(Step S107) The parent of the node v is set as the node u. Thereafter, the process proceeds to step S108.
(Step S108) This node u is set as a node v. Thereafter, the process returns to step S102.
(Step S109) It is determined that there is a loop, and the process is terminated.

  Node weight calculation step S22: Among the nodes of the motion graph excluding nodes classified as TYPE1, end nodes and attached nodes are identified, and these nodes are identified as nodes classified as TYPE2. Thereafter, the process of the node weight calculation step S23 is performed.

  Node weight calculation step S23: Among the nodes of the motion graph, nodes other than those classified as TYPE1 and TYPE2 are identified as nodes classified as TYPE3. Thereafter, the node weight calculation process is terminated.

  Through the above-described motion graph construction step, motion graph data of a certain genre tempo database (tempo iDB) is generated. The motion graph data includes information on nodes (beat frames) included in the motion graph, information on edges between nodes (bidirectional edges or unidirectional edges) (including edge weights), and two directions corresponding to bidirectional edges. Minute blending motion data and node weights.

  The motion graph generation unit 35 stores motion graph data generated for each tempo database of each genre in the database 12. Thereby, the motion graph data is stored in the database 12 for each tempo database of each genre.

  This completes the description of the metadata construction device 10. The processing of the metadata construction device 10 is executed offline, and the database 12 is constructed. Online processing of the video content generation apparatus 20 is performed using the database 12. Hereinafter, online processing of the video content generation apparatus 20 will be described.

  The video content generation device 20 generates video data synchronized with the music data of the music for which video content generation is to be distributed distributed by the streaming method, and reproduces the music data and the video data. FIG. 14 is a conceptual diagram showing the flow of music data distributed by the streaming method and the generated video data. The video content generation device 20 stores music data distributed by the streaming method in the music data storage unit 21 (back-buffer). When the music data for a predetermined time (for example, 10 seconds) is accumulated in the music data storage unit 21 (back-buffer), the video content generation device 20 generates video data synchronized with the music data, The music data and the generated video data are stored in the video content storage unit 28 (front-buffer) (synchronization processing is performed). Then, the video content generation device 20 reproduces the music data stored in the video content storage unit 28 (front-buffer) and the video data, while reproducing the music data for a predetermined time again as the music data storage unit 21 (back- When the data is stored in the buffer), the music data is synchronized.

  As described above, the video content generation apparatus 20 performs the synchronization process when the music data is accumulated for a certain period of time until the distribution of the music data is completed while reproducing the music data and the video data. Therefore, music data and video data synchronized with the music data can be reproduced without waiting until the distribution of the music data for one song is completed.

  In addition, if the processing time of the synchronization process is shorter than the “certain time” for a certain period of time, the music data and the video data should be reproduced without waiting time (in real time) after performing the synchronization process first. Can do.

  FIG. 15 is a schematic diagram showing the relationship between the processing time of the synchronization processing and the playback time of video content (music data and video data synchronized with the music data) generated by the synchronization processing. In the illustrated example, areas 1501 to 1503 are shown as the processing time of the synchronization process, and areas 1504 to 1506 are shown as the playback time of the video content generated by the synchronization process. The content generation device 20 plays back the video content generated by the synchronization process in the time of the area 1501 in the time of the area 1504. Also, the content generation device 20 generates video content by synchronization processing at the time of the area 1502 while reproducing the video content at the time of the area 1504. Then, the content generation apparatus 20 plays back the video content generated by the synchronization process in the time of the area 1502 in the time of the area 1505. Hereinafter, the video content generation device 20 performs the synchronization process and the reproduction in the same manner until the distribution of the music data by the streaming method is completed.

  Note that if the processing time of the synchronization processing is shorter than the playback time of the video content, during the synchronization processing as shown in FIG. 15 (for example, between the region 1501 and the region 1502, or between the region 1502 and the region 1503). Therefore, the “certain time” may be changed in accordance with the playback time of the video content.

  FIG. 16 shows the relationship between the processing time of the synchronization process and the playback time of the video content (music data and video data synchronized with this music data) generated by the synchronization process when the “certain time” is varied. It is the shown schematic. In the illustrated example, areas 1601 to 1603 are shown as the processing time of the synchronization processing, and areas 1604 to 1606 are shown as the playback times of the video content generated by the synchronization processing. The content generation apparatus 20 plays back the video content generated by the synchronization processing in the time of the area 1601 in the time of the area 1604. In addition, the content generation apparatus 20 generates the video content by the synchronization process at the time of the area 1602 while reproducing the video content at the time of the area 1604. In this way, by increasing the “certain time” so that the time of the area 1602 for performing the synchronization process becomes the same as the time of the area 1604 for playing back the video content, 1601 and the area 1602 or between the area 1602 and the area 1603). In addition, by lengthening the “certain time”, the length of the video content generated by the synchronization processing is also lengthened. In the illustrated example, the region 1605 is longer than the region 1604, and the region 1606 is longer than the region 1605.

  Hereinafter, each part of the video content generation apparatus 20 included in the video content generation system 1 shown in FIG.

[Music data storage]
The video content generation apparatus 20 distributes (inputs) music data of the music for which video content is to be generated from the music file 3 in a streaming manner. The music data storage unit 21 accumulates (stores) music data distributed by the streaming method.

[Music Analysis Department]
When the music data storage unit 21 stores music data for a certain period of time, the music analysis unit 22 reads out the music data from the music data storage unit 21 and analyzes it to acquire music feature values. In the present embodiment, using the technique described in Non-Patent Document 2, beat intervals and beat times are acquired from music data for a certain period of time. Further, in the present embodiment, the excitement of music data for a certain period of time is calculated by the equation (25).

  However, s (I, j) is the i-th signal of the j-th channel of the music, N (k) is the number of music signals of the k-th segment, and W is a fixed length (for example, 2). , I (k) is the rise of the k-th segment.

  Further, the music analysis unit 22 calculates the tempo of the music according to the equation (26). The tempo of music is the number of beats per minute.

Where TB _music is the beat interval (seconds).
The music analysis unit 22 stores the beat interval, beat time, tempo, and excitement of music data for a predetermined time playback in the music analysis data storage unit 212 as music feature amounts. In addition, the music analysis unit 22 stores music data for a certain period of time in the video content storage unit 28.

[Optimum path search unit]
First, the optimal path search unit 24 selects motion graph data that matches the music for which video content is to be generated, from the motion graph data in the database 12 of the metadata construction apparatus 10. In this selection, motion graph data that matches the tempo of the music for which the video content is to be generated is selected from the motion graph data of the genre that matches the music for which the video content is to be generated. The genre suitable for the music for which the video content is to be generated is input from the user or set in advance.

  Specifically, the optimum path search unit 24 calculates the music tempo analyzed by the music analysis unit 22 and the minimum tempo in the motion graph data of the selected genre by applying the above equation (18). Next, the optimum path search unit 24 selects motion graph data corresponding to the calculated value i of the equation (18) from motion graph data of a genre input by a user or set in advance.

  Next, the optimal path search unit 24 uses the selected motion graph data and the music feature value analyzed by the music analysis unit 22 (stored in the music analysis data storage unit 23) to connect the optimal motion graph nodes. Search for a unique path. Hereinafter, the optimal path searching method will be described.

[Start point selection step]
In the start point selection step, a node candidate (start point candidate node) that is the start point of the motion of the video content is selected from the nodes in the motion graph. As the start point candidate nodes, all the nodes in the motion graph corresponding to the first beat frame of each motion data are extracted. Therefore, there are usually a plurality of start point candidate nodes.

[Optimum path search step]
Next, in the optimum path search step, the optimum path from the start point candidate node on the motion graph is searched for each start point candidate node, and the path with the lowest cost is selected from among the optimum paths related to each start point candidate node. For this path search method, the path search technique described in Non-Patent Document 5 is used. The path search technique described in Non-Patent Document 5 searches for an optimal path by dynamic programming from a certain starting point. Details of the optimum path search step will be described below.

  First, the cost of each path from a certain starting point candidate node u to all the nodes i on the motion graph is initialized by equation (27). This operation is called the 0th shortest path calculation operation.

  However, InitS is a set of a plurality of starting points.

  Next, in the k-th shortest path calculation operation after the first time, the cost of the optimal path from the starting point candidate node u to all the nodes v on the motion graph is calculated by the equation (28).

  V is a set of nodes on the motion graph. shorttestPath (v, k) is the cost of the optimum path from the starting point candidate node u to the node v by the k-th shortest path calculation operation. edgeCost (i, v) is the edge cost from node i to node v. The formula for calculating the edge cost is (30).

  The first and subsequent shortest path calculation operations using equation (28) are repeated until the Kth time. Here, K is the number of beats included in music data for a certain period of time that is a video content generation target. The number of beats K included in music data for a certain period of time is equal to the total number of beat times of music data for a certain period of time. Since the beat time of music data for a certain period of time is stored in the music analysis data storage unit 212, the beat number K can be obtained by counting the stored number.

  The shortest path calculation operation using the above equations (27) and (28) is performed on all the start point candidate nodes. Next, the path with the minimum cost is selected from the results of the K-th shortest path calculation operation related to all the start point candidate nodes according to the equation (29).

  shorttestPath (v, K) is the cost of the optimum path from the starting point candidate node u to the node v by the Kth shortest path calculation operation. shorttestPath (K) is the cost of the path with the lowest cost (path from the start node u to the end node v).

  Here, TH is the number of beats until the music ends. As an estimation method of TH, for example, the number of beats included in one second is calculated, and the remaining time until the music ends (the length of music until the end) is divided by the number of beats included in one second. There is a way to estimate. If the number of beats until the music ends (the length of the music) is not known, TH is set to infinity.

  From equation (29), the weight w (v) of the node included in the path with the lowest cost satisfies the condition (condition w (v)> = TH) that is equal to or greater than the number of beats until the music is completed. The continuity of motion data is not interrupted during the process. For example, when the number of beats TH until the end of music is unknown, the weight of the node included in the path with the lowest cost is always infinite.

  The edge cost is calculated every time using equation (30).

In the optimum path search step, the path with the minimum cost selected by the above equation (29) is set as the optimum path of the search result. The K nodes included in the optimum path are one start node u, (K−2) transit nodes i, and one end node v. Here, since there are usually a plurality of start point candidate nodes, there are as many optimal paths as the search results as the number of start point candidate nodes. From these optimum paths, the path with the lowest cost and its starting point are selected as the optimum path of the final result. The K nodes included in the optimal path of the final result are one optimal start point node u ^opt , (K−2) via nodes i ^opt , and one end point node v ^opt .

[Synchronization part]
The synchronization unit 25 generates synchronization information that associates motion data and music data according to the optimum path searched by the optimum path search unit 24. Details of the synchronization information generation method will be described below.

First, K nodes included in the optimum path searched by the optimum path search unit 24 (one start point node u ^opt , (K−2) transit nodes i ^opt , and one end point node v ^opt ). Time between adjacent beat frames in the optimal path order for K beat frames (one start beat frame, (K-2) via beat frames, and one end beat frame) corresponding to Ask for. Further, the frame rate between each adjacent beat frame is obtained. In addition, the time between beats that are temporally adjacent to each other is obtained for K beats of the music for which video content is to be generated.

  Next, an adjustment to increase / decrease the frame rate of motion is performed using equation (31) so that the beat interval of motion is equal to the beat interval of music. FIG. 17 shows a conceptual diagram of processing for adjusting the frame rate of motion. Expression (31) is a calculation expression for calculating a frame rate between the nth beat frame and the (n + 1) th beat frame (where n is a natural number from 1 to (K−1)). ).

^However, _{t motion node2} the time of the beat frame among target adjacent beat ^frames, _{t motion node1} is the time of the beat frame after out of the adjacent beat frames. t ^music _{node 2} is the time of the previous beat of the adjacent beats of the ^music , and t ^music _{node 1} is the time of the later beat of the adjacent beats. rate_old is the original frame rate. rate_new is the adjusted frame rate.

  The synchronization unit 25 uses the above-described synchronization information generation method to generate one start point beat frame that is the start point of the motion of the video content, one end point beat frame that is the end point of the motion of the video content, and the end point from the start point beat frame The (K-2) via-beat frames that are passed through to the beat frame and the adjusted frame rate between the adjacent beat frames are obtained. The synchronization unit 25 stores the start point beat frame information, the transit beat frame information, the end point beat frame information, the adjusted frame rate information, and the blending motion data between the beat frames in the synchronization information storage unit 26 as synchronization information. To do. Note that the blending motion data need only be data in the direction along the optimal path of the search result of the optimal path search unit 24.

[Video data generator]
Based on the synchronization information stored in the synchronization information storage unit 26, the video data generation unit 27 generates video data to be reproduced together with the music data of the music for which video content is to be generated. Specifically, the motion data necessary from the start point beat frame to the end point beat frame via the via beat frame is acquired from the motion database 2.

  Next, a portion connecting the acquired motion data (a portion corresponding to a bidirectional edge) is replaced with blending motion data. At this time, the movement data is translated in the direction and the root coordinate of the movement data at the connection portion of the movement data. When motion data is connected, the root coordinates of each motion data remain as local coordinates unique to each motion data. In this state, the reproduced image of the motion data after connection does not move smoothly because the root coordinates are not matched. For this reason, the root coordinate of the subsequent motion data is offset to the position represented by the last frame of the previous motion data in the connection portion of the motion data. As a result, the interpolation processing is performed at the connection portion of the motion data so that the reproduced image of the motion data after the connection has a smooth motion. Similarly, when motion data is connected, the root direction of each motion data is offset to the direction expressed by the last frame of the previous motion data.

  Next, the adjusted frame rate information between adjacent beat frames is added to the connected motion data. The video data generation unit 27 stores the generated video data in the video content storage unit 28 in association with the stored music data for a certain period of time.

[Playback section]
The reproduction unit 29 reproduces both the video data stored in association with the video content storage unit 28 and the music data for a certain period of time of the music for which the video content is to be generated. At this time, the playback unit 29 sets the frame rate between adjacent beat frames in accordance with the frame rate information added to the video data. Thereby, the video data and the music data are reproduced in synchronism with each other's beats. Note that the playback unit 29 may be a device independent of the video content generation device 20.

  In the above-described example, the video content generation apparatus 20 performs the synchronization process sequentially for each piece of music data for a certain period of time. However, the present invention is not limited to this. For example, as shown in FIG. 18, the synchronization process may be performed on music data including the same playback portion by shortening the time interval for performing the synchronization process.

  FIG. 18 is a schematic diagram showing the relationship between music data for a certain period of time to be synchronized and the path selected by the synchronization. In the example shown in the drawing, an area 1801 is shown as the time of the playback portion of the music data that has been streamed and played for a certain period of time at time a. Further, at time b, an area 1802 is shown as the time of the reproduction portion of music data for a certain period of time that has been streamed. Also, at time c, an area 1801 is shown as the time of the playback portion of the music data that has been streamed and played for a certain period of time. In addition, a path 1804 selected by synchronization processing based on music data for a certain period of time that is streamed and distributed at time a is shown. In addition, a path 1805 selected by synchronization processing based on music data for a certain period of time that has been streamed and distributed at time b is shown.

  In this embodiment, music data is distributed by a streaming method. Therefore, by shortening the time interval for performing the synchronization process, it is possible to select a path based on newly distributed music data, so it is possible to select a path for generating more relevant video data Can do.

  Note that the metadata construction apparatus 10 and the video content generation apparatus 20 according to the present embodiment may be realized by dedicated hardware, or may be configured by a computer system such as a personal computer, as shown in FIG. The functions may be realized by executing a program for realizing the functions of the respective sections of the metadata construction device 10 and the video content generation device 20.

In addition, an input device, a display device, and the like are connected to the metadata construction device 10 and the video content generation device 20 as peripheral devices. Here, the input device refers to an input device such as a keyboard and a mouse. The display device refers to a CRT (Cathode Ray Tube), a liquid crystal display device or the like.
The peripheral device may be directly connected to the metadata construction device 10 and the video content generation device 20, or may be connected via a communication line.

Also, a program for realizing each step performed by the metadata construction device 10 and the video content generation device 20 shown in FIG. 1 is recorded on a computer-readable recording medium, and the program recorded on the recording medium is stored in a computer system. The video content generation processing may be performed by causing the video content to be read and executed. Here, the “computer system” may include an OS and hardware such as peripheral devices.
Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.
“Computer-readable recording medium” refers to a flexible disk, a magneto-optical disk, a ROM, a writable nonvolatile memory such as a flash memory, a portable medium such as a DVD (Digital Versatile Disk), and a built-in computer system. A storage device such as a hard disk.

Further, the “computer-readable recording medium” means a volatile memory (for example, DRAM (Dynamic DRAM) in a computer system that becomes a server or a client when a program is transmitted through a network such as the Internet or a communication line such as a telephone line. Random Access Memory)), etc., which hold programs for a certain period of time.
The program may be transmitted from a computer system storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, and what is called a difference file (difference program) may be sufficient.

As mentioned above, although embodiment of this invention was explained in full detail with reference to drawings, the specific structure is not restricted to this embodiment, The design change etc. of the range which does not deviate from the summary of this invention are included.
For example, the mobile terminal may be configured to include each unit of the metadata construction device 10 and each unit of the video content generation device 20. In addition, each unit of the metadata construction device 10, each unit of the video content generation device 20, a distribution unit that distributes music data and video data stored in the video content storage unit 28 to other devices such as a mobile terminal, You may comprise as a video content delivery apparatus provided with.
In the above-described embodiment, human motion data is handled. However, the present invention can be applied to motion data of various objects. Here, the object includes people, animals, plants and other living things, and things other than living things (robots and the like).

  The present invention can also be used for generating three-dimensional content.

  DESCRIPTION OF SYMBOLS 1 ... Video content generation system, 2 ... Motion database, 3 ... Music file, 10 ... Metadata construction apparatus, 11 ... Motion analysis part, 12 ... Database, 20 ... Video content generation device, 21 ... music data storage unit, 22 ... music analysis unit, 23 ... music analysis data storage unit, 24 ... optimal path search unit, 25 ... synchronization unit, 26 ..Synchronization information storage unit, 27... Video data generation unit, 28... Video content storage unit, 29... Reproduction unit, 31 .. beat extraction unit, 32.・ ・ ・ Surge calculation unit 34 swell information storage unit 35 Motion graph generation unit 311 Physical quantity conversion unit 312 Data division unit 313 Short time principal component analysis Part, 314 ... Component coordinate connecting portion, 315 ... beat estimator, 316 ... post-processing unit

Claims (17)

A motion graph construction unit that constructs a motion graph including nodes, edges, and edge weights based on feature quantities extracted from motion capture data applicable to video content;
A metadata construction unit for calculating metadata indicating the possibility of synchronization with music in each node of the motion graph;
A music analysis unit that inputs music data of a video content generation target song by a predetermined amount in a playback order and acquires music feature values from the input music data of the predetermined amount;
On the motion graph, among nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data, a path in which the final node of the node group has a certain synchronization possibility can be connected. An optimal path search unit to search;
A synchronization unit that generates synchronization information that associates the motion data corresponding to the path searched by the optimal path search unit with the predetermined amount of music data;
A video data generating unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information;
Equipped with a,
The metadata construction unit generates a motion graph including a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to the video content, and an edge indicating connectivity of the node. For each included node, there is a weight calculation means for calculating a weight according to the number of nodes that can be connected after the node,
The optimum path search unit searches for the path based on the weight calculated by the weight calculation unit.
The video content generation system which is characterized a call. The weight calculating means calculates a node in which the number of nodes that can be connected after the node is zero and a node in which the number of nodes that can be connected after the node is infinite. The video content generation system according to claim 1 , wherein a weight of another node is estimated from the node. The optimal path search unit is a path connecting nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph, and the weight of the last node is a certain amount or more. video content generation system according to claim 1 or claim 2, characterized in that searching for a path is. The optimal path search unit is a path connecting nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph, and determines only a part of the head. The video content generation system according to claim 1 or 2 , wherein: The video according to any one of claims 1 to 4 , wherein the music analysis unit determines a predetermined amount of music data according to a calculation time of the optimum path search unit and a music data rate. Content generation system. A storage unit for storing the music data of the input video content generation target music;
The music analysis unit has a predetermined amount of the music data used in the next processing, which is a smaller data amount of the maximum storage capacity of the storage unit or the storage amount specified by the user, and the video data 6. The video content generation system according to claim 5 , wherein a data amount that can be processed within a reproduction time of the video data generated by the generation unit is set. For each node included in a motion graph composed of a frame node having the same motion feature amount among the feature amounts extracted from motion capture data applicable to video content and an edge indicating connectivity of the node, the node A metadata construction apparatus comprising weight calculation means for calculating a weight in accordance with the number of nodes that can be connected after. The weight calculating means calculates a node in which the number of nodes that can be connected after the node is zero and a node in which the number of nodes that can be connected after the node is infinite. The metadata construction apparatus according to claim 7 , wherein a weight of another node is estimated from the node. A motion graph composed of a frame node having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content, and an edge indicating connectivity of the node, and a node included in the motion graph An acquisition unit that acquires a weight according to the number of nodes that can be connected after the node,
A music analysis unit that inputs music data of a video content generation target song by a predetermined amount in a playback order and acquires music feature values from the input music data of the predetermined amount;
On the motion graph, an optimal path search unit that searches for a path connecting nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data based on the weights;
A synchronization unit that generates synchronization information that associates the motion data corresponding to the path searched by the optimal path search unit with the predetermined amount of music data;
A video data generating unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information;
A video content generation apparatus comprising: The optimal path search unit is a path connecting nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph, and the weight of the last node is a certain amount or more. The video content generation device according to claim 9 , wherein a path that is is searched. The optimal path search unit is a path connecting nodes having motion feature amounts corresponding to music feature amounts of the predetermined amount of music data on the motion graph, and determines only a part of the head. The video content generation device according to claim 9 . The said music analysis part determines the predetermined amount of music data by the calculation time of the said optimal path | pass search part, and the data rate of music. The one of Claim 9 to 11 characterized by the above-mentioned. Video content generation device. A storage unit for storing the music data of the input video content generation target music;
The music analysis unit has a predetermined amount of the music data used in the next processing, which is a smaller data amount of the maximum storage capacity of the storage unit or the storage amount specified by the user, and the video data 13. The video content generation apparatus according to claim 12 , wherein the video content generation apparatus has a data amount that can be processed within a reproduction time of the video data generated by the generation unit. A motion graph construction unit that constructs a motion graph including nodes, edges, and edge weights based on feature quantities extracted from motion capture data applicable to video content;
A metadata construction unit for calculating metadata indicating the possibility of synchronization with music in each node of the motion graph;
A music analysis unit that inputs music data of a video content generation target song by a predetermined amount in a playback order and acquires music feature values from the input music data of the predetermined amount;
On the motion graph, among nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data, a path in which the final node of the node group has a certain synchronization possibility can be connected. An optimal path search unit to search;
A synchronization unit that generates synchronization information that associates the motion data corresponding to the path searched by the optimal path search unit with the predetermined amount of music data;
A video data generating unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information;
Equipped with a,
The metadata construction unit generates a motion graph including a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to the video content, and an edge indicating connectivity of the node. For each included node, there is a weight calculation means for calculating a weight according to the number of nodes that can be connected after the node,
The optimum path search unit searches for the path based on the weight calculated by the weight calculation unit.
Mobile terminal, wherein a call. A motion graph construction unit that constructs a motion graph including nodes, edges, and edge weights based on feature quantities extracted from motion capture data applicable to video content;
A metadata construction unit for calculating metadata indicating the possibility of synchronization with music in each node of the motion graph;
A music analysis unit that inputs music data of a video content generation target song by a predetermined amount in a playback order and acquires music feature values from the input music data of the predetermined amount;
On the motion graph, among nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data, a path in which the final node of the node group has a certain synchronization possibility can be connected. An optimal path search unit to search;
A synchronization unit that generates synchronization information that associates the motion data corresponding to the path searched by the optimal path search unit with the predetermined amount of music data;
A video data generating unit that generates video data to be reproduced together with the predetermined amount of music data based on the synchronization information;
A distribution unit that distributes the predetermined amount of music data and video data reproduced together with the predetermined amount of music data;
Equipped with a,
The metadata construction unit generates a motion graph including a node of a frame having the same motion feature amount among feature amounts extracted from motion capture data applicable to the video content, and an edge indicating connectivity of the node. For each included node, there is a weight calculation means for calculating a weight according to the number of nodes that can be connected after the node,
The optimum path search unit searches for the path based on the weight calculated by the weight calculation unit.
The video content delivery device comprising a call. For each node included in a motion graph composed of a frame node having the same motion feature amount among the feature amounts extracted from motion capture data applicable to video content and an edge indicating connectivity of the node, the node A computer program for causing a computer to execute a weight calculation step of calculating a weight according to the number of nodes that can be connected after the. A motion graph composed of a frame node having the same motion feature amount among feature amounts extracted from motion capture data applicable to video content, and an edge indicating connectivity of the node, and a node included in the motion graph An acquisition step of acquiring a weight according to the number of nodes that can be linked after the node;
A music analysis step of inputting music data of the video content generation target music piece by a predetermined amount in a playback order and obtaining a music feature amount from the input music data of the predetermined amount;
On the motion graph, an optimal path search step of searching for a path connecting nodes having a motion feature amount corresponding to the music feature amount of the predetermined amount of music data based on the weight;
A synchronization step of generating synchronization information that associates the motion data corresponding to the path searched in the optimal path search step with the predetermined amount of music data;
A video data generating step for generating video data to be reproduced together with the predetermined amount of music data based on the synchronization information;
A computer program for causing a computer to execute.

Images