JP6547807B2 - Image generation apparatus, image generation method and program - Google Patents

Description

  The present invention relates to an image generation apparatus that generates animation data, an image generation method, and a program.

  Conventionally, a technique is known in which a plurality of markers are set on a person, the motion of each marker is measured, and animation data is acquired.

JP 2012-248233 A

However, in the prior art, for example, even when it is desired to change the motion of a part of animation data, it has been necessary to set a marker on a person each time to acquire new animation data.
Therefore, the present invention has been made in view of the above-described situation, and has as its object to easily make acquired animation data correspond to a user's desired motion.

And a data acquisition means for acquiring multiple animation data,
An input unit for inputting an arbitrary value of an index related to at least one of the strength of kicking of the foot and the inclination of the body axis when running animation data;
First generation means for generating animation data according to an arbitrary value of the index inputted by the input means, using a plurality of animation data obtained by the data acquisition means ;
Characterized in that it comprises a.

  According to the present invention, the acquired animation data can be easily made to correspond to the user's desired motion.

1 shows the configuration of an image generation system according to an embodiment of the present invention. It is a block diagram which shows the structure of the hardware of the image generation apparatus which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the hardware of the terminal device which concerns on one Embodiment of this invention. FIG. 5 is a functional block diagram showing a functional configuration for performing pre-preparation processing among the functional configurations of the image generation apparatus. It is a schematic diagram which shows the outline | summary of a skeleton. It is a functional block diagram which shows the functional structure for performing a skeleton drawing process among the functional structures of a terminal device. It is a flowchart explaining an example of the flow of the prior preparation process which the image generation apparatus of FIG. 2 which has the functional structure of FIG. 4 performs. It is a flowchart explaining an example of a flow of normalization skeleton acquisition processing among prior preparation processing. It is a flowchart explaining an example of a flow of standard skeleton acquisition processing among prior preparation processing. It is a flow chart explaining an example of a flow of index correlation processing among prior preparation processing. It is a flow chart explaining an example of a flow of axis parameter decision processing among prior preparation processing. It is a flowchart explaining an example of the flow of the skeleton drawing process which the terminal device of FIG. 3 which has the functional structure of FIG. 6 performs. It is a schematic diagram which shows the example of a display screen displayed by skeleton drawing process. It is a flowchart explaining an example of a flow of synthetic skeleton generation processing among skeleton drawing processing. It is a schematic diagram which shows the skeleton space of time series. It is a schematic diagram which shows the skeleton space of a flame | frame (frame number = k) different from FIG. It is a schematic diagram which shows the synthetic | combination skeleton in skeleton space, (a) is a state where standard skeleton operation | movement was set to skeleton space, (b) is a feature skeleton according to the index value regarding different motion, (c) is It is a figure which shows a synthetic | combination skeleton. It is a flowchart explaining the flow of the 1st arrow drawing process which the terminal device of FIG. 3 which has the functional structure of FIG. 5 performs. It is a figure which shows the state by which the skeleton is drawn repeatedly. It is a schematic diagram which shows the example of a display screen with which the viewpoint direction was changed. It is a schematic diagram which shows the example of a display screen in the time different from FIG. It is a flowchart explaining the flow of the 2nd arrow drawing process which the terminal unit of Drawing 3 which has functional composition of Drawing 5 performs. It is a figure which shows the state by which the skeleton is drawn repeatedly. It is a schematic diagram which shows the example of a display screen with which the arrow which shows the direction of a change of the site | part of a skeleton was displayed. It is a flowchart explaining the flow of the corresponding index alerting | reporting process which the terminal device of FIG. 3 which has the function structure of FIG. 5 performs. It is a schematic diagram which shows the example of a display screen of the skeleton by which the movable joint was drawn large compared with another joint. It is a flowchart explaining the flow of joint interruption movement processing performed as interruption processing in correspondence index information processing. It is a flowchart explaining the flow of the completion | finish interruption processing performed as an interruption processing in corresponding index alerting | reporting processing and joint interruption movement processing. It is a flowchart explaining the flow of the real-time display process which the terminal device of FIG. 3 which has the functional structure of FIG. 5 performs.

  Hereinafter, embodiments of the present invention will be described using the drawings.

In the image generation system according to the present embodiment, by detecting the position of a marker attached to a person who is a model, a plurality of motion capture data obtained by capturing the motion of a human being is acquired, and based on these, a skeleton motion which is taken as a standard. Get (the behavior of the model represented by the skeleton consisting of bones and joints). In addition, in the image generation system according to the present embodiment, an index (e.g., a maximum kick acceleration) related to motion is set with respect to a skeleton motion as a standard, and axis parameters are set for the index. The specific index value of the index relating to the motion is set based on the motions of a plurality of skeletons or based on the parameters (for example, kick acceleration etc.) of the motion measured together when acquiring the motion capture data. The axis parameter is a value indicating the degree of movement set based on the distribution of index values in a plurality of skeleton operations. Then, when the user inputs a change in index value regarding exercise, the standard skeleton operation is changed according to the axis parameter. The post-change skeleton operation identifies and displays the change from the standard skeleton operation.
This makes it possible to change the skeleton (animation) obtained by measuring the motion into a desired motion.
The axis parameters are obtained based on a plurality of skeletons. Therefore, by changing the motion according to the axis parameter, the change in motion of the standard skeleton becomes based on the motion of human beings, and it becomes possible to realize a more appropriate change in motion.
Furthermore, it is possible to present the user with an easy-to-understand presentation of the changed portion when the standard skeleton operation is changed.

FIG. 1 shows the configuration of an image generation system 1 according to an embodiment of the present invention.
In FIG. 1, an image generation system 1 includes an image generation apparatus 100 and a terminal device 200, and the image generation apparatus 100 and the terminal device 200 are configured to be communicable via a network 300 such as the Internet.

FIG. 2 is a block diagram showing the hardware configuration of the image generation apparatus 100 according to an embodiment of the present invention.
The image generation apparatus 100 is configured by, for example, a server.

  The image generation apparatus 100 includes a central processing unit (CPU) 111, a read only memory (ROM) 112, a random access memory (RAM) 113, a bus 114, an input / output interface 115, an input unit 116, and an output unit. A storage unit 118, a communication unit 119, and a drive 120 are provided.

  The CPU 111 executes various processes according to a program stored in the ROM 112 such as a program for preparatory process (described later) or a program loaded from the storage unit 118 to the RAM 113.

  The RAM 113 appropriately stores data and the like necessary for the CPU 111 to execute various processes.

  The CPU 111, the ROM 112 and the RAM 113 are mutually connected via a bus 114. An input / output interface 115 is also connected to the bus 114. An input unit 116, an output unit 117, a storage unit 118, a communication unit 119, and a drive 120 are connected to the input / output interface 115.

The input unit 116 includes various buttons and the like, and inputs various information in accordance with a user's instruction operation.
The output unit 117 is configured by a display, a speaker, and the like, and outputs an image and sound.
The storage unit 118 is configured by a hard disk, a dynamic random access memory (DRAM), or the like, and stores data of skeleton data, index values of indices related to movement, and data such as axis parameters.
The communication unit 119 controls communication with another device via a network including the Internet.

  A removable medium 131 composed of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is appropriately attached to the drive 120. The program read from the removable medium 131 by the drive 120 is installed in the storage unit 118 as necessary. The removable medium 131 can also store various data such as image data stored in the storage unit 118 in the same manner as the storage unit 118.

FIG. 3 is a block diagram showing the hardware configuration of the terminal device 200 according to an embodiment of the present invention.
The terminal device 200 is configured of, for example, a portable terminal called a smart phone.
In FIG. 3, the terminal device 200 includes a CPU 211, a ROM 212, a RAM 213, a bus 214, an input / output interface 215, an imaging unit 216, an input unit 217, an output unit 218, a storage unit 219, and a communication unit. 220 and a drive 221.

  The CPU 211 executes various processes according to a program stored in the ROM 212 such as a program for skeleton drawing processing or a program loaded from the storage unit 219 to the RAM 213.

  The RAM 213 appropriately stores data and the like necessary for the CPU 211 to execute various processes.

  The CPU 211, the ROM 212 and the RAM 213 are connected to one another via a bus 214. An input / output interface 215 is also connected to the bus 214. An imaging unit 216, an input unit 217, an output unit 218, a storage unit 219, a communication unit 220, and a drive 221 are connected to the input / output interface 215.

Although not shown, the imaging unit 216 includes an optical lens unit and an image sensor.
The optical lens unit is configured of a lens that collects light, such as a focus lens or a zoom lens, in order to capture an object.
The focus lens is a lens that forms an object image on the light receiving surface of the image sensor. The zoom lens is a lens that freely changes the focal length in a certain range.
The optical lens unit is also provided with peripheral circuits for adjusting setting parameters such as focus, exposure, white balance, etc., as necessary.

The image sensor includes a photoelectric conversion element, an AFE (Analog Front End), and the like.
The photoelectric conversion element is composed of, for example, a complementary metal oxide semiconductor (CMOS) type photoelectric conversion element or the like. A subject image is incident on the photoelectric conversion element from the optical lens unit. Therefore, the photoelectric conversion element photoelectrically converts (captures) an object image, accumulates an image signal for a certain period of time, and sequentially supplies the accumulated image signal as an analog signal to the AFE.
The AFE performs various signal processing such as A / D (Analog / Digital) conversion processing on this analog image signal. A digital signal is generated by various signal processing, and is output as an output signal (data of a captured image) of the imaging unit 216.

The input unit 217 includes various buttons and the like, and inputs various information in accordance with a user's instruction operation. Further, the input unit 217 includes a microphone, an A / D conversion circuit, and the like, and outputs data of voice input through the microphone to the CPU 11 or the storage unit 219.
The output unit 218 includes a display, a speaker, a D / A conversion circuit, and the like, and outputs an image and sound.
The storage unit 219 is configured of a hard disk, a DRAM, or the like, and stores an image database or the like storing data and attributes of various images.
The communication unit 220 controls communication with other devices via a network including the Internet.

  A removable medium 231 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory or the like is appropriately attached to the drive 221. The program read from the removable medium 231 by the drive 221 is installed in the storage unit 219 as necessary. The removable media 231 can also store various data such as image data stored in the storage unit 219 in the same manner as the storage unit 219.

[Functional Configuration of Image Generating Device]
FIG. 4 is a functional block diagram showing a functional configuration for executing the preparatory process in the functional configuration of the image generation apparatus 100. As shown in FIG.
The preparatory process is a series of processes for acquiring a standard skeleton (hereinafter referred to as “standard skeleton”) from motion capture data obtained by motion capture and setting axis parameters related to motion with respect to the standard skeleton. is there.
In the following, a case in which animation data of a person who is running will be described as an example.

When the preparatory process is executed, the normalized skeleton acquisition unit 151, the standard skeleton acquisition unit 152, the index association processing unit 153, and the axis parameter determination unit 154 function as the animation acquisition unit 150 in the CPU 111. .
Even when part of the functions of the normalized skeleton acquisition unit 151, the standard skeleton acquisition unit 152, the index association processing unit 153, and the axis parameter determination unit 154 is transferred to a functional unit that performs image processing such as GA (Graphic Accelerator). Good.

  The normalized skeleton acquiring unit 151 executes a normalized skeleton acquiring process described later. Specifically, the normalized skeleton acquiring unit 151 detects data of motion of a person obtained by motion capture (motion capture data) and force detected by a force plate device (device that measures force) for a plurality of skeleton motions. Acquire data of (ground reaction force here). In the present embodiment, it is assumed that the motion capture data and the ground reaction force data acquired in advance are acquired by the normalized skeleton acquisition unit 151. However, the image generation device 100 may be provided with a motion capture function and a force plate device, and motion capture data and ground reaction force data may be measured. Also, the normalized skeleton acquiring unit 151 generates data of skeleton operation from the motion capture data, and normalizes the generated data of skeleton operation in one cycle of the running form. Thereby, a plurality of normalized skeleton operations (normalized skeleton operations) are acquired.

FIG. 5 is a schematic view showing an outline of a skeleton.
Skeleton generally refers to the entire skeleton of animals and humans, and is used to express motion with the skeleton itself or to animate a computer graphics-bound model. It has a hierarchical structure and is composed of a joint (joint) that is a movable part and a bone that is a rigid body.
In FIG. 5, a skeleton S has a configuration in which a root joint J201 serving as a reference, a bone B201a, a joint J202, a bone B202a, and a joint J203 are connected in this order.

When the skeleton S includes posture change on the time axis, a skeleton operation which is a skeleton that changes in time is configured. The skeleton operation is composed of a plurality of frames. The time axis variable of the skeleton operation is XYZ coordinate position as the root of the root joint J201, coordinate conversion information from world coordinates as the joint of the root joint J201 to local coordinates of the root joint J201 In this embodiment, it is represented as four-dimensional quaternion.) Total 12-dimensional information of coordinate conversion information of joint J 202 (coordinate conversion information from root joint J 201 to joint J 202). By using quaternions, it is possible to perform calculations to position coordinates at high speed.
Moreover, since the length of the bone is a rigid body, it does not change with time and becomes a constant. The above-mentioned coordinate conversion information means the rotation direction in which the bone extends from the joint. In the following, when all joints are referred to, root joints and joints are included.

  Returning to FIG. 4, the standard skeleton acquisition unit 152 executes a standard skeleton acquisition process described later. Specifically, the standard skeleton acquisition unit 152 extracts time axis parameters (position, rotation, etc.) of each joint in each normalized skeleton, and calculates an average value at each joint. Then, the standard skeleton acquisition unit 152 sets the calculated average value as a time axis parameter for the corresponding joint of the standard skeleton. Set the average for bones as well. Thereby, a standard skeleton operation (standard skeleton operation) is set.

  The index association processing unit 153 executes an index association process described later. Specifically, the index association processing unit 153 acquires an index value of an index related to a specific motion from motion capture data in a skeleton operation. Here, it is assumed that the index association processing unit 153 acquires the average front and rear body axes (the average value of the inclinations of the body axes from the vertical direction to the traveling direction). In addition, the index association processing unit 153 acquires an index value (here, an index value representing the maximum kick force) related to motion from the measurement result (ground reaction force) of the force plate device in the skeleton operation. Then, the index association processing unit 153 associates the acquired index values with all the skeleton operations.

  The axis parameter determination unit 154 selects, as the upper side feature skeleton operation and the lower side feature skeleton operation, an index value relating to motion in the upper and lower constant ratios (here, 20%) in all normalized skeleton operations. Then, in the selected upper feature skeleton operation and lower feature skeleton operation, the upper average value and the lower average value of the index value are calculated. Then, the axis parameter determination unit 154 excludes, from the normalized skeleton operation, those whose average value and the Euclidean distance between the normalized skeleton operation are larger than the specified value (threshold value) D. Further, the axis parameter determination unit 154 selects the upper feature skeleton operation and the lower feature skeleton operation again in the normalized skeleton operation after exclusion, and calculates the upper average value and the lower average value in the index value. Exclude the normalized skeleton operation based on the threshold D. The axis parameter determination unit 154 executes the process for such exclusion several times (here, three times). Thereby, in the data of the skeleton operation having a large number of dimensions, the upper and lower index values can be selected with higher accuracy.

  The axis parameter determination unit 154 executes an axis parameter determination process described later. Specifically, the axis parameter determination unit 154 sets the upper average value and the lower average value obtained as described above for the upper feature skeleton operation and the lower feature skeleton operation of each index value to each index value. Are stored in the storage unit 118 as upper reference values and lower reference values of axis parameters corresponding to

[Functional configuration of terminal device]
FIG. 6 is a functional block diagram showing a functional configuration for executing a skeleton drawing process among the functional configurations of the terminal device 200.
The skeleton drawing process is a series of processes for generating a skeleton operation (synthesis skeleton operation) in which the degree of movement is changed according to the input index value, and drawing the generated synthesis skeleton operation.

When the skeleton drawing process is performed, in the CPU 211, the index value acquisition unit 251, the combined skeleton generation unit 252, and the drawing processing unit 253 function.
Note that some of the functions of the index value acquisition unit 251, the combined skeleton generation unit 252, and the drawing processing unit 253 may be transferred to a functional unit that performs image processing such as GA (Graphic Accelerator).
The index value acquisition unit 251 acquires an index value (index value after change) input by the user on the screen of the user interface. For example, when “40” is input as the index value of the maximum kick acceleration by the user when the index value of the kick force of the standard skeleton operation is “25” on the screen of the user interface, the index value acquisition unit 251 Acquire "40" as an index value of the maximum kick acceleration after change.

  The synthesis skeleton generation unit 252 executes synthesis skeleton generation processing described later. Specifically, the synthesis skeleton generation unit 252 changes the index value of the standard skeleton based on the changed index value acquired by the index value acquisition unit 251, and generates a synthesis skeleton operation. For example, when the index value “40” of the maximum kick acceleration after the change is acquired by the index value acquisition unit 251, the composite skeleton generation unit 252 changes the maximum kick acceleration in the standard skeleton from “25” to “40”. Generate a skeleton action and make it a synthetic skeleton action.

  The drawing processing unit 253 displays the synthetic skeleton motion generated by the synthetic skeleton generation unit 252 as an animation. At this time, the drawing processing unit 253 visually identifies and displays the index changed from the standard skeleton operation in the synthesis skeleton operation. For example, when the maximum kick acceleration of the combined skeleton operation is changed from the standard skeleton operation, the drawing processing unit 253 displays an arrow on a portion (for example, a leg) involved in the maximum kick acceleration to change the standard skeleton operation. Identify and display the

[Operation]
Next, the operation will be described.
[Prepare process]
FIG. 7 is a flowchart for explaining an example of the flow of preliminary preparation processing executed by the image generation apparatus 100 of FIG. 2 having the functional configuration of FIG. 4.
The preparatory process is started in response to the activation of the preparatory process being input via the terminal device 200 or the like.

When the preliminary preparation process is started, in step S101, the normalized skeleton acquisition unit 151 executes a normalized skeleton acquisition process (described later).
In step S102, the standard skeleton acquisition unit 152 executes a standard skeleton acquisition process (described later).
In step S103, the index association processing unit 153 executes an index association process (described later).
In step S104, the axis parameter determination unit 154 executes an axis parameter determination process (described later).

In step S105, the axis parameter determination unit 154 determines whether or not the axis parameter determination process has been completed for all the indices.
If the axis parameter determination process has not been completed for all the indices, it is determined as NO in step S105, and the process returns to step S104.
On the other hand, if the axis parameter determination process has been completed for all the indices, it is determined as YES in step S105, and the preliminary preparation process ends.

[Normalized skeleton acquisition processing]
Next, the normalized skeleton acquisition process executed in step S101 of the preliminary preparation process will be described.
FIG. 8 is a flow chart for explaining an example of the flow of normalized skeleton acquisition processing in the preliminary preparation processing.
When the normalized skeleton acquisition process is started, in step S201, the normalized skeleton acquisition unit 151 calculates motion capture data and ground reaction force data measured by the optical motion capture device and the force plate device (operation during running) To get the

Specifically, the normalized skeleton acquiring unit 151 acquires the form of the running motion represented by the motion of the marker attached to the measurement target of the motion from the optical motion capture device. An optical motion capture device is a device that tracks a marker attached to a site where movement of a person is to be measured with a large number of cameras.
Here, the marker is mounted at a position at which a joint site for constructing a skeleton can be estimated in a person to be measured for movement. That is, motion capture makes it possible to track the joint position of the skeleton.

Further, the normalized skeleton acquiring unit 151 acquires data of ground reaction force during running from the force plate device.
As an example, the force plate device is embedded in the ground, and a person to be measured travels thereon to acquire information on ground reaction force (a force that the ground gives to the object to be measured). From the ground reaction force, it is possible to obtain a kicking force in the vertical direction from the object to be measured to the ground and a propulsive force in the traveling direction.
Since the ground reaction force is a force, it can be derived from the following equation (1) according to Newton's law of motion.
F = m × a (1)
Here, "F" represents force, "m" represents mass, and "a" represents acceleration. Since "F" depends on mass, it depends on the weight of the person to be measured. Therefore, by treating the ground reaction force as acceleration instead of force, it becomes an index independent of the weight of the person to be measured.

In the present embodiment, since the maximum kick acceleration (the largest among the time-series kick accelerations) is used as the motion index, an index value representing the maximum kick acceleration is stored in association with the motion capture data. Use in the process of
In step S202, the normalized skeleton acquisition unit 151 acquires a skeleton operation. Specifically, the normalized skeleton acquisition unit 151 generates a skeleton operation from motion capture data acquired from the motion capture device. The normalized skeleton acquiring unit 151 estimates the joint position of the skeleton from the motion capture data, and sets a joint between the joints as a bone. However, when the bone is determined in this manner, the bone may not be rigid and may expand and contract, and the normalized skeleton acquiring unit 151 may estimate the joint while constraining the bone to be a rigid body. Global optimization is performed to generate skeleton motion so as to satisfy the position as much as possible.

In step S203, the normalized skeleton acquisition unit 151 normalizes the skeleton operation.
Specifically, the normalized skeleton acquiring unit 151 normalizes the skeleton operation in accordance with the periodicity of the running form. In this embodiment, the moving image of the skeleton motion is cut out at a point where the knee of the right foot passes the knee of the left leg, and interpolation is performed so that all have the same number of frames (for example, 100 frames) in order to match the number of cut out frames. Perform normalization. When interpolation is performed, data interpolation is performed on position information of the root joint and angle information of the root joint and the joint.

The normalized skeleton acquiring unit 151 performs normalization of the skeleton operation for a plurality of skeleton operations (here, N trials), and generates N normalized skeleton operations. Also, each normalized skeleton operation is associated with an index value representing the maximum kick acceleration as one of the index values.
In step S204, the normalization skeleton acquisition unit 151 determines whether normalization of the skeleton operation has been performed for N trials.
If normalization of the skeleton operation has not been performed for N trials, NO is determined in step S204, and the process returns to step S201.
On the other hand, when normalization of the skeleton operation is performed for N trials, YES is determined in step S204, and the process returns to the preparatory process.

[Standard skeleton acquisition process]
Next, the standard skeleton acquisition process executed in step S102 of the preliminary preparation process will be described.
FIG. 9 is a flow chart for explaining an example of the flow of the standard skeleton acquisition process in the preliminary preparation process.
When the standard skeleton acquisition processing is started, in step S301, the standard skeleton acquisition unit 152 determines whether the processing of all joints is completed. That is, the standard skeleton acquisition unit 152 determines, for each of all joints (root joints and each joint), whether or not each frame has been processed.

If all frames have been processed for each joint, YES is determined in step S301, and the process returns to the preparatory process.
On the other hand, if all the frames have not been processed for each joint, NO is determined in step S301, and the process proceeds to step S302.
In step S302, the standard skeleton acquisition unit 152 acquires joint parameters of all attempts.

In step S303, the standard skeleton acquisition unit 152 calculates the parameters of the joints of the standard skeleton. The parameters for the joints of the standard skeleton are obtained as the average value of the variables at each joint of all the extracted attempts. The average value here is calculated independently for each of the components of the X axis, the Y axis, and the Z axis.
For example, in the case of the root joint, the average of the position coordinates of all the trials in the corresponding frame is obtained, and this is used as the position coordinate of the corresponding frame in the standard skeleton operation.
Moreover, if it is angle information shown by quaternion, it will be calculated according to the following (2) Formula. That is, in the case of the average of five quaternions q1, q2, q3, q4, q5,
Qmean = exp ((1/5) × (ln (q1) + ln (q2) + ln (q3) + ln (q4) + ln (q5)) (2)
It is calculated as
Here, "exp" represents an exponential function whose base is a Napier number, and "ln" represents a natural logarithm.
Thus, by converting to logarithmic quaternion, it is possible to obtain an average value for standard skeleton operation.

In step S304, the standard skeleton acquisition unit 152 determines whether all frames have been processed.
If all frames have been processed, YES is determined in step S304, and the process returns to the preparatory process.
On the other hand, when all the frames have not been processed, it is determined as NO in step S304, and the process returns to step S302.

[Index association processing]
Next, the indicator associating process performed in step S103 of the preliminary preparation process will be described.
FIG. 10 is a flow chart for explaining an example of the flow of index association processing in the preliminary preparation processing.
When the index association processing is started, in step S401, the index association processing unit 153 ends the processing of all skeleton operations (that is, the processing for generating the standard skeleton operations for all of the normalized skeleton operations). It is determined whether the

When the processing of all the skeleton operations is completed, YES is determined in step S401, and the processing returns to the preparatory processing.
On the other hand, when the processing of all the skeleton operations is not completed, it is determined as NO in step S401, and the process proceeds to step S402.
In step S402, the index association processing unit 153 determines whether all the indices related to exercise have been processed.
If all the indices related to exercise have been processed, YES is determined in step S402, and the process returns to step S401.
On the other hand, if all the indices related to exercise have not been processed, it is determined as NO in step S402, and the process proceeds to step S403.

  In step S403, the index association processing unit 153 calculates an average front and rear body axis. The average anterior-posterior body axis is calculated by obtaining the average value of the inclination in the longitudinal direction of the body axis from the normalized skeleton motion. Here, "body axis" means a vector directed from the center of the left and right hip joints to the center of the left and right shoulder joints. In this embodiment, the average body axis vector in all frames of one normalized skeleton operation is calculated, and the inclination from the vertical direction to the running direction is the average front and rear body axes. The same is true for the standard skeleton motion, in which an average body axis vector in all frames is calculated, and an average front and back body axis is calculated. Then, the calculated average front and rear body axes are associated with each skeleton motion.

In step S404, the index association processing unit 153 calculates the maximum kick acceleration.
The maximum kick acceleration is obtained as the maximum value of the vertical component of the ground reaction force obtained by the force plate device. Then, in one cycle of operation, an index value representing the maximum kick acceleration is associated with the normalized skeleton operation of the frame corresponding to the maximum value. For the standard skeleton motion, the average of the maximum kick acceleration of the normalized skeleton motion used when generating the standard skeleton motion is taken as an index value representing the maximum kick acceleration of the standard skeleton motion.
After step S404, the process returns to step S402.

[Axis parameter determination processing]
Next, the axis parameter determination process performed in step S104 of the preparatory process will be described.
FIG. 11 is a flowchart for explaining an example of the flow of axis parameter determination processing in the preliminary preparation processing.
When the axis parameter determination processing is started, in step S501, the axis parameter determination unit 154 determines whether or not processing of all indices related to motion has been completed.
If all the indices related to exercise have been processed, YES is determined in step S501, and the process returns to the preparatory process.
If all the indices related to exercise have not been processed, it is determined as NO in step S501, and the process proceeds to step S502.

In step S502, the axis parameter determination unit 154 obtains a normalized skeleton motion whose maximum kick acceleration is the top 20% from among all normalized skeleton motions. However, this is excluded if it is included that is smaller than the maximum kick acceleration of the standard skeleton motion.
In step S503, the axis parameter determination unit 154 sets the variable Loop to 0 (Loop = 0).
In step S504, the axis parameter determination unit 154 calculates an average skeleton operation of the acquired normalized skeleton operation (that is, an average value of the upper feature skeleton operations).
In step S505, the axis parameter determination unit 154 determines whether or not the variable Loop incremented by one is less than 3 (Loop ++ <3).

If the variable Loop is not less than 3, it is determined as NO in step S505, and the process proceeds to step S508.
On the other hand, if the variable Loop is less than 3, YES is determined in step S505, and the process proceeds to step S506.
In step S506, the axis parameter determination unit 154 calculates the Euclidean distance between the selected skeleton operation and the average skeleton operation. That is, the axis parameter determination unit 154 calculates the average value of the upper feature skeleton operations as in the generation of the standard skeleton operations, and generates the upper average skeleton operations. Then, the axis parameter determination unit 154 develops the generated upper average skeleton motion in the position space. That is, the axis parameter determination unit 154 converts the variables and constants of all the frames (the position and angle information of the root joint, the bone length, and the angle information at each joint) into the root joint position and the joint position as spatial position coordinates. Do. Then, the axis parameter determination unit 154 calculates the spatial position error integrated in all frames between the generated upper average skeleton motion and each normalized skeleton motion used to generate the upper average skeleton motion. Find the (Euclidean error). The calculated Euclidean error is the Euclidean distance.

In step S507, the axis parameter determination unit 154 excludes, from the selection, the normalized skeleton operation in which the Euclidean error (Euclidean distance) is larger than the specified value D. That is, the skeleton motion whose distance is distant in position coordinates is excluded from the average skeleton on the upper side. Thereafter, the process returns to step S504, and again generates the upper average skeleton operation.
In step S508, the axis parameter determination unit 154 calculates the feature skeleton operation on the upper side. That is, the axis parameter determination unit 154 sets the generated upper average skeleton motion as a representative of the skeleton motion having a large maximum kick acceleration, and sets the generated upper average skeleton motion regarding the maximum kick acceleration. Further, the axis parameter determination unit 154 obtains the upper side maximum kick acceleration index corresponding to the upper side feature skeleton operation. The upper side maximum kick acceleration index value is an average of the maximum kick acceleration index values of the normalized skeleton operations used to generate the upper side feature skeleton operation.

In step S509, the axis parameter determination unit 154 calculates the average value (upper reference value) of the indices of the upper feature skeleton operation. That is, the axis parameter determination unit 154 calculates the average of the maximum kick acceleration from the selected upper feature skeleton operation, and sets the average as the upper reference value.
In step S510, the axis parameter determination unit 154 acquires a lower 20% normalized skeleton operation with a smaller maximum kick acceleration out of all the normalized skeleton operations. However, this excludes the one that is larger than the maximum kick acceleration of the standard skeleton.
In step S511, the axis parameter determination unit 154 sets the variable Loop to 0 (Loop = 0).
In step S512, the axis parameter determination unit 154 calculates an average skeleton operation of the acquired normalized skeleton operation (that is, an average value of lower feature skeleton operations).

In step S513, the axis parameter determination unit 154 determines whether or not the variable Loop incremented by one is less than 3 (Loop ++ <3).
If the variable Loop is not less than 3, NO is determined in step S513, and the process proceeds to step S516.
On the other hand, if the variable Loop is less than 3, YES is determined in step S513, and the process proceeds to step S514.

  In step S514, the axis parameter determination unit 154 calculates the Euclidean distance between the selected skeleton operation and the average skeleton operation. That is, the axis parameter determination unit 154 calculates the average value of the lower feature skeleton in the same manner as when generating the standard skeleton operation, and generates the lower average skeleton operation. Then, the axis parameter determination unit 154 develops the generated lower average skeleton motion in the position space. That is, the axis parameter determination unit 154 converts the variables and constants of all the frames (the position and angle information of the root joint, the bone length, and the angle information at each joint) into the root joint position and the joint position as spatial position coordinates. Do. Then, the axis parameter determination unit 154 calculates the spatial position error integrated in all frames between the generated lower average skeleton motion and each normalized skeleton motion used to generate the lower average skeleton motion. Find the (Euclidean error). The calculated Euclidean error is the Euclidean distance.

In step S515, the axis parameter determination unit 154 excludes, from the selection, the normalized skeleton operation in which the Euclidean error (Euclidean distance) is larger than the specified value D. That is, skeleton motions whose distances are distant in position coordinates are excluded from the lower average skeleton. Thereafter, the process returns to step S512, and again generates a lower average skeleton operation.
In step S516, the axis parameter determination unit 154 calculates the lower feature skeleton operation. That is, the axis parameter determination unit 154 sets the obtained lower average skeleton motion as a representative of the skeleton motion with a smaller maximum kick acceleration and a lower feature skeleton motion related to the maximum kick acceleration. Further, the axis parameter determination unit 154 obtains the lower side maximum kick acceleration index corresponding to the lower side feature skeleton operation. The lower side maximum kick acceleration index value is an average of the maximum kick acceleration index values of the normalized skeleton motions used to generate the lower side feature skeleton motions.

In step S517, the axis parameter determination unit 154 calculates the average value (lower reference value) of the indices of the lower feature skeleton operation. That is, the axis parameter determination unit 154 calculates the average of the maximum kick acceleration from the selected lower feature skeleton operation and sets the average as the lower reference value.
In step S518, the axis parameter determination unit 154 sets, as the axis parameter of the maximum kick acceleration, an index value indicating the maximum kick acceleration of the standard skeleton motion, the upper side feature skeleton motion of the maximum kick acceleration, and the upper reference value of the maximum kick acceleration; The storage unit 118 stores the lower feature skeleton operation of the maximum kick acceleration and the lower reference value of the maximum kick acceleration.
After step S518, the process returns to step S501.
Although the evaluation based on the position space is performed in steps S507 and S515, the evaluation may be performed based on the deviation of the rotation amount itself of all the joints.

Skeleton drawing processing
FIG. 12 is a flowchart illustrating an example of the flow of skeleton drawing processing executed by the terminal device 200 of FIG. 3 having the functional configuration of FIG. 5.
The skeleton drawing process is started in response to the input of activation of the skeleton drawing process via the input unit 217.
When the skeleton drawing process is started, in step S601, the drawing processing unit 253 initializes a frame variable (Frame No = 0).
In step S602, the index value acquisition unit 251 acquires an index value regarding each input motion. For example, when an exercise index is given by the slide bar in the user interface, the index value acquisition unit 251 acquires the value indicated by the slide bar.

In step S603, the synthesis skeleton generation unit 252 executes a synthesis skeleton generation process (described later). In the synthesis skeleton generation process, a synthesis skeleton of the frame number (FrameNo) is generated based on the acquired index value regarding the movement.
In step S604, the drawing processing unit 253 controls the output unit 218 so as to draw a combined skeleton. In drawing a composite skeleton, the position coordinate of the root joint in the direction of movement is fixed so that the composite skeleton does not move in the direction of movement.

In step S605, the drawing processing unit 253 determines whether the frame to be drawn is smaller than the maximum frame number (FrameNo <MaxFrameNo).
If the frame to be drawn is not smaller than the maximum frame number, NO is determined in step S605, and the process proceeds to step S607.
In step S607, the drawing processing unit 253 resets the frame number (Frame No = 0).
Thereafter, the processing proceeds to step S608.
On the other hand, if the frame to be drawn is smaller than the maximum frame number, YES is determined in step S605, and the process proceeds to step S606.

In step S606, the skeleton drawing processing unit 110 increments the frame number by 1 (Frame No ++).
In step S608, the drawing processing unit 253 determines whether to adjust the drawing timing. That is, by changing the index value relating to exercise, it becomes necessary to interpolate or delete the frame, and it is determined whether the drawing timing of the frame is adjusted.
When adjusting the drawing timing, the process of step S608 is repeated.
On the other hand, when the drawing timing is not adjusted, it is determined as NO in step S608, and the process proceeds to step S609.

In step S609, the drawing processing unit 253 determines whether an operation to end the skeleton drawing process has been performed.
If the end operation has not been performed, it is determined as NO in step S609, and the process returns to step S602.
On the other hand, if the end operation has been performed, it is determined as YES in step S609, and the skeleton drawing process ends.

FIG. 13 is a schematic view showing an example of a display screen displayed by the skeleton drawing process.
As shown in FIG. 13, on the left side of the display screen, a slide bar indicating an average front and rear body axis and maximum kick acceleration as an index related to exercise and current setting values (average front and rear body axis 5.0 [deg], maximum kick acceleration 25 [M / s2] is displayed. In addition, the synthetic skeleton SK 401 is displayed on the right side of the display screen.
In the example of the display screen shown in FIG. 13, the index value can be changed by the slide bar for the index related to movement, that is, the average front-rear body axis and the maximum kick acceleration.

[Composite skeleton generation process]
Next, the combined skeleton generation process executed in step S603 of the skeleton drawing process will be described.
FIG. 14 is a flowchart for explaining an example of the flow of synthesis skeleton generation processing in the skeleton drawing processing.
In the present embodiment, the description will be made by taking as an example the case where the index value representing the maximum kick acceleration is changed.
When the synthesis skeleton generation process is started, in step S701, the synthesis skeleton generation unit 252 determines whether or not processing of indices related to all motions is completed.

When the processing of the indices related to all the movements is completed, YES is determined in step S701, and the process proceeds to step S709.
On the other hand, when the processing of the indices related to all the exercises is not completed, it is determined as NO in step S701, and the process proceeds to step S702.
In step S702, the combined skeleton generation unit 252 sets the index value for each motion and the axis parameter of the maximum kick acceleration (index value representing the maximum kick acceleration of the standard skeleton operation acquired in step S518 of the axis parameter determination process, The upper side feature skeleton operation of the maximum kick acceleration, the upper reference value of the maximum kick acceleration, the lower side feature skeleton operation of the maximum kick acceleration, and the lower reference value of the maximum kick acceleration) are acquired.

In step S703, the synthetic skeleton generation unit 252 inputs the standard skeleton operation (time = frame number FrameNo) to the variable Sbase, the index value regarding the motion of the standard skeleton operation to the variable Dbase (index value representing the maximum kick acceleration), and the variable Din. Set the index value of the maximum kick acceleration.
For example, Sbase = standard skeleton operation (frame of frame number FrameNo), Dbase = index value of maximum kick acceleration of standard skeleton operation “25”, Din = index value of maximum kick acceleration input “40”.
In step S704, the combined skeleton generation unit 252 determines whether or not the index value of the maximum kick acceleration input is larger than the index value related to the motion of the standard skeleton motion (Dbase <Din).
If the index value of the maximum kick acceleration input from the index value related to the motion of the standard skeleton motion is large, YES is determined in step S704, and the process proceeds to step S705.

In step S 705, the combined skeleton generation unit 252 sets the variable Ss to the upper feature skeleton operation of the axis parameter (time = frame number FrameNo) and sets the variable Ds to the upper reference value of the maximum kick acceleration of the axis parameter.
For example, Ss = axis parameter upper side feature skeleton operation (frame of frame number FrameNo), Ds = axis parameter upper reference value “35” is set.
Thereafter, the processing proceeds to step S 707.
On the other hand, when the index value of the maximum kick acceleration input from the index value related to the motion of the standard skeleton motion is small, NO is determined in step S704, and the process proceeds to step S706.

In step S706, the combined skeleton generation unit 252 sets the variable Ss to the lower feature skeleton operation of the axis parameter (time = frame number FrameNo), and sets the variable Ds to a lower reference value of the maximum kick acceleration of the axis parameter.
For example, Ss = axis parameter upper side feature skeleton operation (frame of frame number FrameNo), Ds = axis parameter lower reference value “10” is set.
In step S 707, the synthesis skeleton generation unit 252 calculates a ratio r = (Din−Dbase) / (Ds−Dbase).

Specifically, assuming that Dbase is 1 with Dbase as a reference, a ratio at which the input index value Din is located is determined as r. That is, r = 0 is the input index value Din = Dbase, and when r = 1, Din = Ds, when r> 1, Din> Ds, and when 0 <r <1, Dbase <Din < It becomes Ds.
In step S 708, the synthesis skeleton generation unit 252 obtains a feature skeleton (a skeleton after change) corresponding to the ratio r. The position coordinates of the root joint are calculated independently in each of the X, Y, and Z axes, for example, in the X coordinate, according to the following equation (3).
X ′ = x_base + r × (x_s−x_base) (3)

Here, “X ′” is the X coordinate value of the root joint of the feature skeleton. Similarly, Y coordinate values and Z coordinate values are also obtained. “X_s” indicates the X coordinate value of the root joint in the frame of the upper feature skeleton operation indicated by Ss, and “x_base” indicates the X coordinate value of the root joint in the frame of the standard skeleton operation indicated by Sbase There is.
Also, with regard to angle information indicated by quaternion, similarly, it is possible to obtain a value after change by the following interpolation formula.
q '= q_base × (inv (q_base) × q_s) ^r
Here, “q_base” indicates a quaternion of a standard skeleton, “q_s” indicates a quaternion of the upper side feature skeleton operation Ss, and “inv ()” indicates an inverse function.

The synthesis skeleton generation unit 252 performs calculation for all angle information to obtain a feature skeleton. For bones, use the bone of standard skeleton.
Thereafter, the process returns to step S701.
In step S709, the synthesis skeleton generation unit 252 takes the average of the feature skeletons of the indices related to all the motions, and sets it as a synthesis skeleton.
Thereafter, the process returns to the skeleton drawing process.

FIG. 15 is a schematic view showing a time-series skeleton space.
As shown in FIG. 15, the variables in the skeleton space are constituted by dimensions including position coordinates of the root joint, angle information of the root joint and the joint.
Here, in order to simplify the explanation, it is assumed that the angle information is in two dimensions of θ1 and θ2.
In a desired frame, it is assumed that the standard skeleton operation is SK301, the upper feature skeleton operation is SK302, and the lower feature skeleton operation is SK303. The modified feature skeleton SK 304 is set on the axis 312 from the standard skeleton operation SK 301 to the upper feature skeleton operation SK 302. The feature skeleton SK304 is different for each time, that is, when the frame is different, the variable of each skeleton operation is also different, so the feature skeleton SK304 is also different.

FIG. 16 is a schematic view showing a skeleton space of a frame (frame number = k) different from that of FIG.
An axis 312 is an interpolation axis when the upper feature skeleton operation is viewed from the standard skeleton operation SK301. An axis 313 is an interpolation axis when the lower feature skeleton action SK303 is viewed from the standard skeleton action SK301. That is, the feature skeleton SK 304 is set on the axis 312 or the axis 313 when the ratio r is variously changed for the index related to a specific motion.

FIG. 17 is a schematic view showing a combined skeleton in the skeleton space.
Note that FIG. 17 (a) shows a state in which the standard skeleton motion SK301 is set in the skeleton space, and FIG. 17 (b) shows feature skeletons SK324 and SK334 corresponding to index values for different movements. It is done. Further, FIG. 17C shows a synthetic skeleton SK340.
The axes 322 and 323 and the axes 332 and 333 shown in FIG. 17 (a) are interpolation axes according to the indices relating to different motions, and when the ratio r in the indices relating to these motions is input, FIG. Feature skeletons SK324 and SK334 are set.

Then, as shown in FIG. 17C, by acquiring the average value of the coordinates of the feature skeletons SK324 and SK334, the angle information of the composite skeleton SK340 is obtained, and the composite skeleton SK340 is set.
In the above description, two interpolation directions of the upper feature skeleton operation and the lower feature skeleton operation are set for an index related to one motion, and linear axes are set by the one skeleton operation. However, even when a plurality of skeleton operations are set and axes are set not by straight lines but by broken lines, synthetic skeletons can be set by the same algorithm. By setting an axis with a larger number of folds, it is possible to appropriately cope with more complicated operations.

[Specific form of visualization]
Next, a specific form in which the drawing processing unit 253 of the terminal device 200 draws a composite skeleton will be described.

[First arrow drawing process]
FIG. 18 is a flowchart for explaining the flow of a first arrow drawing process performed by the terminal device 200 of FIG. 3 having the functional configuration of FIG. 5.
The first arrow drawing process is a process of generating an arrow displayed to visualize the change state of the skeleton.
In the first arrow drawing process, in a state in which the skeleton is repeatedly drawn (see FIG. 19), for example, an operation of clicking the label of the maximum kick acceleration in the user interface and selecting the index value of the maximum kick acceleration is performed It is started by. At this time, the first arrow drawing process is called with the frame number to be drawn (Frame No) and the index value of the maximum kick acceleration as arguments.

When the first arrow drawing processing is started, in step S801, the drawing processing unit 253 sets the index at the time (Frame No) of the selected index related to the exercise to a maximum or a minimum. Calculates the space orientation of the skeleton (the spatial position coordinates of the composite skeleton).
In step S802, the drawing processing unit 253 sets a joint of the right ankle, the left ankle, the chest, the right wrist, and the left wrist as a search joint in the calculated skeleton space posture (space position coordinates of the synthesized skeleton).

In step S803, the drawing processing unit 253 obtains differences (vectors) at each of the search joints with respect to the skeleton space posture (the space position coordinates of the combined skeleton) at the two maximum and minimum index values.
In step S804, the drawing processing unit 253 selects two joints having a large difference amount.
In step S805, the drawing processing unit 253 generates an arrow in which the difference amount is “the length of the arrow” and the direction of the vector is the “direction of the arrow”. Specifically, for example, when the largest joint is the joint of the left ankle and the second largest joint is the joint of the right ankle, the drawing processing unit 253 sets the difference amount to the length (or size) of the arrow. Then, draw an arrow at each joint with the difference direction as the direction of the arrow.

In step S806, the drawing processing unit 253 changes the viewpoint about the vertical direction from the ground to the viewpoint where the calculated difference between the search joints is clear. The viewpoint direction is determined such that, for example, the direction of the arrow at the joint with the largest difference amount is parallel to the drawing surface (that is, viewed from the front).
In step S 807, the drawing processing unit 253 controls the output unit 218 so as to draw the generated arrow on the corresponding joint.
Thereafter, the first arrow drawing process ends.

FIG. 20 is a schematic view showing an example of a display screen in which the viewpoint direction is changed.
In FIG. 20, with respect to the display screen example of FIG. 19, the composite skeleton is changed to a side view, and the arrow A 441 at the joint of the left ankle and the arrow A 442 at the joint of the right ankle are in front view ing. In FIG. 20, the slide bar of the index related to the selected exercise (maximum kick acceleration) and the area of the current setting value are surrounded by a rectangular cursor.
Moreover, since the index value regarding exercise changes for each frame, the display screen example as shown in FIG. 20 changes with time.

FIG. 21 is a schematic view showing an example of a display screen at a time different from that in FIG.
In the display screen example shown in FIG. 21, arrows A442 and A443 are displayed on the joint of the right ankle and the joint of the chest.
In such a skeleton state, the position change of the joint of the left ankle is small even if the index value of the maximum kick acceleration changes, so in the frame shown in FIG. Arrows do not appear on neck joints.

[2nd arrow drawing process]
FIG. 22 is a flowchart for explaining the flow of the second arrow drawing process executed by the terminal device 200 of FIG. 3 having the functional configuration of FIG. 5.
The second arrow drawing process is a process of drawing an arrow indicating the direction of change of the portion of the skeleton when the index is further changed when an operation of selecting an index value related to exercise is performed.
In the second arrow drawing process, in a state where the skeleton is repeatedly drawn (see FIG. 23), for example, an operation of clicking the label of the maximum kick acceleration in the user interface and selecting the index value of the maximum kick acceleration is performed It is started by.

When the second arrow drawing process is started, in step S901, the drawing processing unit 253 calculates the space position coordinate P1 of the combined skeleton at the time (FrameNo) after the index is changed.
In step S902, the drawing processing unit 253 calculates the space position coordinates P2 of the combined skeleton when the index value further changes in the same direction.
In step S 903, the drawing processing unit 253 sets the joints of the right ankle, the left ankle, the chest, the right wrist, and the left wrist as search joints.

In step S904, the drawing processing unit 253 obtains differences (vectors) at each of the search joints based on the skeleton space position coordinate P1 with the two skeleton space position coordinates. That is, the drawing processing unit 253 calculates the difference of the space position coordinate P2 as viewed from the space position coordinate P1 at the search joint.
In step S905, the drawing processing unit 253 selects the joint with the largest difference amount.

In step S 906, the drawing processing unit 253 controls the output unit 218 so as to draw an arrow A 443 in which the difference amount is “size of arrow” and the direction of vector is “direction of arrow” on the corresponding joint. .
Thereafter, the second arrow drawing process ends.

FIG. 24 is a schematic view showing an example of a display screen on which an arrow A 443 indicating the direction of change of the portion of the skeleton is displayed.
In FIG. 24, the skeleton SK441 before the change of the index value in FIG. 23 is drawn by a broken line, and the skeleton SK442 after the change of the index value is drawn by a solid line. Furthermore, in the vicinity of the joint of the right ankle of the skeleton SK 442, an arrow A 443 indicating a direction in which the right ankle changes when the index value is further changed is displayed.
In the example of the display screen shown in FIG. 24, the skeleton SK 441 before the change of the index value may not be drawn. The arrow 443 and the pre-change skeleton SK441 of the index value disappear after a preset time after the operation of changing the index value.

[Response index notification process]
FIG. 25 is a flowchart for explaining the flow of the correspondence index notification process performed by the terminal device 200 of FIG. 3 having the functional configuration of FIG. 5.

The correspondence index notification process is a process of notifying the user of an index that changes corresponding to the operation part by operating a skeleton that is stationary.
The corresponding index notification process is started in response to the skeleton manual operation mode (a mode for receiving the user's operation on the skeleton) being selected.

When the corresponding index notification process is started, in step S1001, the drawing processing unit 253 stops automatic updating of the frame of the skeleton operation, and draws the skeleton in a still state with the frame number (FrameNo) selected by the user. .
In step S1002, the drawing processing unit 253 draws movable joints as large circles (see FIG. 26).
In step S1003, the drawing processing unit 253 performs interrupt setting for the end operation and interrupt setting for the movement operation of the joint. That is, it is set in a state to receive an interrupt signal indicating the end operation and an interrupt signal indicating the movement operation of the joint.
After step S1003, the drawing processing unit 253 transitions to the standby mode.

FIG. 26 is a schematic view showing an example of a display screen of a skeleton in which movable joints are drawn larger than other joints.
In the display screen example shown in FIG. 26, when the user moves the movable joint on the screen (for example, when the joint J 462 is moved in the direction of the arrow A463 in FIG. 26), the moving operation of the joint is shown. An interrupt signal is generated and joint movement interrupt processing is performed.

FIG. 27 is a flow chart for explaining the flow of joint interrupt movement processing which is executed as interrupt processing in the correspondence index notification processing.
When the joint interrupt movement process is started, in step S1011, the drawing processing unit 253 generates a new skeleton N1 corresponding to the position of the moved joint. Note that even movable joints can not be moved freely, and movement operations are permitted under the restriction that the bones are rigid.
In step S1012, the drawing processing unit 253 draws a new skeleton N1.
In step S1013, the drawing processing unit 253 draws the moved joint part for the next operation.

In step S1014, the drawing processing unit 253 searches for a frame of the skeleton closest to the position of the moved joint among the frames having the frame number of FrameNo in the set of normalized skeleton operations acquired in advance. At this time, the motion parameters may be compared and searched, or may be converted to spatial position coordinates and then compared and searched.
In step S1015, the drawing processing unit 253 moves the slide bar of the display screen in association with the index value of the retrieved normalized skeleton operation.
In FIG. 26, when the joint J462 of the right ankle is moved in the direction of the arrow A463, an index value corresponding to the skeleton after the movement is displayed.
After step S1015, the drawing processing unit 253 shifts to the standby mode again.
In addition, when the end operation is performed in the standby mode in the correspondence index notification process and the joint interrupt movement process, an interrupt signal indicating the end operation is generated, and the end interrupt process is executed.

FIG. 28 is a flowchart for explaining the flow of an end interrupt process which is executed as an interrupt process in the correspondence index notification process and the joint interrupt movement process.
When the end interrupt process is started, preparation for the end of the corresponding index notification process (storage of necessary parameters, reset of setting, etc.) is executed in step S1021.
After step S1021, the end interrupt process ends. In addition, the corresponding index notification process also ends.

According to the above embodiment, for example, in the image generation apparatus 100, standard skeletons configured based on motion capture data of a plurality of individuals acquired as a sample are stored, and in the terminal device 200, in the terminal device 200, User-specific behavior can be entered as an index value.
In this way, it is possible to generate a synthetic skeleton motion that reflects a specific individual's motion from a standard skeleton motion that integrates the motions of a plurality of individuals.
In this case, it is possible to realize a form in which the service provider provides data of the standard skeleton operation, and each user who uses the service generates a synthetic skeleton operation in which the user's own movement is reflected.

As described above, the image generation system 1 of the present embodiment includes the image generation apparatus 100 and the terminal device 200.
The image generation device 100 includes an animation acquisition unit 150, and the animation acquisition unit 150 acquires a plurality of animation data.
The terminal device 200 includes an index value acquisition unit 251, a combined skeleton generation unit 252, and a drawing processing unit 253.
The index value acquisition unit 251 inputs the value of the index related to the motion of the animation data, and the composite skeleton generation unit 252 inputs the index value acquisition unit 251 using the plurality of animation data acquired by the animation acquisition unit 150. Animation data (data of synthetic skeleton motion) corresponding to the value of the index is generated.
Therefore, it becomes possible to change the animation data (skeleton data) obtained by measuring the motion into a desired motion.

In addition, the index value acquisition unit 251 inputs values of a plurality of indexes related to the motion of the animation data, and the combined skeleton generation unit 252 generates animation data according to the values of the plurality of indexes input by the index value acquisition unit 251. Generate
Therefore, in the animation data, it is possible to associate a plurality of motion-related indicators and change each of the indicators, so that it is possible to realize more complex motion changes.

  The image generation apparatus 100 further includes a standard skeleton acquisition unit 152, and the standard skeleton acquisition unit 152 selects standard animation data associated with an index related to motion from the plurality of animation data acquired by the animation acquisition unit 150. Generate The synthetic skeleton generation unit 252 generates animation data according to the value of the index input by the index value acquisition unit 251 from standard animation data.

In addition, a reference value of an index related to motion is set in the standard animation data, and the combined skeleton generation unit 252 acquires an index value acquisition unit based on the reference value (for example, upper or lower reference value) in the standard animation data. An animation data is generated according to the value of the index input at 251.
Therefore, based on the reference values set based on the plurality of animation data obtained by measuring the motion, it is possible to generate animation data in which the index related to movement is changed. , Based on the measured motion, it is possible to realize more appropriate change in motion.

The terminal device 200 further includes a drawing processing unit 253 that causes the output unit 218 to display an animation based on standard animation data and a user interface for inputting an index value related to the exercise set in the animation data, The value acquisition unit 251 inputs a value of an index related to exercise by operating the user interface.
Therefore, it is possible to input the value of the index related to the exercise in a visually understandable form.

When the index value acquiring unit 251 receives the value of the index relating to the motion, the drawing processing unit 253 corresponds an animation based on the animation data generated by the composite skeleton generating unit 252 to the value of the input index. The part is identified and displayed on the output unit 218.
Therefore, it is possible to clearly display the changing part of the animation due to the change of the index related to the movement.

In addition, the drawing processing unit 253 identifies a portion of the animation that changes in accordance with the value of the index related to the motion input by the index value acquisition unit 251 and causes the output unit 218 to display the portion.
Therefore, it is possible to clearly display the part that affects when the index related to exercise is changed.

In addition, the drawing processing unit 253 causes the output unit 218 to display an index related to the motion corresponding to the animation data after the input of the value related to the motion by the index value acquisition unit 251.
Therefore, since the index related to the motion can be changed in response to the direct operation on the animation, the operability in the case of changing the animation data can be enhanced.

[Modification]
In the above-mentioned embodiment, the input of the index value regarding exercise was performed by the slide bar in the display screen generated by the application.
On the other hand, skeleton values may be displayed by inputting an index value relating to exercise by real-time exercise.
Specifically, the user wears an acceleration sensor on his / her shoes, runs while wearing a display device (for example, a portable terminal or the like) on his / her arm, inputs an index value according to the user's actual movement, and performs skeleton operation. Can be generated.
FIG. 29 is a flow chart for explaining the flow of real-time display processing executed by the terminal device 200 of FIG. 3 having the functional configuration of FIG.
The real time display process is a process of displaying a skeleton operation by inputting an index value regarding exercise in real time.
The real-time display process is started in response to an input of activation of the real-time display process in a state where the acceleration sensor is attached to the shoe of the user and the terminal device 200 is attached to the arm.

When the real-time display process is started, in step S1031, the terminal device 200 acquires an index related to motion from the acceleration sensor. For example, speed information (information on the speed of running) is acquired as an index on exercise. The speed information can be acquired based on a preset stride by acquiring the pitch of running from an acceleration sensor attached to a shoe. The speed information acquired by the acceleration sensor is transmitted to the terminal device 200 by wireless communication.
In step S1032, the terminal device 200 draws on the display screen a synthesized skeleton according to the speed as an index related to exercise.
In step S1033, the terminal device 200 executes a weight for time adjustment in real-time processing.
Thereafter, real time display processing is repeatedly executed.

In addition, also when displaying a skeleton operation | movement by a real-time display process, it becomes a premise that the setting of the axis | shaft parameter by prior preparation processing is performed by making the parameter | index utilized for the display of a skeleton operation into object.
In addition to acceleration sensors attached to shoes, a plurality of types of sensors can be used to acquire indices relating to a plurality of types of exercise, and composite skeletons corresponding to the indices relating to a plurality of types of exercise can be displayed in real time. Also in this case, if the data is generated by the preparation process, it is possible to generate the synthesis skeleton operation at low calculation cost.

  The present invention is not limited to the above-described embodiment, and modifications, improvements, and the like in the range in which the object of the present invention can be achieved are included in the present invention.

In the above-described embodiment, the index value set for the standard skeleton operation is obtained from the normalized skeleton operation of the population that generates the standard skeleton operation. On the other hand, the index value set to the standard skeleton operation can be obtained from a set of skeleton operations different from the mother set generating the standard skeleton operation.
As a result, even if the data of normalized skeleton motion of the mother set which generated the standard skeleton motion can not be used, it is possible to set the index value obtained from the set of other skeleton motions to the standard skeleton. It is possible to realize the present invention more flexibly.
Also, in the above-described embodiment, the set of skeleton operations used to generate the standard skeleton operations may be configured from a plurality of motion capture data of a specific individual or may be configured from motion capture data of a plurality of individuals. Both are possible.

Further, in the above-described embodiment, the image generation apparatus 100 to which the present invention is applied is configured by a server, and the terminal device 200 is described as an example configured by a smartphone. However, the present invention is not particularly limited thereto.
For example, the present invention can be realized by an electronic device generally having an information processing function. Specifically, for example, the image generation device 100 and the terminal device 200 in the present invention are configured by a laptop personal computer, a printer, a television receiver, a video camera, a portable navigation device, a portable telephone, a portable game machine, etc. It is possible.

Further, in the above-described embodiment, the case where the image generation apparatus 100 and the terminal apparatus 200 are configured as separate apparatuses has been described as an example, but the present invention is not particularly limited thereto.
For example, the image generation apparatus 100 can be configured by a PC or the like, and the image generation apparatus 100 can be provided with the function of the terminal device 200 in the above-described embodiment.

The series of processes described above can be performed by hardware or software.
In other words, the functional configurations of FIG. 4 and FIG. 6 are merely illustrative and not particularly limited. That is, it is sufficient if the image generation apparatus 100 and the terminal device 200 have a function capable of executing the above-described series of processes as a whole, and what function blocks are used to realize this function is described in particular in FIG. It is not limited to the example of FIG.
Further, one functional block may be configured by hardware alone, may be configured by software alone, or may be configured by a combination of them.

When the series of processes are executed by software, a program that configures the software is installed on a computer or the like from a network or a recording medium.
The computer may be a computer incorporated in dedicated hardware. The computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose personal computer.

  A recording medium including such a program is not only constituted by the removable medium 131 of FIG. 2 and the removable medium 231 of FIG. 3 distributed separately from the apparatus main body to provide the program to the user, It is configured by a recording medium or the like provided to the user in a state of being incorporated in advance. The removable media 131 and 231 are formed of, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. The optical disc is constituted of, for example, a CD-ROM (Compact Disk-Read Only Memory), a DVD (Digital Versatile Disk) or the like. The magneto-optical disk is configured by an MD (Mini-Disk) or the like. The recording medium provided to the user in a state of being incorporated in the apparatus main body is, for example, the ROM 112 of FIG. 2 and the ROM 212 of FIG. 3 in which the program is recorded, the storage unit 118 of FIG. A hard disk etc. is included in the unit 219.

  In the present specification, in the step of describing the program to be recorded on the recording medium, the processing performed chronologically along the order is, of course, parallel or individually not necessarily necessarily chronologically processing. It also includes the processing to be performed.

  Also, in the present specification, the term "system" is intended to mean an overall device composed of a plurality of devices, a plurality of means, and the like.

  While some embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take other various embodiments, and furthermore, various changes such as omissions and substitutions can be made without departing from the scope of the present invention. These embodiments and modifications thereof are included in the scope and the gist of the invention described in the present specification, etc., and are included in the invention described in the claims and the equivalent scope thereof.

The invention described in the claims at the beginning of the application of the present application is appended below.
[Supplementary Note 1]
Data acquisition means for acquiring a plurality of animation data;
An input unit for inputting a value of an index related to motion of animation data;
A first generation unit configured to generate animation data according to the value of the index input by the input unit using the plurality of animation data acquired by the data acquisition unit;
An image generation apparatus comprising:
[Supplementary Note 2]
The input means inputs values of a plurality of indices related to motion of animation data;
The image generation device according to claim 1, wherein the first generation means generates animation data in accordance with values of a plurality of indices inputted by the input means.
[Supplementary Note 3]
The system further comprises second generation means for generating standard animation data associated with an index related to movement from the plurality of animation data acquired by the data acquisition means.
The image generation apparatus according to claim 1, wherein said first generation means generates animation data according to the value of the index input by said input means from said standard animation data.
[Supplementary Note 4]
In the standard animation data, a reference value of an indicator related to the movement is set,
The image according to appendix 3, wherein the first generation means generates animation data according to the value of the index input by the input means based on the reference value in the standard animation data. Generator.
[Supplementary Note 5]
The apparatus further comprises display control means for causing display means to display an animation based on the standard animation data and a user interface for inputting the value of the index related to the motion set in the animation data.
The image generating apparatus according to claim 3 or 4, wherein the input unit inputs the value of the index related to the exercise by operating the user interface.
[Supplementary Note 6]
The display control means, when the value of the index related to the motion is input by the input means, corresponds an animation based on animation data generated by the first generation means to the value of the input index The image generating apparatus according to claim 5, wherein the part is identified and displayed on the display means.
[Supplementary Note 7]
The display control means identifies the portion of the animation that changes in accordance with the value of the index relating to the motion input by the input means, and causes the display means to display the portion. Image generation system.
[Supplementary Note 8]
The image according to any one of Appendices 5 to 7, wherein the display control means causes the display means to display an index corresponding to the animation data after the input of the value relating to the motion by the input means. Generator.
[Supplementary Note 9]
A data acquisition step for acquiring a plurality of animation data;
An input step of inputting a value of an index related to motion of the animation data;
A generation step of generating animation data according to the value of the index input in the input step using the plurality of animation data acquired in the data acquisition step;
A method of generating an image, comprising:
[Supplementary Note 10]
Computer,
Data acquisition means for acquiring multiple animation data,
Input means for inputting a value of an index related to motion of animation data,
Generation means for generating animation data according to the value of the index input by the input means using the plurality of animation data obtained by the data acquisition means;
A program characterized by acting as

  DESCRIPTION OF SYMBOLS 1 ... Image generation system, 100 ... Image generation apparatus, 111, 211 ... CPU, 112, 212 ... ROM, 113, 213 ... RAM, 114, 214 ... Bus, 115, 215 ... input / output interface, 216 ... imaging unit, 116, 217 ... input unit, 117, 218 ... output unit, 118, 219 ... storage unit, 119, 220 ... communication unit , 120, 221: drive, 131, 231: removable media, 150: animation acquisition unit, 151: normalized skeleton acquisition unit, 152: standard skeleton acquisition unit, 153: index Association processing unit 154: axis parameter determination unit 251: index value acquisition unit 252: combined skeleton generation unit 253: drawing processing unit

Claims (15)

Data acquisition means for acquiring a plurality of animation data;
An input unit for inputting an arbitrary value of an index related to at least one of the strength of kicking of the foot and the inclination of the body axis when running animation data;
First generation means for generating animation data according to an arbitrary value of the index inputted by the input means, using a plurality of animation data obtained by the data acquisition means;
An image generation apparatus comprising: Providing display control means for identifying a portion of animation based on animation data generated by the first generation means, which changes in accordance with an arbitrary value of the index inputted by the input means, and displaying the portion on the display means The image generation apparatus according to claim 1, characterized in that:   The display control means corresponds an animation based on animation data generated by the first generation means to an arbitrary value of the input index when an arbitrary value of the index is input by the input means. The image generating apparatus according to claim 2, wherein an arrow is displayed on a portion to be displayed on the display unit.   4. The image generating apparatus according to claim 3, wherein the display control means represents the amount of change after the arbitrary value of the index is input by the length or size of the arrow. .   The image generation apparatus according to claim 3 or 4, wherein the display control means indicates a direction of change after the arbitrary value of the index is input in the direction of the arrow. The display control means is configured to change a portion of an animation based on animation data generated by the first generation means, which changes in accordance with an arbitrary value of the index input by the input means, before and after changing different lines. The image generation apparatus according to claim 2, wherein the display means is displayed by a seed.   The display control means is characterized by changing the viewpoint to a point of view where a portion of the animation that changes in accordance with an arbitrary value of the index input by the input means becomes clear and causing the display means to display it. The image generation apparatus according to any one of claims 2 to 6. The input means inputs an arbitrary value of an index related to the kicking strength of the foot and the inclination of the body axis when the animation data is run,
The image generation according to any one of claims 2 to 7, wherein the first generation means generates animation data in accordance with arbitrary values of two indices inputted by the input means. apparatus. A second method of generating standard animation data in which an index related to at least one of the strength of kicking of a foot and the inclination of the body axis is associated from a plurality of animation data acquired by the data acquisition means. Further comprising generation means,
The first generation means generates animation data according to an arbitrary value of the index inputted by the input means from the standard animation data, according to any one of claims 2 to 8. Image generation apparatus as described. In the standard animation data, a reference value of an index related to at least one of the strength of kicking of the foot and the inclination of the body axis at the time of running is set,
10. The apparatus according to claim 9, wherein said first generation means generates animation data according to an arbitrary value of the index inputted by said input means based on said reference value in said standard animation data. Image generation apparatus as described. The display control means is an animation based on the standard animation data, and any value of an index relating to at least one of the strength of kicking of a foot and the inclination of a body axis set in the animation data. Display a user interface for input on the display means,
10. The apparatus according to claim 9, wherein the input unit inputs an arbitrary value of an index related to at least one of the strength of kicking of the foot and the inclination of the body axis by operating the user interface. Or the image generation apparatus as described in 10.   The display control means displays, as display means, an index corresponding to the animation data after input of an arbitrary value of an index related to at least one of the strength of kicking of a foot and the inclination of the body axis when running by the input means. The image generation apparatus according to any one of claims 2 to 11, wherein the image is displayed.   The display control means is configured to input one of arbitrary values of a plurality of indices relating to at least one of the strength of kicking of the foot and the inclination of the body axis by the input means. 9. The image generation apparatus according to claim 8, wherein an animation based on animation data generated by the first generation means is displayed on a display means by identifying a portion corresponding to the input index. A data acquisition step for acquiring a plurality of animation data;
An input step of inputting an arbitrary value of an index related to at least one of the strength of kicking of the foot and the inclination of the body axis when running animation data;
A generation step of generating animation data according to the value of the index input in the input step using the plurality of animation data acquired in the data acquisition step;
A method of generating an image, comprising: Computer,
Data acquisition means for acquiring multiple animation data,
An input means for inputting an arbitrary value of an indicator related to at least one of the strength of kicking of the foot and the inclination of the body axis when running animation data;
Generation means for generating animation data according to the value of the index input by the input means using the plurality of animation data obtained by the data acquisition means;
A program characterized by acting as

Images