KR20200021702A - Apparatus and method for motion retargeting - Google Patents

Abstract

According to one embodiment of the present invention, a motion retargeting method performed by a motion retargeting apparatus comprises the motion of: acquiring source motion feature values according to source motion data; combining target information with the acquired source motion feature values to generate retargeting feature values; acquiring retargeted motion data by using the generated retargeting feature values; and updating the acquired retargeted motion data according to a loss function.

Description

Motion Retargeting Device and Motion Retargeting Method {APPARATUS AND METHOD FOR MOTION RETARGETING}

The following embodiments relate to a motion retargeting apparatus and a motion retargeting method, and more particularly, to obtain retargeted motion data using retargeting feature values, and to obtain the retargeted motion data according to a loss function. The present invention relates to a motion retargeting apparatus and a motion retargeting method for updating the data.

Character motion in traditional 3D computer animation was created using the animator's keyframing technique, which is still one of the most representative ways of producing character animation in the film and animation industries. To produce these keyframing animations, animators work mostly with sophisticated character rigs.

Character rigs are generally intuitive and familiar controllers designed to allow animators to more conveniently manipulate joint motion and natural body deformation for animation production. Animators can use these controllers to adjust the character's joints and deformers' parameters, making them useful for creating new keyframing animations or for modifying existing animations.

The keyframing method relies heavily on the animator's capabilities and has the disadvantage of being time-consuming and expensive, resulting in motion capture and retargeting techniques that make getting realistic motion easier and faster.

Motion capture and retargeting technology is now very important in real-world game, film and animation production, along with traditional keyframing techniques, thanks to many inventions and technology developments.

Machine learning is an area of artificial intelligence that evolves from the study of pattern recognition and computer learning theory, and develops algorithms and technologies that enable computers to learn.

Machine learning is a technology for researching and building systems and algorithms for learning based on empirical data, making predictions and improving their own performance. Machine learning algorithms do not perform strictly defined static program instructions, but rather build a specific model to make predictions or decisions based on input data.

The key to machine learning is in representation and generalization. Representation is the evaluation of data, and generalization is the processing of data that is not yet known. This is also the field of computational learning theory.

Deep learning is defined as a set of machine learning algorithms that attempt to achieve a high level of abstraction through a combination of several nonlinear transformations. It can be said to be a field.

Many studies have been conducted to express any data when it is in a form that can be understood by a computer and apply it to learning. Learning techniques are applied to the fields of computer vision, speech recognition, natural language processing, speech / signal processing, etc. to show the latest results.

Republic of Korea Patent Publication No. 10-1501405 (2015.03.04.registration)

According to an embodiment of the present invention, the motion retargeting device obtains source motion feature values according to the source motion data, and the motion retargeting device applies the source motion feature values to the obtained source motion feature values. Combining target information to generate retargeting features values, motion retargeting device to obtain retargeted motion data using the generated retargeting features values, and motion The retargeting device includes updating the obtained retargeted motion data according to a loss function.

In addition, the updating of the acquired retargeted motion data may update the retargeted motion data through a kinematic optimization process until the loss function value becomes minimum. have.

In addition, the kinematic optimization process may use a back-propagation algorithm to adjust the retargeting feature values.

In addition, the source motion data and the retargeted motion data may be animation data.

In addition, the source motion data and the retargeted motion data may include joint position information.

In addition, the source motion data may include bone information.

In addition, the target information may include length ratio information of bones.

According to another embodiment of the present invention, in a motion retargeting apparatus, the motion retargeting apparatus includes a processor, wherein the processor acquires source motion feature values according to source motion data, and Retargeting features are generated by combining the target information with the obtained source motion feature values, and retargeting motion data is generated using the retargeting features. Obtain and update the acquired retargeted motion data according to a loss function.

In addition, updating the obtained retargeted motion data may update the retargeted motion data through a kinematic optimization process until the loss function value becomes minimum. .

In addition, the kinematic optimization process may use a back-propagation algorithm to adjust the retargeting feature values.

In addition, the source motion data and the retargeted motion data may be animation data.

In addition, the source motion data and the retargeted motion data may include joint position information.

In addition, the target information may include length ratio information of bones.

1 is a view showing the operation of the motion retargeting apparatus according to an embodiment of the present invention.
2 is a view showing the configuration of a motion retargeting apparatus according to an embodiment of the present invention.
3 is a flowchart illustrating a motion retargeting method according to an embodiment of the present invention.
4 is a diagram illustrating retargeted motion data in which an error of source motion data is automatically corrected according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating motion data retargeted to various types and sizes according to an embodiment of the present invention.
6 is a block diagram of an exemplary computer system for implementing one embodiment of the present invention.

Specific structural or functional descriptions of the embodiments according to the inventive concept disclosed herein are provided for the purpose of describing the embodiments according to the inventive concept only. It may be embodied in various forms and is not limited to the embodiments described herein.

Embodiments according to the inventive concept may be variously modified and have various forms, so embodiments are illustrated in the drawings and described in detail herein. However, this is not intended to limit the embodiments in accordance with the concept of the invention to the specific forms disclosed, and includes all changes, equivalents, or substitutes included in the spirit and scope of the present invention.

Terms such as first or second may be used to describe various components, but the components should not be limited by the terms. The terms are only for the purpose of distinguishing one component from another, for example, without departing from the scope of the rights according to the inventive concept, the first component may be called a second component and similarly The second component may also be referred to as the first component.

When a component is said to be "connected" or "connected" to another component, it may be directly connected to or connected to that other component, but it may be understood that another component may be present in the middle. Should be. On the other hand, when a component is said to be "directly connected" or "directly connected" to another component, it should be understood that there is no other component in between. Other expressions describing the relationship between components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring", should be interpreted as well.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. Singular expressions include plural expressions unless the context clearly indicates otherwise.

As used herein, the terms "comprise" or "having" are intended to indicate that the described feature, number, step, operation, component, part, or combination thereof exists, but one or more other features or numbers are present. It should be understood that it does not exclude in advance the possibility of the presence or addition of steps, actions, components, parts or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art.

Terms such as those defined in the commonly used dictionaries should be construed as having meanings consistent with the meanings in the context of the related art, and are not construed in ideal or excessively formal meanings unless expressly defined herein. Do not.

In the following description, the same identification symbol means the same configuration, and unnecessary descriptions and redundant descriptions will be omitted.

In the embodiment of the present invention, 'communication', 'communication network' and 'network' may be used as the same meaning. The three terms refer to wired and wireless local and wide area data transmission and reception networks capable of transmitting and receiving files between a user terminal, a terminal of other users, and a download server.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating an operation of a motion retargeting apparatus according to an embodiment of the present invention.

Referring to FIG. 1, the motion retargeting apparatus 100 according to an embodiment of the present invention receives the source motion data 101 and acquires the retargeted motion data 103.

The source motion data 101 is input to the motion retargeting device 100.

The source motion data 101 may be animation data.

The source motion data 101 may include joint positions information.

In this case, the joint positions information may be a vector.

The source motion data 101 may include hierarchy information of a bone.

In this case, the hierarchy information of the bone may be information on which arm the hand is coupled to, and information on which shoulder the arm is coupled to the hand is coupled to.

The source motion data 101 may include connectivity information.

The source motion data 101 may include information about a bone of a fixed length.

The source motion data 101 may include bone information.

In this case, the bone information may be at least one of bone rotation information, bone position information or bone rotation information and position information.

The source motion data 101 may include rotation information of a bone.

The source motion data 101 may include location information of a bone.

The source motion data 101 may include rotation information and position information of a bone.

In this case, the bone may correspond to at least one of the bones constituting the human body.

In addition, the bone may correspond to at least one of the bones constituting the upper body or lower body.

In addition, the bone may correspond to at least one of the bones of the parts of the body that is arbitrarily classified according to the needs (eg, retargeting motion) of the bones constituting the human body.

In addition, the bone may correspond to at least one of a spine, a left arm, a right arm, a left leg, and a right leg bone.

In addition, the bone may correspond to at least one of each bone constituting the spine, left arm, right arm, left leg and right leg. Can be.

In addition, the bone may correspond to at least one of a spine, an arm, and a leg.

In addition, the bone may correspond to at least one of each bone constituting a spine, arm and leg.

The source motion data 101 may be indexed data.

The source motion data 101 may be data in which rotation information of the bone is indexed.

The source motion data 101 may be data in which bone position information is indexed.

The source motion data 101 may be data obtained by indexing bone rotation information and position information.

The source motion data 101 may be data classified according to bone rotation information.

The source motion data 101 may be data classified according to the location information of the bone.

The source motion data 101 may be classified data according to bone rotation information and position information.

The source motion data 101 may be data classified according to the rotation information of the indexed bone.

The source motion data 101 may be data classified according to indexed bone position information.

The source motion data 101 may be data classified according to the rotation information and the position information of the indexed bone.

2 is a view showing the configuration of a motion retargeting apparatus according to an embodiment of the present invention.

2, a motion retargeting apparatus 100 according to an embodiment of the present invention may include a first convolutional neural network, a retargeting feature value generation module 120, and a second convolutional. A neural network 130 and a loss function value determination module 140.

The first convolutional neural network 110 is preferably a convolutional neural network learned according to an embodiment of the present invention, but is not limited thereto.

The first convolutional neural network 110 obtains source motion data 101.

In this case, the source motion data 101 may be data classified as bones constituting the human body according to indexing bone rotation information.

In addition, the source motion data 101 may be data classified into bones constituting the upper or lower body according to indexing bone rotation information.

In addition, the source motion data 101 is data classified into bones of a portion of the human body that is arbitrarily classified according to the need for the bones constituting the human body according to indexing bone rotation information (eg, retargeting motion). Can be entered.

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg and a right leg according to indexing bone rotation information. right leg).

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg and a right leg according to indexing bone rotation information. The data may be classified into each bone constituting the right leg).

In addition, the source motion data 101 may be data classified into spine, arm, and leg according to indexing bone rotation information.

In addition, the source motion data 101 may be data classified into respective bones constituting a spine, an arm, and a leg according to indexing bone rotation information.

In addition, the source motion data 101 may be data classified into bones constituting the human body according to indexed bone position information.

In addition, the source motion data 101 may be data classified into bones constituting the upper body or the lower body according to indexed bone position information.

In addition, the source motion data 101 is data classified into bones of a portion of the human body that is arbitrarily classified according to the need for the bones constituting the human body according to indexed bone position information (eg, retargeting motion). Can be entered.

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg, and a right leg according to indexed bone position information. right leg).

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg, and a right leg according to indexed bone position information. The data may be classified into each bone constituting the right leg).

In addition, the source motion data 101 may be data classified into spine, arm, and leg according to indexed bone position information.

In addition, the source motion data 101 may be data classified into respective bones constituting a spine, arm, and leg according to indexed bone position information.

In addition, the source motion data 101 may be data classified as bones constituting the human body according to indexed bone rotation information and position information.

In addition, the source motion data 101 may be data classified as bones constituting the upper or lower body according to the rotation information and the position information of the indexed bone.

In addition, the source motion data 101 may be a bone of a portion of the human body that is arbitrarily classified according to the need (eg, a retargeted motion) as a bone constituting a human body according to indexed bone rotation information and position information. Classified data.

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg and a leg according to indexed bone rotation information and position information. The data may be classified into the right leg.

In addition, the source motion data 101 may include a spine, a left arm, a right arm, a left leg and a leg according to indexed bone rotation information and position information. The data may be classified into each bone constituting the right leg.

In addition, the source motion data 101 may be data classified into spines, arms, and legs according to indexed bone rotation information and position information.

In addition, the source motion data 101 may be data classified into respective bones constituting the spine, the arm, and the leg according to the rotational information and the location information of the indexed bone. .

The first convolutional neural network 110 obtains source motion feature values according to the source motion data 101.

The first convolutional neural network 110 may obtain source motion feature values according to the source motion data 101 classified according to the indexed bone rotation information.

The first convolutional neural network 110 may obtain source motion feature values according to the source motion data 101 classified according to the location information of the indexed bone.

The first convolutional neural network 110 may obtain source motion feature values according to the source motion data 101 classified according to the indexed bone rotation information and the location information.

The first convolutional neural network may include a plurality of subconvolutional neural networks.

The first convolutional neural network 110 may include as many subconvolutional neural networks as the number of bones constituting the human body.

The first convolutional neural network 110 may include two subconvolutional neural networks corresponding to the upper or lower body.

The first convolutional neural network 110 may include sub-convolutional neural networks corresponding to the number of parts that are arbitrarily classified according to a need (eg, a retargeted motion).

The first convolutional neural network 110 includes five subconvolutions corresponding to the spine, left arm, right arm, left leg and right leg. It can include voluminous neural networks.

The first convolutional neural network 110 may include three subconvolutional neural networks corresponding to a spine, an arm, and a leg.

The first convolutional neural network 110 may match the source motion data 101 with the subconvolutional neural networks.

The first convolutional neural network 110 may obtain source motion feature values using the matched subconvolutional network.

The first convolutional neural network 110 constitutes the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexing bone rotation information. Match the same number of subconvolutional neural networks as the number of bones.

The first convolutional neural network 110 has the same number of subconvolutions as the number of bones constituting the human body matched with the source motion data 101 classified according to the indexed bone rotation information. Neural networks may be used to obtain source motion feature values.

The first convolutional neural network 110 includes the source motion data 101 classified according to the indexed bone rotation information and the upper or lower body included in the first convolutional neural network 110. Corresponding two subconvolutional neural networks may be matched.

The first convolutional neural network 110 includes two subconvolutional neural networks corresponding to the upper or lower body matched with the source motion data 101 classified according to indexed bone rotation information. Source motion feature values can be obtained.

The first convolutional neural network 110 needs the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone rotation information ( For example, the subconvolutional neural networks corresponding to the number of randomly classified parts may be matched according to the retargeted motion).

The first convolutional neural network 110 randomly classifies the human body matched with the source motion data 101 classified according to the indexed bone rotation information according to a need (eg, a retargeted motion). Source motion feature values may be obtained using subconvolutional neural networks corresponding to the number of one portion.

The first convolutional neural network 110 includes the source motion data 101 classified according to indexed bone rotation information and the spine included in the first convolutional neural network 110. Five subconvolutional neural networks corresponding to the left arm, the right arm, the left leg, and the right leg may be matched.

The first convolutional neural network 110 may include the spine, the left arm, and the right matched with the source motion data 101 classified according to indexed bone rotation information. Source motion feature values may be obtained using five subconvolutional neural networks corresponding to the right arm, the left leg, and the right leg.

The first convolutional neural network 110 includes the source motion data 101 classified according to indexed bone rotation information and the spine included in the first convolutional neural network 110. The three subconvolutional neural networks corresponding to the arm and the leg may be matched.

The first convolutional neural network 110 includes the spine, the arm and the leg matched with the source motion data 101 classified according to the indexed bone rotation information. Source motion feature values may be obtained using three subconvolutional neural networks corresponding to.

The first convolutional neural network 110 constitutes the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone position information. Match the same number of subconvolutional neural networks as the number of bones.

The first convolutional neural network 110 includes the same number of subconvolutions as the number of bones constituting the human body matched with source motion data 101 classified according to indexed bone position information. Neural networks may be used to obtain source motion feature values.

The first convolutional neural network 110 may include source motion data 101 classified according to indexed bone position information and the upper or lower body included in the first convolutional neural network 110. Corresponding two subconvolutional neural networks may be matched.

The first convolutional neural network 110 includes two subconvolutional neural networks corresponding to the upper or lower body matched with the source motion data 101 classified according to the indexed bone position information. Source motion feature values can be obtained.

The first convolutional neural network 110 needs the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone position information ( For example, the subconvolutional neural networks corresponding to the number of randomly classified parts may be matched according to the retargeted motion).

The first convolutional neural network 110 arbitrarily classifies the human body matched with the source motion data 101 classified according to the indexed bone position information according to a need (eg, a retargeted motion). Source motion feature values may be obtained using subconvolutional neural networks corresponding to the number of one portion.

The first convolutional neural network 110 includes the source motion data 101 classified according to the indexed bone position information and the spine included in the first convolutional neural network 110. Five subconvolutional neural networks corresponding to the left arm, the right arm, the left leg, and the right leg may be matched.

The first convolutional neural network 110 may include the spine, the left arm, and the right matched with source motion data 101 classified according to indexed bone position information. Source motion feature values may be obtained using five subconvolutional neural networks corresponding to the right arm, the left leg, and the right leg.

The first convolutional neural network 110 includes the source motion data 101 classified according to the indexed bone position information and the spine included in the first convolutional neural network 110. The three subconvolutional neural networks corresponding to the arm and the leg may be matched.

The first convolutional neural network 110 includes the spine, the arm and the leg matched with the source motion data 101 classified according to the indexed bone position information. Source motion feature values may be obtained using three subconvolutional neural networks corresponding to.

The first convolutional neural network 110 includes the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone rotation information and position information. The same number of subconvolutional neural networks as the number of bones constituting may be matched.

The first convolutional neural network 110 serves the same number of sub-numbers as the number of bones constituting the human body matched with the source motion data 101 classified according to the rotation information and the position information of the indexed bone. Convolutional neural networks can be used to obtain source motion feature values.

The first convolutional neural network 110 includes the upper body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone rotation information and position information. Alternatively, two subconvolutional neural networks corresponding to the lower body may be matched.

The first convolutional neural network 110 includes two subconvolutions corresponding to the upper or lower body matched with the source motion data 101 classified according to the rotational information and the position information of the indexed bone. Neural networks may be used to obtain source motion feature values.

The first convolutional neural network 110 includes the human body included in the source motion data 101 and the first convolutional neural network 110 classified according to indexed bone rotation information and position information. May match subconvolutional neural networks corresponding to the number of arbitrarily classified parts according to needs (eg, retargeted motion).

The first convolutional neural network 110 may need the human body matched with the source motion data 101 classified according to the indexed bone rotation information and the location information (eg, a retargeted motion). Accordingly, source motion feature values may be obtained using sub-convolutional neural networks corresponding to the number of arbitrarily classified portions.

The first convolutional neural network 110 includes the source motion data 101 classified according to the rotation information and the position information of the indexed bone and the spine included in the first convolutional neural network 110. 5 subconvolutional neural networks corresponding to the spine, the left arm, the right arm, the left leg, and the right leg. have.

The first convolutional neural network 110 includes the spine and the left arm matched with the source motion data 101 classified according to the indexed bone rotation information and the location information. Source motion feature values may be obtained using five sub-convolutional neural networks corresponding to the right arm, left leg, and right leg.

The first convolutional neural network 110 includes the source motion data 101 classified according to the rotation information and the position information of the indexed bone and the spine included in the first convolutional neural network 110. Three subconvolutional neural networks corresponding to the spine, the arm and the leg may be matched.

The first convolutional neural network 110 may include the spine, the arm, and the matched source motion data 101 classified according to indexed bone rotation information and position information. Source motion feature values may be obtained using three subconvolutional neural networks corresponding to a leg.

Source motion feature values acquired by the first convolutional neural network 110 may be referred to as a source motion feature map.

Source motion feature values acquired by the first convolutional neural network 110 may be referred to as source motion vectors.

 An array of source motion feature values acquired by the first convolutional neural network 110 may be referred to as a source motion vector.

The retargeting feature value generation module 120 combines the target information with the source motion feature values obtained by the first convolutional neural network 110 to generate retargeting feature values.

In this case, the target information may include length information of a bone.

In addition, the target information may include ratio information of bones.

In addition, the target information may include bone length ratio information.

In addition, the target information may include bone length information, bone ratio information, and bone length ratio information.

In addition, the target information may include length information of a spine, a left arm, a right arm, a left leg, and a right leg bone.

In addition, the target information may include ratio information of a spine, a left arm, a right arm, a left leg, and a right leg bone.

In addition, the target information may include length ratio information of a spine, a left arm, a right arm, a left leg and a right leg bone.

The target information may include length information of the spine, the left arm, the right arm, the left leg and the right leg bones, the spine, the left arm. (left arm), right arm, left leg and right leg bone ratio information and spine, left arm, right arm, left The length ratio information of the left leg and right leg bones may be included.

In addition, the target information may be a vector.

In addition, the target information may include length information of a spine, arm, and leg bone.

In addition, the target information may include ratio information of the spine, arm, and leg bones.

In addition, the target information may include length ratio information of the spine, arm, and leg bones.

Further, the target information may include length information of the spine, arm, and leg bones, ratio information of the spine, arm, and leg bones, and spine. And length ratio information of the arm, leg, and leg bones.

The retargeting feature values generation module 120 may include a fully connected layer.

The retargeting feature values generation module 120 may be implemented with a fully connected layer.

The retargeting feature values generation module 120 may generate retargeting features by combining the target information with the source motion feature values using the fully connected layer.

The retargeting feature values generating module 120 may generate retargeting feature values by combining the target information with the source motion feature values in the fully connected layer.

The second convolutional neural network 130 uses the retargeting feature values generated by the retargeting feature values generation module 120 to retarget the motion data 103. ).

The second convolutional neural network 130 is preferably a convolutional neural network learned according to an embodiment of the present invention, but is not limited thereto.

In this case, the retargeted motion data 103 may be animation data.

In addition, the retargeted motion data 103 may include joint positions information.

In this case, the joint positions information may be a vector.

In addition, the retargeted motion data 103 may include rotation information of a bone.

In addition, the retargeted motion data 103 may include location information of a bone.

In addition, the retargeted motion data 103 may include rotation information and position information of a bone.

In this case, the bone may correspond to at least one of the bones constituting the human body.

In addition, the bone may correspond to at least one of the bones constituting the upper body or lower body.

In addition, the bone may correspond to at least one of the bones of a portion that is arbitrarily classified according to the needs (eg, retargeting motion) constituting the human body.

In addition, the bone may correspond to at least one of a spine, a left arm, a right arm, a left leg, and a right leg bone.

In addition, the bone may correspond to at least one of each bone constituting the spine, the left arm, the right arm, the left leg, and the right leg. Can be.

In addition, the bone may correspond to at least one of a spine, an arm, and a leg.

In addition, the bone may correspond to at least one of each bone constituting a spine, arm and leg.

In addition, the retargeted motion data 103 may be indexed data.

In addition, the retargeted motion data 103 may be data in which rotation information of the bone is indexed.

In addition, the retargeted motion data 103 may be data in which bone position information is indexed.

In addition, the retargeted motion data 103 may be data in which bone rotation information and position information are indexed.

In addition, the retargeted motion data 103 may be data classified according to bone rotation information.

In addition, the retargeted motion data 103 may be data classified according to bone position information.

In addition, the retargeted motion data 103 may be data classified according to bone rotation information and position information.

In addition, the retargeted motion data 103 may be data classified according to indexed bone rotation information.

In addition, the retargeted motion data 103 may be data classified according to indexed bone position information.

In addition, the retargeted motion data 103 may be data classified according to the rotation information and the position information of the indexed bone.

In addition, the retargeted motion data 103 may be data classified as bones constituting a human body according to indexing bone rotation information.

In addition, the retargeted motion data 103 may be data classified into bones constituting the upper body or the lower body according to indexing bone rotation information.

In addition, the retargeted motion data 103 classifies a bone constituting a human body according to indexing bone rotation information (eg, a retargeted motion) as a bone of a randomly classified portion of the human body. May be data.

In addition, the retargeted motion data 103 may include a spine, left arm, right arm, left leg, and right leg according to indexing bone rotation information. The data may be classified into right legs.

In addition, the retargeted motion data 103 may include a spine, left arm, right arm, left leg, and right leg according to indexing bone rotation information. The data may be classified into each bone constituting the right leg.

In addition, the retargeted motion data 103 may be data classified into spines, arms, and legs according to indexing bone rotation information.

In addition, the retargeted motion data 103 may be data classified into respective bones constituting a spine, arm, and leg according to indexing bone rotation information. .

In addition, the retargeted motion data 103 may be data classified into bones constituting a human body according to indexed bone position information.

In addition, the retargeted motion data 103 may be data classified into bones constituting the upper body or the lower body according to indexed bone position information.

In addition, the retargeted motion data 103 classifies a bone constituting a human body according to indexed bone position information (eg, a retargeted motion) as a bone of a part of the body that is arbitrarily classified. May be data.

In addition, the retargeted motion data 103 may include a spine, left arm, right arm, left leg, and right leg according to indexed bone position information. The data may be classified into right legs.

In addition, the retargeted motion data 103 may include a spine, left arm, right arm, left leg, and right leg according to indexed bone position information. The data may be classified into each bone constituting the right leg.

In addition, the retargeted motion data 103 may be data classified into spine, arm, and leg according to indexed bone position information.

In addition, the retargeted motion data 103 may be data classified into respective bones constituting the spine, arm, and leg according to the indexed bone position information. .

In addition, the retargeted motion data 103 may be data classified as bones constituting a human body according to indexed bone rotation information and position information.

In addition, the retargeted motion data 103 may be data classified into bones constituting the upper or lower body according to the rotation information and the position information of the indexed bone.

In addition, the retargeted motion data 103 is a part that arbitrarily classifies the human body according to the bone constituting the human body according to indexed bone rotation information and position information (eg, a retargeted motion). The data may be classified as bones of.

In addition, the retargeted motion data 103 may include a spine, a left arm, a right arm, and a left leg according to the rotational information and the position information of the indexed bone. And right leg.

In addition, the retargeted motion data 103 may include a spine, a left arm, a right arm, and a left leg according to the rotational information and the position information of the indexed bone. And the bones constituting the right leg.

In addition, the retargeted motion data 103 may be data classified into spine, arm, and leg according to indexed bone rotation information and position information.

In addition, the retargeted motion data 103 includes data classified into respective bones constituting the spine, arm, and leg according to the rotational information and the position information of the indexed bone. Can be.

The second convolutional neural network 130 may include a plurality of subconvolutional neural networks.

The second convolutional neural network 130 may include the same number of subconvolutional neural networks as the number of bones constituting the human body.

The second convolutional neural network 130 may include two subconvolutional neural networks corresponding to the upper or lower body.

The second convolutional neural network 130 may include sub-convolutional neural networks corresponding to the number of parts that are arbitrarily classified according to a need (eg, retargeted motion).

The second convolutional neural network 130 includes five sub-convolutions corresponding to the spine, left arm, right arm, left leg and right leg. It can include voluminous neural networks.

The second convolutional neural network 130 may include three subconvolutional neural networks corresponding to the spine, arm, and leg.

The second convolutional neural network 130 may match the retargeting feature values generated by the retargeting feature values generation module 120 with the subconvolutional neural networks.

The second convolutional neural network 130 may acquire the retargeted motion data using the matched subconvolutional network.

The second convolutional neural network 130 may include the retargeting feature values generated by the retargeting feature values generation module 120 and the number of bones constituting the human body included in the second convolutional neural network 130. It is possible to match the same number of subconvolutional neural networks.

The second convolutional neural network 130 generates a number of subconvolutional neural networks equal to the number of bones constituting the human body matched with the generated retargeting feature values by the retargeting feature values generating module 120. The retargeted motion data may be obtained by using the same.

The second convolutional neural network 130 corresponds to the upper body or lower body included in the retargeting feature values generated by the retargeting feature values generation module 120 and the second convolutional neural network 130. Subconvolutional neural networks may be matched.

The second convolutional neural network 130 uses the two subconvolutional neural networks corresponding to the upper or lower body matched with the generated retargeting feature values by the retargeting feature value generation module 120. Targeted motion data can be obtained.

The second convolutional neural network 130 needs the retargeting feature values generated by the retargeting feature values generation module 120 and the human body included in the second convolutional neural network 130 (eg, the retargeting feature values generation module 120). Subconvolutional neural networks corresponding to the number of segments classified arbitrarily according to the targeted motion).

The second convolutional neural network 130 is configured to selectively classify the human body matched with the generated retargeting feature values by the retargeting feature value generation module 120 according to a need (eg, a retargeted motion). Retargeted motion data may be obtained using subconvolutional neural networks corresponding to the number.

The second convolutional neural network 130 may include the retargeting feature values generated by the retargeting feature values generation module 120 and the spine and left arm included in the second convolutional neural network 130. Five subconvolutional neural networks corresponding to a left arm, a right arm, a left leg, and a right leg may be matched.

The second convolutional neural network 130 may include the spine, left arm, and right arm matched with the retargeting feature values generated by the retargeting feature values generation module 120. Retargeted motion data may be obtained using five subconvolutional neural networks corresponding to a left leg and a right leg.

The second convolutional neural network 130 includes three sub-targets corresponding to the retargeting feature values generated by the retargeting feature value generating module 120 and the spine, the arm, and the leg. Match convolutional neural networks.

The second convolutional neural network 130 corresponds to the spine, arm, and leg matched with the retargeting feature values generated by the retargeting feature values generation module 120. Three sub-convolutional neural networks may be used to obtain retargeted motion data.

According to another embodiment of the present invention, it is also possible to implement the first convolutional neural network 110 and the second convolutional neural network 130 as one convolutional neural network.

The loss function value determination module 140 updates the obtained retargeted motion data according to a loss function.

Loss function value determination module 140 may include a loss function.

In this case, the loss function may be implemented to increase the loss function value when kinematic constraints are violated.

The loss function value determination module 140 may determine the loss function value using the loss function included above.

The loss function value determination module 140 may update the obtained retargeted motion data according to the determined loss function value.

Loss function value determination module 140 may include a loss function for foot contacts.

Loss function value determination module 140 may determine a loss function value using a loss function for the foot contacts.

The loss function value determination module 140 may update the obtained retargeted motion data according to the determined loss function value using a loss function for the foot contacts.

The loss function value determination module 140 may include a loss function with respect to the trajectory.

The loss function value determination module 140 may determine the loss function value using a loss function for the trajectory.

The loss function value determination module 140 may update the obtained retargeted motion data according to the determined loss function value by using a loss function of the trajectory.

Loss function value determination module 140 may include a loss function with respect to bone length.

The loss function value determination module 140 may determine the loss function value using the loss function for the bone length.

The loss function value determination module 140 may update the obtained retargeted motion data according to the determined loss function value using a loss function relating to the bone length.

The loss function value determination module 140 may update the retargeted motion data through a kinematic optimization process until the loss function value becomes minimum.

The loss function value determination module 140 uses a back-propagation algorithm to adjust the retargeting feature values generated by the retargeting feature value generation module 120 in the kinematic optimization process. Can be.

At this time, the weight of the second convolutional neural network 130 is fixed.

The loss function value determination module 140 may update the retargeted motion data through a kinematic optimization process until the loss function value for the foot contacts becomes minimum. Can be.

The loss function value determination module 140 iteratively repeats the retargeted motion data through a kinematic optimization process until the loss function value for the foot contacts becomes minimum. You can update it.

The loss function value determination module 140 may update the retargeted motion data through a kinematic optimization process until the loss function value of the trajectory becomes minimum. .

The loss function value determination module 140 may repeatedly update the retargeted motion data through a kinematic optimization process until the loss function value of the trajectory becomes minimum. Can be.

The loss function value determination module 140 may update the retargeted motion data through a kinematic optimization process until the loss function value for the bone length becomes minimum. Can be.

The loss function value determination module 140 iteratively repeats the retargeted motion data through a kinematic optimization process until the loss function value for the bone length becomes minimum. You can update it.

As used herein, the term 'device or module' refers to a logical unit, and it is obvious to those skilled in the art that the present invention is not necessarily a physically divided component.

3 is a flowchart illustrating a motion retargeting method according to an embodiment of the present invention.

Referring to FIG. 3, in the motion retargeting method according to an embodiment of the present invention, the motion retargeting apparatus recognizes source motion data (300).

In this case, the source motion data may be animation data.

In addition, the source motion data may be data classified according to the rotation information of the indexed bone.

The motion retargeting device obtains source motion feature values (310).

In this case, the motion retargeting apparatus may obtain source motion feature values according to the source motion data 101 classified according to indexed bone rotation information and position information.

The motion retargeting apparatus combines the target information with the obtained source motion feature values to generate retargeting features values (320).

In this case, the target information may include length ratio information of a bone.

The motion retargeting device obtains the retargeted motion data using the generated retargeting features values (330).

In this case, the motion retargeting apparatus may reconstruct using the subconvolutional neural networks corresponding to the number of parts that are arbitrarily classified according to the needs (eg, retargeted motions) of the human body matched with the generated retargeting feature values. Targeted motion data can be obtained.

The motion retargeting device updates the obtained retargeted motion data according to a loss function (340).

In this case, the motion retargeting apparatus may update the retargeted motion through a kinematic optimization process until the loss function value becomes minimum.

4 is a diagram illustrating retargeted motion data in which an error of source motion data is automatically corrected according to an embodiment of the present invention.

Referring to FIG. 4A, the motion data on the left side of FIG. 4A represents source motion data, and the motion data on the right side of FIG. 4A represents retargeted motion data.

In the source motion data on the left side of FIG. 4A, the front portion of the right foot 401 of the source motion data indicated by the arrow is lifted to the left. This is a violation of kinematic constraints.

The front portion of the right foot 402 of the retargeted motion data on the right side of FIG. 4A shows the violated kinematic constraints contained in the source motion data according to one embodiment of the invention. This is automatically corrected to maintain the kinematic constraints.

Referring to FIG. 4B, motion data on the left side of FIG. 4B represents source motion data, and motion data on the right side of FIG. 4B represents retargeted motion data.

The source motion data on the left side of FIG. 4B is a state in which the front part of the right foot 403 of the source motion data indicated by the arrow is lifted to the right. This is a violation of kinematic constraints.

The front portion of the right foot 404 of the retargeted motion data on the right side of FIG. 4B shows the violated kinematic constraints contained in the source motion data according to one embodiment of the invention. This is automatically corrected to maintain the kinematic constraints.

Referring to FIG. 4C, motion data on the left of FIG. 4C represents source motion data, and motion data on the right of FIG. 4C represents retargeted motion data.

In the source motion data on the left side of FIG. 4C, the front portion of the right foot 405 of the source motion data indicated by the arrow is vertically stretched. This is a violation of kinematic constraints.

The front portion of the right foot 406 of the retargeted motion data on the right side of FIG. 4C shows the violated kinematic constraints contained in the source motion data according to one embodiment of the invention. This is automatically corrected to maintain the kinematic constraints.

FIG. 5 is a diagram illustrating motion data retargeted to various types and sizes according to an embodiment of the present invention.

Referring to FIG. 5, according to an embodiment of the present invention, motion data in which various source motion data are retargeted to various types and various sizes forms a pair of left and right 500 and 510. In this case, the motion data 501 and 511 on the left side correspond to the source motion data in the pair 500 and 510, and the motion data 502 and 512 on the right side represent the retargeted motion data.

Referring to the pair of motion data 500 in the lower middle portion of FIG. 5, it can be seen that the retargeted motion data 502 is a retargeted leg portion from the source motion data 501.

At this time, kinematic constraints are maintained in the retargeted motion data 502 in which the leg portion is retargeted in the source motion data 501.

Referring to the pair of motion data 510 of the upper middle portion of FIG. 5, it can be seen that the retargeted motion data 512 has been retargeted to the arm, leg, and size of the source motion data 511.

At this time, kinematic constraints are maintained in the retargeted motion data 512 in which the arm, leg, and size portions of the source motion data 511 are retargeted.

6 is a block diagram of an exemplary computer system for implementing one embodiment of the present invention.

Referring to FIG. 6, an exemplary computer system for implementing one embodiment of the present invention includes a bus or other communication channel 601 for exchanging information, and the processor 602 uses a bus (or bus) for processing information. 601).

Computer system 600 includes main memory 603, which is a random access memory (RAM) or other dynamic storage device coupled to bus 601 to store information and instructions processed by processor 602.

In addition, main memory 603 may be used to store temporary variables or other intermediate information during execution of instructions by processor 602.

Computer system 600 may include Read Only Memory (ROM) and other static storage 604 coupled to bus 601 to store static information or instructions for processor 602.

Data storage 605 and its corresponding drive, such as a magnetic disk, zip or optical disk, may also be coupled to computer system 600 to store information and instructions.

The computer system 600 may be connected to a display device 610, such as a cathode ray tube or an LCD, via a bus 601 to display information to an end user.

A character input device, such as keyboard 620, may be coupled to bus 601 to convey information and instructions to processor 602.

Another type of user input device is a cursor control device 630, such as a mouse, trackball or cursor direction keys for conveying direction information and command selection to the processor 602 and controlling the movement of the cursor on the display 610.

The communication device 640 is also connected to the bus 601.

Communication device 640 may include an interface device used to connect with a modem, network interface card, Ethernet, token ring, or other type of physical combination to support connection to a local or wide area network. In this manner, computer system 600 may be connected to multiple clients and servers through conventional network infrastructure, such as the Internet.

In the above description, all the components constituting the embodiments of the present invention are described as being combined or operating in combination, but the present invention is not necessarily limited to the embodiments. That is, within the scope of the present invention, all the components may be selectively combined to operate at least one.

In addition, although all of the components may be implemented in one independent hardware, each or all of the components are selectively combined to perform some or all of the functions combined in one or a plurality of hardware. It may be implemented as a computer program having a. Codes and code segments constituting the computer program may be easily inferred by those skilled in the art.

Such a computer program may be stored in a computer readable storage medium and read and executed by a computer, thereby implementing embodiments of the present invention. The storage medium of the computer program may include a magnetic recording medium, an optical recording medium, and the like.

In addition, the terms "comprise", "comprise", or "having" described above mean that the corresponding component may be included unless otherwise stated, and thus, other components are excluded. It should be construed that it may further include other components instead.

All terms, including technical and scientific terms, have the same meanings as commonly understood by one of ordinary skill in the art unless otherwise defined. Terms commonly used, such as terms defined in a dictionary, should be interpreted to coincide with the contextual meaning of the related art, and shall not be interpreted in an ideal or excessively formal sense unless explicitly defined in the present invention.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art may make various modifications and changes without departing from the essential characteristics of the present invention.

Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100 ... Motion Retargeting Device 110 ... First Convolutional Neural Network
120 Retargeting feature value generation module 130 Second convolutional neural network
140 ... Loss Function Value Determination Module

Claims (15)

The motion retargeting apparatus obtaining source motion feature values according to the source motion data;
Generating, by the motion retargeting device, retargeting features by combining target information with the obtained source motion feature values;
Obtaining, by the motion retargeting device, the retargeted motion data using the generated retargeting features values; And
The motion retargeting device updating the obtained retargeted motion data according to a loss function
Motion retargeting method comprising a. The method of claim 1,
Updating the obtained retargeted motion data,
And retargeting the motion data through a kinematic optimization process until the loss function is minimized. The method of claim 2,
The kinematic optimization process,
Motion retargeting method using a back-propagation algorithm to adjust the retargeting feature values. The method of claim 1,
And the source motion data and the retargeted motion data are animation data. The method of claim 1,
And the source motion data and the retargeted motion data include joint positions information. The method of claim 1,
And the source motion data includes bone information. The method of claim 1,
The target information is a motion retargeting method comprising length ratio information of bones. A computer program performing the method of any one of claims 1 to 7, when executed on one or more processes. In a motion retargeting device,
The motion retargeting device includes a processor,
The processor,
Obtaining source motion feature values according to the source motion data,
Retargeting features values are generated by combining target information with the obtained source motion feature values;
Obtaining retargeted motion data by using the generated retargeting features values,
And a motion retargeting device for updating the obtained retargeted motion data according to a loss function. The method of claim 9,
Updating the obtained retargeted motion data,
And retargeting the motion data through kinematic optimization until the loss function is minimized. The method of claim 10,
The kinematic optimization process,
Motion retargeting apparatus using a back-propagation algorithm to adjust the retargeting feature values. The method of claim 9,
And the source motion data and the retargeted motion data are animation data. The method of claim 9,
And the source motion data and the retargeted motion data include joint position information. The method of claim 9,
And the source motion data includes bone information. The method of claim 9,
The target information is a motion retargeting device including the length ratio information of the bone (bone).

Images