KR102310757B1 - Method for generating human motion using sequential networks and apparatus thereof - Google Patents

Abstract

Disclosed are a human motion generating method using sequential networks and an apparatus thereof, which can generate various human motions through a motion manifold. According to one embodiment of the present invention, the human motion generating method comprises: a step of constructing a motion manifold by using sequential networks; and a step of generating a target motion corresponding to motion input data by using the constructed motion manifold. The constructing step can construct the motion manifold by using sequential networks including one encoder and two decoders. The generating step can output joint rotation angle information for the target motion by one of the two decoders, output joint rotation speed information for the target motion by the other decoder, and generate the target motion based on the joint rotation angle information and the joint rotation speed information.

Description

Method and device for generating human motion using sequential network

The present invention relates to a human motion generation technology, and more specifically, a person capable of constructing a motion manifold using sequential networks and generating various human motions through the motion manifold. It relates to a method for generating motion and an apparatus therefor.

The potential spatial construction of human motion is an important issue because the range of applications such as motion recognition, prediction, interpolation, and synthesis is wide. An ideal motion space should be reduced in order for random sampling in space to induce plausible motion and to generate a wide range of human motion. In addition, the local linear arrangement of semantically related hidden vectors can be helpful for motion synthesis by simple algebraic operations.

However, since each part of the human body is closely related when performing general motion, and the joint is limited to satisfy the bone length and range of motion, it is difficult to construct a reduced motion space and regenerate effective motion from it. remains as The high dimensionality of the joint space adds additional difficulties to this problem.

Although several methods for constructing a motion manifold have been developed in the prior art, studies on motion data are lacking compared to studies on manifold learning for other data such as images. Linear methods such as PCA can only model human motion in the local domain. The conventional technique of one embodiment applies local PCA to generate a motion manifold including a range of human motion, and applies it to synthesize motion from a low-dimensional input such as the position of an end effector. In another conventional technique, a Gaussian Process Latent Variable Model (GPLVM) is used to find a low-dimensional latent space for high-dimensional motion data. Another conventional technique of an embodiment proposes a modified Restricted Boltzmann Machine capable of handling temporal consistency of motion data. Another conventional technique of one embodiment proposes a motion field method, which is a new representation of motion data that can create human motion in response to any external disturbance. Recently, with the development of deep learning technology, a technology using deep learning technology has been proposed, and the conventional technology according to an embodiment uses a convolutional neural network (CNN)-based encoder to configure a motion manifold. The conventional technique according to another embodiment compares various deep learning frameworks for human motion data modeling.

There have been studies on motion sequence learning for predicting the joint position sequence of a 3D human body given a past motion, and the conventional technique of an embodiment is a new sequence-to-sequence encoder decoder model that predicts the motion of a person given a short duration of the past motion. However, the presented results have some limitations in that unreliable motion such as foot sliding is generated and the initial pose of the predicted motion is somewhat discontinuous with the input motion. The conventional technique of another embodiment uses joint rotation-based loss for short-term prediction and selectively uses joint position-based loss for long-term prediction, wherein the joint position-based loss includes anterior kinematics for calculating the joint position. However, the basic sequence-to-sequence model can only predict short-term motion and has limitations in predicting non-trivial and long-term motion. Also, a loss function that only minimizes the prediction error is not guaranteed to form a compact, versatile motion manifold.

A conventional technology of one embodiment for recognizing and generating human motion proposes a hierarchical dynamic framework for extracting the highest skeletal joint shape and inferring a motion sequence using the learned representation, and the technique of another embodiment of the prior art is regression We model temporal motion sequences using a recursive neural network (RNN) and propose a hierarchical structure for action recognition. With respect to motion synthesis, the conventional technique of an embodiment proposes a new class of Recurrent Temporal Restricted Boltzmann Machine (RTRBM). Structured RTRBM explicitly uses graphs to model dependent structures to improve motion synthesis quality. Conventionally, the technology of another embodiment proposes an Encoder-Recurrent-Decoder (ERD) that combines expression learning and learning time dynamics to recognize and predict a human body posture in video and motion capture, and the technology of another conventional embodiment is We propose a structured RNN to combine the power of high-level spatiotemporal graphs.

Embodiments of the present invention provide a human motion generation method and apparatus capable of constructing a motion manifold using sequential networks and generating various human motions through the motion manifold do.

A method for generating human motion according to an embodiment of the present invention includes: configuring a motion manifold using a sequential network; and generating a target motion corresponding to the motion input data using the configured motion manifold.

In the configuring, the motion manifold may be configured using a sequential network including one encoder and two decoders.

The generating may include outputting joint rotation angle information for the target motion from any one of the two decoders, and outputting joint rotation speed information for the target motion from the other decoder, and the joint rotation The target motion may be generated based on the angle information and the joint rotation speed information.

The configuring may configure the learning model-based motion manifold by learning the learning model of the sequential network so that the distribution of the motion manifold matches a preset pre-distribution.

In the configuring step, in learning the learning model of the sequential network, the joint position is calculated using a forward kinematics technique, and the calculated joint position is further reflected to learn the learning model. , it is possible to configure a motion manifold based on the learning model.

A method for generating human motion according to another embodiment of the present invention includes: configuring a motion manifold using a sequential network including one encoder and two decoders; and generating a target motion corresponding to the motion input data using the configured motion manifold.

The generating may include outputting joint rotation angle information for the target motion from any one of the two decoders, and outputting joint rotation speed information for the target motion from the other decoder, and the joint rotation The target motion may be generated based on the angle information and the joint rotation speed information.

An apparatus for generating human motion according to an embodiment of the present invention includes: a configuration unit that configures a motion manifold using a sequential network; and a generator configured to generate a target motion corresponding to the motion input data by using the configured motion manifold.

The configuration unit may configure the motion manifold using a sequential network including one encoder and two decoders.

The generator outputs joint rotation angle information for the target motion from one of the two decoders, and outputs joint rotation speed information for the target motion from the other decoder, and the joint rotation angle information and the target motion may be generated based on the joint rotation speed information.

The configuration unit may configure the learning model-based motion manifold by learning the learning model of the sequential network so that the distribution of the motion manifold matches a preset distribution.

In learning the learning model of the sequential network, the configuration unit calculates the position of the joint using a forward kinematics technique, and further reflects the calculated position of the joint to learn the learning model, It is possible to configure a motion manifold based on a learning model.

According to embodiments of the present invention, a motion manifold may be configured using a sequential network, and various human motions may be generated through the motion manifold.

According to embodiments of the present invention, since a motion manifold is built by learning limited motion data through a sequential deep learning network, using such a motion manifold, denoising data and regenerating new motion through interpolation It can be applied to various applications such as , motion recognition and category classification. In addition, since the motion manifold or motion reduction space in the present invention regenerates dynamic and various new motions, considerable cost and time can be saved when applied to computer graphics and animation fields.

The present invention can help to generate motion data of an avatar in an augmented reality (AR)/virtual reality (VR) space, and can further be used in a technology for generating automatic movement, such as an AI-based character.

1 is a flowchart illustrating a method for generating a human motion according to an embodiment of the present invention.
2 is a diagram illustrating an example for explaining a process of configuring a motion manifold.
3 shows an exemplary diagram for explaining each loss function.
4 and 5 show exemplary views of the target motion generated by the method of the present invention.
6 illustrates a conceptual configuration of a human motion generating apparatus according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the art to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments, and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and/or "comprising" refers to the presence of one or more other components, steps, operations and/or elements mentioned. or addition is not excluded.

Unless otherwise defined, all terms (including technical and scientific terms) used herein may be used with the meaning commonly understood by those of ordinary skill in the art to which the present invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

Hereinafter, preferred embodiments of the present invention will be described in more detail with reference to the accompanying drawings. The same reference numerals are used for the same components in the drawings, and repeated descriptions of the same components are omitted.

Embodiments of the present invention are directed to providing a novel framework for constructing latent motion manifolds and generating various human motions in motion manifolds.

Here, the present invention can be based on a sequence-versus-sequence model to accommodate the temporal characteristics of human motion. The sequence-versus-sequence model is based on existing studies on the motion side (MARTINEZ J., BLACK MJ, ROMERO J.: On human In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), IEEE, pp. 4674-4683., PAVLLO D., GRANGIER D., AULI M.: Quaternet: A quaternion- based recurrent model for human motion. arXiv preprint arXiv:1805.06485 (2018).)

This invention makes several new technical contributions to achieve a latent motion manifold and a motion generation method.

First, the model of the present invention is characterized as a combination of one encoder and two decoders. When using motion manifold vectors, one decoder learns to generate joint rotation and the other learns to output joint rotation velocity. The joint rotation decoder has the advantage of being able to better reconstruct long term motion. On the other hand, the joint velocity decoder has the advantage of improving the continuity of motion. By complementing each other, the two decoder models of the present invention show higher reconstruction accuracy than a single decoder model.

Second, unlike previous studies that dealt only with joint angle or joint position, the joint angle-based technique of the present invention adds an anterior kinematic (FK) layer to satisfy the bone length constraint and simplify the limit expression of joint angle. get an advantage The present invention further takes into account the joint position calculated by the anterior kinematics layer during training, thereby reducing joint position errors that are visually better than joint angle errors.

Finally, the present invention includes several loss functions that each contribute to improving the quality of the motion manifold in different aspects. The reconstruction loss reduces the difference between the reconstructed motion and the input motion, allowing the manifold to synthesize the motion content and details observed in the training motion data set. The regularizer loss improves the distribution quality of the motion manifold, enabling random sampling and interpolation operations in the manifold. Also, the adversarial loss increases the naturalness of the motion generated in the motion manifold.

Based on this technical contribution, the present invention can be applied to various applications such as random motion generation, motion interpolation, motion denoising, and motion analogy.

The present invention will be described in detail with reference to FIGS. 1 to 6 .

1 is a flowchart illustrating a method for generating a human motion according to an embodiment of the present invention.

Referring to FIG. 1 , in the method for generating human motion according to an embodiment of the present invention, a step of configuring a motion manifold using a sequential network ( S110 ) and a target motion corresponding to motion input data using the motion manifold are generated. and generating (S120).

In step S110, a motion manifold may be configured using a sequential network including one encoder and two decoders, and by training a learning model of the sequential network so that the distribution of the motion manifold matches a preset pre-distribution, learning A model-based motion manifold can be configured.

At this time, in step S110, in learning the learning model of the sequential network, the joint position is calculated using a forward kinematics technique, and the calculated joint position is additionally reflected to learn the learning model. Model-based motion manifolds can also be configured. Of course, the learning model of the sequential network may be learned through preset training data based on at least one or more preset loss functions.

In step S120, by outputting joint rotation angle information for the target motion from any one of the two decoders constituting the motion manifold, and outputting joint rotation speed information for the target motion from the other decoder, joint rotation A target motion corresponding to the motion input data may be generated based on the angle information and the joint rotation speed information.

The method of the present invention will be described in detail with reference to FIGS. 2 and 5 .

For the present invention, we define the notation and sequentially describe the design of the network structure and loss function for training.

Expressions and Notations

로 설정된 사람 모션과 Q에 해당하는 무작위 변수를 나타낸다. [t, t+βt-1]의 시간 영역에서 모션은

로 나타낼 수 있다. 여기서, q_t는 시간 t에서의 포즈를 의미할 수 있다. 포즈는 지수좌표로 쓰여진 관절 각도 집합 즉,

와 같이 나타낼 수 있다. 여기서,

는 지수좌표의 세 구성요소를 의미하고, n_joint는 관절의 수를 의미할 수 있다. 그러므로, 사람 모션의 차원은

이다. 마지막으로, p_t는 q_t에 해당하는 시간 t의 관절 위치로 대표되는 포즈이고,

이다. 여기서, P는 모션 집합

의 무작위 변수를 의미할 수 있다.the present invention

Indicates the human motion set to , and a random variable corresponding to Q. In the time domain of [t, t+βt-1], the motion is

can be expressed as Here, q _t may mean a pose at time t. A pose is a set of joint angles written in exponential coordinates, i.e.,

can be expressed as here,

denotes three components of exponential coordinates, and n _joint can denote the number of joints. Therefore, the dimension of human motion is

am. Finally, p _t is the pose represented by the joint position at time t corresponding to _{q t ,}

am. where P is the motion set

may mean a random variable of

using a sequential network motion manifold

The present invention constructs a motion manifold in an end-to-end manner using a network of sequential networks in order to minimize the difference between the actual data motion spatial distribution and the reconstructed motion spatial distribution extracted from the latent motion manifold. For this, the present invention may use a sequential model composed of an RNN including a Gated Recurrent Unit (GRU). The model of the present invention may have a continuous structure frequently used for machine translation, and this continuous structure has been studied in previous studies (MARTINEZ J., BLACK MJ, ROMERO J.: On human motion prediction using recurrent neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2017), IEEE, pp. 4674-4683.). This RNN structure is effective in maintaining the temporal consistency of motion.

2 shows the structure of one embodiment of a sequential network for configuring a motion manifold.

As shown in Fig. 2, in the model of the present invention, one encoder and two decoders are combined with a regularizer. The encoder takes the source motion as input and maps it into the latent motion space. The normalizer may normalize the encoded motion distribution to approximate some preset prior distribution. The two decoders can each be designed to map the latent motion space to joint angles and joint velocities.

encoder

는 순차적으로 GRU에 입력되고, GRU는 현재 프레임을 암호화하며, 이전 프레임에 의해 히든(hidden) 표현으로 조건화된다. 구체적으로는 i번째 프레임의 포즈 q_i는 아래 <수학식 1>과 같이 인코딩될 수 있다.As shown in FIG. 2 , the encoder includes a plurality of Gated Recurrent Units (GRUs) and one linear layer. βt pose in motion

are sequentially input to the GRU, the GRU encrypts the current frame, and is conditioned with a hidden representation by the previous frame. _{Specifically, the pose q i} of the i-th frame may be encoded as shown in Equation 1 below.

[Equation 1]

는 GRU의 히든 차원이 d_h인 트레이닝 파라미터를 의미할 수 있다.Here, h _i means a hidden state in frame I,

may mean a training parameter in which the hidden dimension of the GRU is d _h.

의 선형 층은 ht+βt-1을 받고 압축하여 d_m 차원 코드

를 생성한다. 여기서,

는 모션 매니폴드를 의미할 수 있다. 이 때, 상기 압축은 입력 데이터를 디노이징(denoising)하는 것의 이점을 가져올 수 있다. 이러한 과정을 통해 인코더 매핑 Enc:

가 완료될 수 있다.After reading the last pose of the input motion,

The linear layer of d receives and compresses ht+βt-1 to d _m dimensional code.

create here,

may mean a motion manifold. In this case, the compression may bring the advantage of denoising the input data. Through this process, encoder mapping Enc:

can be completed.

Wasserstein regularizer potential using motion manifold

와 어울리는 미리 설정된 사전 분포 P_Z에 일치시키기 위해 미리 설정된 정규화기 예를 들어, Wasserstein regularizer를 사용할 수 있다. 이 때, Wasserstein regularizer는 기존 연구(TOLSTIKHIN I., BOUSQUET O., GELLY S., SCHOELKOPFB.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017).)에 자세히 기재되어 있으며, 본 발명의 기술 분야에 종사하는 당업자에게 있어서 자명하기에 그 상세한 설명은 생략한다. VAE와 달리, Wasserstein regularizer로 트레이닝된 순차적 네트워크는 비 랜덤 인코더가 입력을 잠재 코드에 결정적으로 매핑할 수 있도록 한다. 따라서, 모션 매니폴드에서 무작위 샘플링되거나 보간된 지점들이 그럴듯한 모션에 대응되도록 할 수 있다. The present invention relates to the distribution of motion manifolds.

We can use a preset regularizer, for example, the Wasserstein regularizer, to match to a preset prior distribution P _{Z that matches with .} At this time, the Wasserstein regularizer has been described in detail in previous studies (TOLSTIKHIN I., BOUSQUET O., GELLY S., SCHOELKOPFB.: Wasserstein auto-encoders. arXiv preprint arXiv:1711.01558 (2017).), and the technical field of the present invention Since it is obvious to those skilled in the art, a detailed description thereof will be omitted. Unlike VAE, sequential networks trained with Wasserstein regularizers allow non-random encoders to deterministically map inputs to latent codes. Thus, randomly sampled or interpolated points in the motion manifold can be made to correspond to plausible motion.

Decoder with joint rotation and joint velocity

In the present invention, the decoder model is composed of two types as shown in FIG. 2 . One decoder learns the joint rotation and the other decoder learns the joint rotation speed. The two decoders are based on the GRU, while the two decoders have different connection structures. Unlike the joint rotation decoder, the joint rotation speed decoder generates joint rotation by adding a residual connection between the input and output. Then, each decoder generates a reconstructed joint angle sequence in reverse temporal order. At this time, the decoder may be trained simultaneously with backpropagation.

In the present invention, the limitations of individual decoder models can be reduced by combining two decoders. Since the joint rotation decoder learns the joint angle itself, it has strength when reconstructing long-term motion. Conversely, pause discontinuity may occur between frames. Joint rotation speed decoders have the advantage of reconstructing continuous human motion because they usually output the difference between consecutive rotations that are small and easy to learn. However, the training rate tends to be unstable in long sequences because the longer the motion the more errors accumulate. Since the two decoders of the present invention have opposite advantages and disadvantages, they complement each other with a synergistic effect when combined.

Unlike the previous study of the motion prediction, the recognition and the manifold is used only rotating joints or articulated position, the model of the present invention contemplates both the joint rotations and position of the _R L motion reconstruction lost item (term). Joint angle loss has the advantage of preventing errors such as inconsistent bone lengths or deviations from the human range of motion, so learning from joint angle loss can produce plausible motion. However, rotation prediction can be paired with a loss of averaging the error over the joint by giving each joint the same weight. Ignorance of the various effects of different joints on the reconstructed motion can lead to large errors in critical joints and degrade the quality of the generated poses.

The joint position loss minimizes the average position error for 3D points, which better reflects the perceptual differences between poses. In order to combine both joint rotation and position in the motion reconstruction loss L _R , we can add a forward kinematics (FK) layer that calculates the joint position from the joint rotation. Through this, the loss between the joint position of the target motion and the reconstruction motion can be calculated. The anterior kinematics module is effective for network training because the output is different in relation to joint rotation.

Finally, the method of the present invention reconstructs the motion in the reverse order of the input sequence. Reversing the target sequence is advantageous for learning in that the first output frame of the decoder only needs to match the last frame input of the encoder, which allows for continuous conversion of hidden space vectors from encoder to decoder.

의 선형 층을 가진 d_h 차원의 히든 공간 벡터로 변환한다. 그런 다음 미래의 프레임을 나타내는 히든 공간 벡터에 의해 조건화되며, GRU와 선형 층은 다음 포즈

이 주어지면 i번째 프레임에서 재구성된 포즈

를 출력한다. 이를 수학식으로 나타내면 아래 <수학식 2>와 <수학식 3>으로 나타낼 수 있다.Joint rotation decoder : The configuration of a detailed embodiment of the joint rotation decoder is the same ^{as Dec r shown in FIG. 2 .} The joint rotation decoder first parameterizes the elements of the motion manifold z ∈ Z.

Convert it to a hidden space vector of dimension _{d h} with a linear layer of . Then it is conditioned by the hidden space vector representing the future frame, the GRU and the linear layer are the next poses

Given this, the pose reconstructed from the i-th frame

to output This can be expressed as <Equation 2> and <Equation 3> below.

[Equation 2]

[Equation 3]

는 GRU의 학습 파라미터를 의미하고

는 선형 층의 파라미터를 의미할 수 있다. here,

is the learning parameter of the GRU,

may mean a parameter of a linear layer.

의 순서로 수행된다. 인코더와 달리 디코더는 이전 프레임의 재구성된 결과를 입력으로 사용하고, 이는 장기 재구성을 위한 파라미터 튜닝이 없는 노이즈 스케줄링과 동일하며, 과적합(overfitting) 방지에 도움이 된다. 디코더는 모션 매니폴드 벡터의 메모리를 유지하기 위해 z의 추가 입력을 수신한다. 따라서, 디코더에 대한 입력은

이다. GRU에 대한 초기 입력

는 이전 프레임의 재구성 결과가 없기 때문에 0으로 설정된다. 재구성된 관절 회전은 타겟 모션에 대하여 각도 손실을 계산하는 데 사용되며, 아래 <수학식 4>에 나타낸 바와 같이 전방 운동학 층을 통과하는 위치

를 계산하는 데도 사용된다.As described above, since the joint rotation decoder uses the inverted input motion as the target motion, the reconstruction is

are performed in the order of Unlike the encoder, the decoder uses the reconstructed result of the previous frame as input, which is equivalent to noise scheduling without parameter tuning for long-term reconstruction, which helps to prevent overfitting. The decoder receives an additional input of z to maintain a memory of the motion manifold vectors. Thus, the input to the decoder is

am. Initial input to GRU

is set to 0 because there is no reconstruction result of the previous frame. The reconstructed joint rotation is used to calculate the angular loss with respect to the target motion, and the position passing through the anterior kinematics layer as shown in Equation 4 below.

It is also used to calculate

[Equation 4]

가 생성된 후에 관절 디코더 매핑 Dec^r:

이 완료된다.last pose

Mapping the joint decoder after Dec ^r is created:

This is done.

를 생성하기 위한 잔여 연결이 있다는 것이다. 이를 수학식으로 나타내면 아래 <수학식 5> 내지 <수학식 7>로 나타낼 수 있다.Joint velocity decoder: The configuration of a detailed embodiment of the ^{joint velocity decoder is the same as Dec v} shown in FIG. 2 , and the joint velocity decoder has a structure similar to that of the joint rotation decoder. The main difference is

that there are residual connections to create . If this is expressed as an equation, it can be expressed as <Equation 5> to <Equation 7> below.

[Equation 5]

[Equation 6]

[Equation 7]

및

는 GRU의 학습 파라미터와 선형 층의 파라미터를 의미할 수 있다.here,

and

may mean a learning parameter of the GRU and a parameter of a linear layer.

와 이전 프레임 포즈

사이의 차이를 학습한다. 따라서, 모델은 각도 차이나 속도를 예측하고 이를 시간에 따라 통합한다. 마지막 포즈가 생성되면 관절 속도 디코더 매핑 Dec^v:

가 완료된다.Residual network poses the current frame

with the previous frame pose

learn the difference between Thus, the model predicts angular differences or velocities and integrates them over time. When the last pose is created, the joint velocity decoder mapping Dec ^v :

is completed

motion manifold training

The present invention models a number of loss functions, each contributing to improving the quality of motion occurring in a motion manifold from different perspectives. In order to reduce the reconstruction loss, the present invention provides two types of loss functions, namely, a motion reconstruction loss L _R that reconstructs motion through an encoder and a decoder, and a manifold reconstruction loss L that helps reconstruct a latent vector through a decoder and an encoder. _M can be used. The present invention also includes _{a Wasserstein loss L W} that punishes the discrepancy between _{P Z and the distribution E Z} induced by the encoder, _{and an adversarial loss L G} to obtain a more natural motion from the motion manifold. Figure 3 shows an exemplary diagram to show an overview of the loss function of the present invention, where each loss item can be evaluated on data processed in the network pipeline indicated by the black arrow, and the red arrow is the individual loss item data used for

Motion reconstruction loss : Motion reconstruction loss punishes the difference between motion and reconstructed motion, and the reconstructed motion is obtained by encoding and then decoding the motion. Specifically, the present invention measures the mismatch between the joint rotation angle q and the joint position p, and the motion reconstruction loss can be expressed as in Equation 8 below.

[Equation 8]

는 유클리드 놈(Euclidean norm)을 의미하고, w_p는 위치 오류의 가중치를 의미할 수 있다. 예를 들어, 위치 오류의 가중치는 5일 수 있다.here,

may mean a Euclidean norm, and w _p may mean a weight of a position error. For example, the weight of the position error may be 5.

및

로 재구성한다. 여기서,

일 수 있다. 매니폴드 재구성 손실은 아래 <수학식 9>와 같이 나타낼 수 있다.Manifold reconstruction loss : Latent codes sampled from a latent distribution must be reconstructed after decoding and encoding. The manifold reconstruction loss encourages a reciprocal mapping between motion and manifold space. For this purpose, existing studies (LEE H.-Y., TSENG H.-Y., HUANG J.-B., SINGH M., YANG M.-H.: Diverse image-to-image translation via disentangled representations. In Proceedings L1 loss similar to that of the European Conference on Computer Vision (ECCV) (2018), pp. 35-51.) can be applied. Pulling a motion manifold vector Z from the encoded motion sequence

and

reconstruct with here,

can be The manifold reconstruction loss can be expressed as in Equation 9 below.

[Equation 9]

Wasserstein Normalization Loss: To create a specific distribution desired in the manifold space so that the present invention can efficiently sample the distribution, to punish the deviation of the latent manifold distribution E _{Z from the desired prior distribution, e.g., a preset prior distribution P Z , or} You can use the Wasserstein regularizer to solve it. In this case, the normalization loss can be expressed as in Equation 10 below.

[Equation 10]

는 유효성 검사를 통해

가 결정되는 다변수 정규 분포로 모델링될 수 있다.here,

through validation

It can be modeled as a multivariate normal distribution in which is determined.

에서의 역 다중 2차 방정식 커널

로 두 분포 사이의 차이를 측정할 수 있다. 본 발명은

과 모션 매니폴드 공간 Z_dim = 64의 차원을 설정할 수 있다.The present invention uses the maximum mean discrepancy MMD _k

Inverse Multiple Quadratic Equation Kernel in

can measure the difference between the two distributions. the present invention

and the dimension of motion manifold space Z _dim = 64 can be set.

Adversarial loss: Finally, the present invention may use a least squares generative adversarial network (LSGAN) to match the distribution of the generated motion with the distribution of the actual motion data. That is, the present invention can make it impossible to distinguish between the motion generated by the model and the actual motion by using the least squares generation adversarial network. The discriminator loss can be expressed as Equation 11 below, and the hostile loss can be expressed as Equation 12 below.

[Equation 11]

[Equation 12]

Here, the discriminator D can distinguish the reconstructed motion from the real motion, and this discriminator can be used to help the decoder of the present invention generate realistic motion.

Total Loss: The present invention jointly trains an encoder, joint rotation decoder, joint velocity decoder and discriminator to optimize the overall objective function, which is the weighted sum of reconstruction loss, Wasserstein regularization loss and adversarial loss. The overall objective function of the manifold network can be expressed as Equation 13 below, and the discriminator loss can be expressed as Equation 14 below.

[Equation 13]

[Equation 14]

_M,

_W,

_G는 유효성 검사를 통해 결정될 수 있으며, 예를 들어, 0.0001, 0.1, 0.001로 결정될 수 있다.Here, the weight parameter

_M ,

_W ,

_G may be determined through validation, for example, may be determined as 0.0001, 0.1, 0.001.

Data preprocessing

The method of the present invention can be tested with a preset data set, for example, the H3.6M data set, and all motions of the data set have the same skeletal structure. All poses are transmitted along with the position and orientation of the root and joint rotation expressed in exponential coordinates. For training, 150 frames of motion clips are randomly selected from the input motion sequence and used to train the motion manifold. For better performance, the root position of the cross section can be removed and other data can be normalized.

H3.6M: Data set H3.6M includes 15 activities, such as walking, smoking, discussing, taking pictures, making phone calls, etc. performed by 7 subjects. In the present invention, by removing duplicate joints, 32 joints of the original data are reduced to 17 joints, and all data can be configured to have a frame rate of 25 Hz. Therefore, 150 frames of motion applied to the model of the present invention is 6 seconds. Here, the activity of any one subject may be used as test data, and the activity of the other subjects may be used as training data.

When the motion manifold is configured in the present invention through the above-described process, motion input data is input to the motion manifold, and a target motion corresponding to the input motion input data is generated.

For example, as shown in FIG. 4 , when two types of motion input data, motion input data for holding and walking and motion input data for walking, are input as inputs of the motion manifold, the motion manifold determines the difference between the two motion input data. It is possible to generate and provide a motion of standing and holding a pose corresponding to , as a target motion. In addition, when two types of motion input data, sitting motion input data and standing motion input data, are input as input to the motion manifold, the motion manifold combines the two motion input data to target the sitting motion while holding a pose. It can be created and provided with motion.

As another example, as shown in FIG. 5 , when two types of motion input data, that is, sitting and smoking motion input data and sitting motion input data, are input as input to the motion manifold, the motion manifold determines the difference between the two motion input data. It is possible to generate and provide a motion corresponding to a standing cigarette as a target motion. In addition, when two types of motion input data, walking motion input data and standing smoking motion input data, are input as input to the motion manifold, the motion manifold combines the two motion input data to target the motion of smoking while walking. can be created and provided.

Furthermore, the motion manifold of the present invention may generate and provide a target motion by removing noise included in the motion input data when motion input data including noise is input, and the motion input data is not received and random When input information for generating motion is received, a target motion may be randomly generated and provided.

Furthermore, the method of the present invention is not limited to the neural network including the GRU and the linear layer described in FIG. 2, and the neural network constituting the encoder and each decoder can perform all types of neural networks capable of performing the method of the present invention. A network structure can be used.

As such, the method according to embodiments of the present invention may configure a motion manifold using a sequential network, and may generate motions of various people through the motion manifold.

In addition, since the method according to the embodiments of the present invention builds a motion manifold by learning limited motion data through a sequential deep learning network, noise removal and interpolation of data using such a motion manifold It can be applied to various applications such as new motion regeneration, motion recognition and categorization. In addition, since the motion manifold or motion reduction space in the present invention regenerates dynamic and various new motions, considerable cost and time can be saved when applied to computer graphics and animation fields.

In addition, in the present invention, a technique for constructing a human motion reduction space, which is a motion manifold, is essential in the field of computer graphics or animation. Even a professional animator needs considerable time and money to create a new character's motion. Also, motion data generation using motion capture has spatial difficulties and equipment difficulties. In contrast, data-driven motion synthesis and regeneration approaches based on machine learning dramatically reduce time and cost. The present invention can generate new motion data through limited motion data based on deep learning.

Recognizing and categorizing human motion is an essential part of computer vision or autonomous driving techniques. This makes it possible to analyze a person's behavior or predict the next one. The present invention enables categorization in near real time after the motion of a recognized person is projected into a reduced motion space. It also makes it easier to analyze human motion in a reduced space.

Since human motion data basically includes both spatial information between joints and spatial information that moves according to time, it occupies a significant size. As the motion time increases, the corresponding data capacity increases dramatically. Since the present invention reduces the dimension of motion data, it can also be used as a meaningful compression technique.

That is, the present invention is a technology for constructing a motion reduction space (motion manifold) through limited motion data based on deep learning and using it to generate new motion data. Motion data reduction, motion recognition, motion prediction, motion It can be used as one of the key core technologies in computer graphics and animation and vision fields such as compositing and motion categorization.

6 is a conceptual block diagram of an apparatus for performing the method of FIGS. 1 to 5 showing a conceptual configuration of a human motion generating apparatus according to an embodiment of the present invention.

Referring to FIG. 6 , an apparatus 600 for generating a human motion according to an embodiment of the present invention includes a configuration unit 610 and a generation unit 620 .

The configuration unit 610 configures a motion manifold using a sequential network.

Here, the configuration unit 610 may configure the motion manifold using a sequential network including one encoder and two decoders, and a learning model of the sequential network so that the distribution of the motion manifold matches the preset distribution. By learning , it is possible to construct a motion manifold based on a learning model.

Furthermore, in learning the learning model of the sequential network, the configuration unit 610 calculates the joint position using a forward kinematics technique, and further reflects the calculated joint position to learn the learning model. , it is also possible to configure a motion manifold based on a learning model. Of course, the learning model of the sequential network may be learned through preset training data based on at least one or more preset loss functions.

The generator 620 generates a target motion corresponding to the motion input data by using the motion manifold.

At this time, the generator 620 outputs joint rotation angle information for the target motion from any one of the two decoders constituting the motion manifold, and joint rotation speed information for the target motion from the other decoder. By outputting , a target motion corresponding to the motion input data may be generated based on the joint rotation angle information and the joint rotation speed information.

Although the description of the device of FIG. 6 is omitted, each component constituting FIG. 6 may include all the contents described with reference to FIGS. 1 to 5 , which will be apparent to those skilled in the art.

The system or apparatus described above may be implemented as a hardware component, a software component, and/or a combination of the hardware component and the software component. For example, the systems, devices, and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA). ), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions, may be implemented using one or more general purpose or special purpose computers. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For convenience of understanding, although one processing device is sometimes described as being used, one of ordinary skill in the art will recognize that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that can include For example, the processing device may include a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as parallel processors.

The software may comprise a computer program, code, instructions, or a combination of one or more thereof, which configures a processing device to operate as desired or is independently or collectively processed You can command the device. The software and/or data may be any kind of machine, component, physical device, virtual equipment, computer storage medium or device, to be interpreted by or to provide instructions or data to the processing device. , or may be permanently or temporarily embody in a transmitted signal wave. The software may be distributed over networked computer systems, and stored or executed in a distributed manner. Software and data may be stored in one or more computer-readable recording media.

The method according to the embodiments may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be specially designed and configured for the embodiment, or may be known and available to those skilled in the art of computer software. Examples of the computer-readable recording medium include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic such as floppy disks. - includes magneto-optical media, and hardware devices specially configured to store and carry out program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine language codes such as those generated by a compiler, but also high-level language codes that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

As described above, although the embodiments have been described with reference to the limited embodiments and drawings, various modifications and variations are possible from the above description by those skilled in the art. For example, the described techniques are performed in a different order than the described method, and/or the described components of the system, structure, apparatus, circuit, etc. are combined or combined in a different form than the described method, or other components Or substituted or substituted by equivalents may achieve an appropriate result.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (12)

constructing a motion manifold using a sequential network; and
generating a target motion corresponding to motion input data using the configured motion manifold
including,
The configuring step is
Construct the motion manifold using a sequential network including one encoder and two decoders,
The generating step is
One decoder of the two decoders outputs joint rotation angle information for the target motion, and the other decoder outputs joint rotation speed information for the target motion, the joint rotation angle information and the joint Human motion generating method, characterized in that for generating the target motion based on the rotation speed information.
delete delete According to claim 1,
The configuring step is
A method for generating human motion, characterized in that by learning a learning model of the sequential network so that the distribution of the motion manifold matches a preset distribution, a motion manifold based on the learning model is configured.
According to claim 1,
The configuring step is
In learning the learning model of the sequential network, by calculating the position of the joint using a forward kinematics technique, and learning the learning model by additionally reflecting the calculated position of the joint, the learning model is based A method for generating human motion, comprising configuring a motion manifold of
delete delete a configuration unit for configuring a motion manifold using a sequential network; and
A generator for generating a target motion corresponding to the motion input data using the configured motion manifold
including,
the component part
Construct the motion manifold using a sequential network including one encoder and two decoders,
the generating unit
One decoder of the two decoders outputs joint rotation angle information for the target motion, and the other decoder outputs joint rotation speed information for the target motion, the joint rotation angle information and the joint Human motion generating apparatus, characterized in that for generating the target motion based on the rotation speed information.
delete delete 9. The method of claim 8,
the component part
The human motion generating apparatus, characterized in that by learning the learning model of the sequential network so that the distribution of the motion manifold matches a preset distribution, a motion manifold based on the learning model is configured.
9. The method of claim 8,
the component part
In learning the learning model of the sequential network, by calculating the position of the joint using a forward kinematics technique, and learning the learning model by additionally reflecting the calculated position of the joint, the learning model is based Human motion generating device, characterized in that constituting the motion manifold of.

Images