KR20210070874A - 3d human body model reconstruction apparatus and method - Google Patents

Abstract

The present invention provides an apparatus and method for restoring a three-dimensional (3D) human model, which comprises: a two-dimensional (2D) posture feature extraction unit for estimating a 2D joint vector corresponding to each joint position of at least one person included in a 2D image by a previously learned pattern estimation method and normalizing the estimated 2D joint vector in a predetermined mode to acquire features of a 2D posture; a multi-step human model estimation unit for receiving 2D posture features for each of the at least one person included in the 2D image, estimating a body type and a posture vector designated by a human model template in order to restore a human model corresponding to the 2D posture feature by a previously learned pattern estimation method, and estimating a location vector in order to dispose the human model being restored at a location corresponding to the 2D posture feature in the 3D space; and a 3D space disposition unit for receiving a status vector composed of a body type vector, a posture vector and a location vector and a 2D posture feature to restore a human model corresponding to the body type vector and the posture vector and disposing the human model restored according to the location vector in 3D space.

Description

3D human model restoration apparatus and method {3D HUMAN BODY MODEL RECONSTRUCTION APPARATUS AND METHOD}

The present invention relates to an apparatus and method for restoring a three-dimensional human model, and to a plurality of apparatus and methods for restoring a three-dimensional human model using a two-dimensional posture and a human template model.

Techniques for extracting a person's posture from an image have recently been developed and are being used in various fields, such as security and medical care, as well as behavior recognition, animation, and existing applications such as virtual reality or augmented reality. In particular, the technique of extracting a three-dimensional posture from a two-dimensional image can analyze even the spatial movement of a person because the coordinates of each joint are expressed in a three-dimensional space.

However, when a person's posture is simply restored by the joint position, detailed movements such as arm rotation cannot be restored, and information such as a person's body type cannot be known. If such information is not obtained, it is difficult to project the movement of a person onto the avatar in the virtual space. And in a situation in which the development of technology is gradually oriented toward the creation of contents in the augmented reality/virtual reality space, the need to restore a human model in a three-dimensional space so that it can express the movement of a person in detail beyond simple human posture extraction have.

In addition, the previously proposed 3D human model restoration methods restore only one model when one image is input, and do not provide distance information about how far away a person is from the camera that captured the image. That is, only the posture of the person is restored, but the position of the person in the three-dimensional space is not restored. Therefore, when several people are included in one image, there is a limitation that not only cannot reconstruct a 3D human model for all people, but also the distance relationship between people cannot be known.

Korean Patent Publication No. 10-2018-0097949 (published on September 3, 2018)

It is an object of the present invention to provide a 3D human model restoration apparatus and method capable of accurately reconstructing a 3D human model for a plurality of people in which positions, rotations, postures, and body types in 3D are reflected from a 2D image.

Another object of the present invention is to provide a 3D human model restoration apparatus and method capable of obtaining a 3D human model capable of various applications in a 3D virtual space regardless of the background of the 2D image.

A 3D human model restoration apparatus according to an embodiment of the present invention for achieving the above object is a 2D joint vector corresponding to each joint position of at least one person included in a 2D image according to a pre-learned pattern estimation method. a two-dimensional posture feature extractor for estimating and normalizing the estimated two-dimensional joint vector in a predetermined manner to obtain two-dimensional posture features; In order to receive the two-dimensional posture characteristic for each of the at least one person included in the two-dimensional image, and to reconstruct a human model of a body type and posture corresponding to the two-dimensional posture characteristic according to a pre-learned pattern estimation method, the human a multi-stage human model estimator for estimating a body shape vector and a posture vector specified by a model template, and estimating a position vector for arranging the restored human model at a position corresponding to the two-dimensional posture feature on a three-dimensional space; and a state vector composed of the body shape vector, the posture vector, and the position vector and the two-dimensional posture feature are applied, and a human model corresponding to the body vector and the posture vector is restored, and the restored human model is restored according to the position vector. and a three-dimensional space arrangement unit for arranging the human model on a three-dimensional space.

The multi-stage human model estimator includes a plurality of model estimators sequentially connected in a multi-stage structure, and each of the plurality of model estimators receives and combines the two-dimensional posture characteristic and the state vector estimated by the model estimator of the previous stage to combine them into a combined state. The vector is generated, and the body shape vector and the posture vector specified by the human model template are used to reconstruct the human model of the body shape and posture corresponding to the two-dimensional posture feature according to the pattern estimation method learned in advance from the generated combined state vector. By estimating and estimating a position vector for arranging the restored human model at a position corresponding to the two-dimensional posture feature in the three-dimensional space, a new state vector including the body vector, the posture vector, and the position vector is generated. It can be passed to the model estimator.

Each of the plurality of model estimator includes a feature assembling unit for generating a combined state vector by combining the two-dimensional posture feature and the state vector estimated by the model estimator of the previous stage; According to the pattern estimation method learned in advance from the combined state vector, the body shape vector corresponding to the two-dimensional posture feature, the position vector corresponding to the position where the human model should be placed in the three-dimensional space, and the rotation angle of each of a plurality of human joints a position estimator for estimating a global rotation vector representing a rotation angle of the entire human model based on a predetermined root joint among posture vectors composed of a plurality of corresponding joint rotation vectors; and receiving the two-dimensional posture characteristic and the global rotation vector, and estimating the relative rotation angle of a joint far from an adjacent joint in the root joint based on the global rotation vector in a predetermined sequence to estimate the global rotation vector among the posture vectors It may include a model posture estimation unit for obtaining a body posture vector including the joint rotation vector of the remaining joints except for.

The position estimator may include: a body shape estimator trained in advance to estimate a body shape vector corresponding to the two-dimensional posture feature from the combined state vector; and a depth/rotation estimator that is learned independently of the body shape estimator and simultaneously estimates the position vector and the global rotation vector corresponding to the two-dimensional posture feature from the combined state vector.

The model posture estimator is provided in a number corresponding to the remaining joints except for the root joint among the plurality of joints of the human model, and applies the global rotation vector or the joint rotation vector obtained in the previous stage and the two-dimensional posture characteristic, respectively. It may include a plurality of joint rotation estimation units for estimating the corresponding joint rotation vector received.

The plurality of joint rotation estimating units are connected in parallel to each of the plurality of parts from the root joint in correspondence to the joint connection structure for each part of the human model, and are connected in series according to the joint connection structure of each part to each joint rotation vector can be estimated.

The two-dimensional posture feature extraction unit converts the two-dimensional joint vector to have a size and a position corresponding to a normalized image plane of a predetermined scale and resolution, and the position of the joint vector corresponding to the root joint in the converted two-dimensional joint vector is It can be moved to be placed at the origin of the normalized image plane.

The two-dimensional posture feature extractor may include a position of a joint vector corresponding to the root joint before moving to the origin of the normalized image plane in the two-dimensional posture feature.

The two-dimensional posture feature extracting unit estimates a two-dimensional skeletal vector corresponding to each skeletal direction of at least one person included in the two-dimensional image, and normalizes the estimated two-dimensional skeletal vector in a predetermined manner to obtain the two-dimensional posture Features may be further included.

), 위치 손실(

), 전역 회전 손실(

), 바디 손실(

)을 계산하고 대응하는 가중치(

)를 적용하며, 바디 회전 손실(

) 추가로 고려하여 총 손실을 수학식 The three-dimensional human model restoration apparatus loses body shape from the truth value of the state vector estimated by the multi-stage human model estimator from the two-dimensional learning image and the state vector labeled in the two-dimensional learning image.

), position loss (

), global rotation loss (

), body loss (

) and calculate the corresponding weights (

), and the body rotation loss (

), further taking into account the total loss in the formula

and a learning unit configured to train the multi-stage human model estimator by calculating and backpropagating the calculated total loss.

A 3D human model restoration method according to another embodiment of the present invention for achieving the above another object is a 2D joint corresponding to the joint position of each of at least one person included in the 2D image according to a pre-learned pattern estimation method. estimating the vector and normalizing the estimated two-dimensional joint vector in a predetermined manner to obtain a two-dimensional posture characteristic; In order to receive the two-dimensional posture characteristic for each of the at least one person included in the two-dimensional image, and to reconstruct a human model of a body type and posture corresponding to the two-dimensional posture characteristic according to a pre-learned pattern estimation method, the human obtaining a state vector by estimating a body shape vector and a posture vector specified by a model template, and estimating a position vector for placing the reconstructed human model at a position corresponding to the two-dimensional posture feature on a three-dimensional space; and a state vector composed of the body shape vector, the posture vector, and the position vector and the two-dimensional posture feature are applied, and a human model corresponding to the body vector and the posture vector is restored, and the restored human model is restored according to the position vector. placing the human model on a three-dimensional sphere.

Therefore, the apparatus and method for restoring a 3D human model according to an embodiment of the present invention obtains a human model by estimating the 3D position, posture, and body type of each person from a 2D image including at least one person, and By allowing the human model to be placed at an exact location on various three-dimensional virtual spaces, it is possible to reconstruct highly realistic human models in various applications such as augmented reality or virtual reality. In addition, since it is possible to reconstruct a location-based three-dimensional human model, it facilitates interaction between multiple people and recognition of group behavior.

1 is a diagram for explaining the concept of reconstructing a human model in a 3D virtual space from a 2D image by an apparatus for reconstructing a 3D human model according to an embodiment of the present invention.
2 shows a schematic structure of an apparatus for reconstructing a 3D human model according to an embodiment of the present invention.
FIG. 3 shows an example of the configuration of the two-dimensional posture feature extraction unit of FIG. 2 .
4 is a diagram for explaining the body type and posture parameters of the SMPL model.
5 is a diagram for explaining the concept of normalizing a two-dimensional joint position to a two-dimensional normalized image plane.
6 is a schematic configuration of the multi-stage human model estimator of FIG. 2 .
FIG. 7 shows a schematic configuration of each of a plurality of model estimators of FIG. 6 .
8 shows an example of a detailed configuration of each of the model estimator of FIG. 7 .
9 shows another example of the detailed configuration of the model posture estimation unit of FIG. 7 .
10 shows a 3D human model reconstruction method according to an embodiment of the present invention.
11 shows an example of a human model reconstructed in a 3D space from a 2D image using the 3D human model reconstruction method of the present invention.

In order to fully understand the present invention, the operational advantages of the present invention, and the objects achieved by the practice of the present invention, reference should be made to the accompanying drawings illustrating preferred embodiments of the present invention and the contents described in the accompanying drawings.

Hereinafter, the present invention will be described in detail by describing preferred embodiments of the present invention with reference to the accompanying drawings. However, the present invention may be embodied in various different forms, and is not limited to the described embodiments. In addition, in order to clearly describe the present invention, parts irrelevant to the description are omitted, and the same reference numerals in the drawings indicate the same members.

Throughout the specification, when a part "includes" a certain component, it means that other components may be further included, rather than excluding other components unless otherwise stated. In addition, terms such as "... unit", "... group", "module", and "block" described in the specification mean a unit that processes at least one function or operation, which is hardware, software, or hardware. and a combination of software.

1 is a diagram for explaining the concept of reconstructing a human model in a 3D virtual space from a 2D image by an apparatus for reconstructing a 3D human model according to an embodiment of the present invention.

Referring to FIG. 1 , the apparatus for reconstructing a 3D human model according to the present embodiment determines an object area for each of at least one person included in the 2D image as shown in (a) when a 2D image is applied. and estimate the two-dimensional joint position of a person in each object area as shown in (b). Then, the estimated two-dimensional joint position for each person is converted into coordinates on a predetermined normalized image plane as shown in (c), and a two-dimensional posture characteristic is obtained based on the converted coordinates. Based on the obtained two-dimensional posture characteristics, a three-dimensional human body model expressing the posture and body type of each person is estimated and obtained, and the obtained human body model is placed at the estimated position in the three-dimensional space as shown in (d). place it

2 shows a schematic structure of an apparatus for reconstructing a 3D human model according to an embodiment of the present invention.

Referring to FIG. 2 , the apparatus for restoring a 3D human model according to the present embodiment includes an image acquisition unit 100 , a 2D posture feature extraction unit 200 , a multi-stage human model estimator 300 , and a 3D spatial arrangement unit ( 400) may be included.

Referring to FIG. 2 , each configuration of the apparatus for reconstructing a 3D human model according to the present embodiment will be described. First, the image acquisition unit 100 is a 2D model including at least one person to be modeled as a 3D model. acquire an image

The two-dimensional posture feature extraction unit 200 estimates and estimates a posture in two dimensions for each of at least one person included in the two-dimensional image obtained by the image acquisition unit 100 according to a pre-learned pattern estimation method. Two-dimensional posture features are extracted from the obtained posture. Here, the two-dimensional posture feature extraction unit 200 estimates the joint position of each person included in the two-dimensional image, and normalizes the estimated joint position, as shown in FIG. 2D posture features can be extracted.

The multi-stage human model estimator 300 estimates the three-dimensional posture and position of each person from the two-dimensional posture estimated by the two-dimensional posture feature extraction unit 200 . In the present embodiment, the multi-stage human model estimator 300 as shown in FIG. 1C , estimates the body shape along with the three-dimensional posture of each of at least one person, as well as a camera that captures a two-dimensional image. It is possible to estimate the position of each person in 3D together based on . That is, the external appearance can be estimated along with the posture, body type, and position of each person in the three-dimensional space.

The 3D space arrangement unit 400 arranges the human model according to the posture and body shape of each person estimated by the multi-stage human model estimator 300 at the estimated position in the 3D virtual space. Accordingly, as shown in FIG. 3D , the relative position, posture, and appearance between each of at least one person included in the two-dimensional image may be restored in the three-dimensional virtual space. That is, the situation of the scene included in the 2D image can be converted into the situation of the 3D virtual space and expressed. In this case, since the human model can be arranged in various spaces regardless of the background in the two-dimensional image, a very realistic human model can be restored in an environment such as augmented reality or virtual reality.

Meanwhile, the 3D human model restoration apparatus according to the present embodiment may further include a learning unit 500 for learning the multi-stage human model estimator 300 . In the present embodiment, the multi-stage human model estimator 300 may be implemented as an artificial neural network for estimating a three-dimensional posture, body type, and position of a human from a two-dimensional posture. And in order for the artificial neural network to perform a required operation, a pattern estimation method must be learned in advance. Accordingly, the learning unit 500 receives a two-dimensional learning image labeled with truth values for posture, body type, and position into the image acquisition unit 100 and the three-dimensional posture of a human estimated by the multi-stage human model estimator 300 . The multi-stage human model estimator 300 by acquiring the error between the position and body shape and the three-dimensional posture and the position and body type labeled in the learning image as a loss and backpropagating the obtained loss to the multi-stage human model estimator 300 . can be learned

The 2D posture feature extracting unit 200 may also be implemented as an artificial neural network, and in this case, the learning unit 500 may be configured to also learn the 2D posture feature extracting unit 200 together. However, artificial neural networks for estimating the two-dimensional posture of each person included in the image from the two-dimensional image have already been variously studied and known. Therefore, here, it is assumed that the two-dimensional posture feature extraction unit 200 uses a conventionally known artificial neural network, and hereinafter, the learning unit 500 learns only the multi-stage human model estimator 300 .

Since the learning unit 500 is configured to learn the multi-stage human model estimator 300 as described above, it can be removed from the 3D human model restoration apparatus after the learning of the multi-stage human model estimator 300 is completed. have.

Figure 3 shows an example of the configuration of the two-dimensional posture feature extraction unit of Figure 2, Figure 4 is a view for explaining the body shape and posture parameters of the SMPL model, Figure 5 is a two-dimensional joint position on the two-dimensional normalized image plane It is a diagram for explaining the concept of normalization.

Referring to FIG. 3 , the two-dimensional posture feature extraction unit 200 includes an object determining unit 210 , a joint vector obtaining unit 220 , a skeletal vector obtaining unit 230 , a joint vector encoding unit 240 , and a skeletal vector encoding unit. It may include a unit 250 , a two-dimensional posture feature acquisition unit 260 , an initial state setting unit 270 , and a normalization unit 280 .

The object determining unit 210 determines each of the object regions in which at least one person is disposed according to a pattern estimation method previously learned from the two-dimensional image applied from the image obtaining unit 100 . The joint vector obtaining unit 220 obtains a joint vector indicating the joint position of a person according to a pattern estimation method learned in advance in each object area determined by the object determining unit 210 . The skeletal vector acquisition unit 230 acquires a skeletal vector indicating the position and direction of a human bone according to a pre-learned pattern estimation method. That is, the joint vector acquisition unit 220 acquires the position of a joint in the human skeleton as a joint vector, and similarly, the skeletal vector acquirer 230 acquires a skeletal vector according to the position and direction of the bone. The skeletal vector acquisition unit 230 may obtain a skeletal vector directly from the two-dimensional image independently of the joint vector acquisition unit 220, but since the skeleton is a structure between joints in the structure of the human body, the skeletal vector acquirer 230 may obtain 2 A skeletal vector may be obtained based on the joint vector obtained by the joint vector obtaining unit 220 rather than the dimensional image.

Here, the joint vector acquirer 220 and the skeletal vector acquirer 230 may obtain a joint vector and a skeletal vector from each of the object regions determined by the object determiner 210 .

The joint vector encoding unit 240 and the skeletal vector encoding unit 250 encode the joint vector obtained by the joint vector obtaining unit 220 and the skeletal vector obtained by the skeletal vector obtaining unit 230 in a predetermined format, respectively. The joint vector encoding unit 240 and the skeletal vector encoding unit 250 may convert, for example, a joint vector and a skeletal vector obtained in a coordinate value format into one-dimensional joint feature values and skeletal feature values.

In general, in order to reconstruct a 3D human model from a 2D image, only a joint vector indicating the position of a joint is obtained. However, the skeletal vector represents the relative distance between the joints, and when estimating the joint position in three dimensions from the two-dimensional joint position, it provides more detailed information on the relationship between the joints so that the three-dimensional joint position can be estimated more accurately. can do.

Accordingly, in this embodiment, the two-dimensional posture feature extraction unit 200 is provided with the skeletal vector acquisition unit 230 together with the joint vector acquisition unit 220 to acquire the joint vector and the skeletal vector together. However, in some cases, the skeleton vector obtaining unit 230 and the skeleton vector encoding unit 250 may be omitted.

The two-dimensional posture feature acquisition unit 260 normalizes the joint feature values and skeletal feature values converted by the joint vector encoding unit 240 and the skeletal vector encoding unit 250 to coordinates on a predetermined two-dimensional normalized image plane, By combining normalized joint feature values and skeletal feature values and extracting features, two-dimensional posture features representing human posture features of a two-dimensional image are obtained.

Since the joint vector obtaining unit 220 and the skeletal vector obtaining unit 230 can obtain a joint vector and a skeletal vector for each person in the object identification area by dividing it, the two-dimensional posture feature obtaining unit 260 also applies to each person. It can be obtained by distinguishing the two-dimensional posture characteristics for

Here, the two-dimensional posture feature acquisition unit 260 normalizes the joint vector and the skeletal vector not only to compensate for differences due to the scale and resolution of different two-dimensional images, but also to equalize the two-dimensional joint vector and/or Alternatively, it is because it is not easy to simultaneously estimate both the posture and the position of a person in a three-dimensional space from a skeleton vector. Although the multi-stage human model estimator 300 is implemented as an artificial neural network and trained, it is difficult to learn to estimate the posture and position of a person in a three-dimensional space from a two-dimensional joint vector. In most cases, it does not show performance.

This is because there are many variables to be estimated and each variable has a high degree of freedom, and this problem can be improved by limiting the degree of freedom by limiting some of the variables to fixed values. In particular, in this embodiment, the 2D posture feature acquisition unit 260 limits the size and resolution of the 2D image and changes according to the position of each person in the 2D image through normalization, thereby Allows the position to be accurately estimated.

In this embodiment, for human model restoration, a human model is restored based on a Skinned Multi-Person Linear (SMPL) model template that can define a human model using a body shape vector (β) and a posture vector (θ) as parameters. assume that

¹⁰)일 수 있다.In the SMPL model template, the body shape vector (β) can set a body shape representing a human appearance by deforming a mesh consisting of 6890 vertices according _{to 10 types of pre-specified body shape variables (β 1} to β _{10 ).} FIG. 4A shows changes in body shape defined by the first and second body shape variables (β ₁ , β _{2 ) among 10} types of body shape variables (β ₁ to β 10 ). In other words, the body shape vector (β) is a 10-dimensional vector (β ∈) that consists of _{10 variables (β 1} ~ β 10 ) and defines the human body shape.

¹⁰ ) can be.

On the other hand, a human joint can be viewed as a kinematic tree structure based on a skeletal connection structure as shown in FIG. 4(b) based on joint interdependency. Accordingly, the human posture may be expressed as a relative rotation angle of each joint sequentially connected in the direction of the end of the body based on a root joint, which is a main central joint, according to a connection structure. According to this kinematic tree structure, the positions of all human joints can be expressed as relative rotation angles from the root joint to the rest of the joints, and the posture vector θ represents the rotation angle of each joint including the root joint.

That is, if the remaining joints are expressed as relative rotation angles from the root joint according to the posture vector (θ) based on the kinematic tree structure, the position and rotation angle of the root joint are estimated, and the positions of the remaining joints are based on the root joint. It can be easily estimated as well as accurately estimated in the form from which the rotation angle is derived. Therefore, if the root joint is set as an anchor point and the relative positions of the remaining joints are estimated based on the root joint, which is the anchor point, the position of the joints is very effectively compared to the method of estimating the positions of all joints individually. can be estimated

^3K)일 수 있다. 여기서는 고관절을 루트 관절로 하여 관절의 개수가 24개인 SMPL 모델을 가정하며, 이에 자세 벡터(θ)는 72차원(3 * 24)의 매개 변수로 설정될 수 있다. 3 * K 차원의 자세 벡터(θ) 중 초기 3개 차원의 자세 벡터(θ₀) = [θ_0,1, θ_0,2, θ_0,3] ∈

³)는 루트 관절을 중심으로 하는 인간 모델 전체의 회전을 나타내는 전역 회전 벡터(θ₀)이다. 그리고 자세 벡터(θ)에서 루트 관절에 대한 전역 회전 벡터(θ₀)를 제외한 나머지 K-1개의 관절에 대한 자세 벡터(θ_J = [θ₁, θ₂, …, θ_K-1])는 루트 관절을 시점으로 연결된 각 관절들 사이의 상대적 회전각으로 각 관절의 위치를 나타내는 바디 회전 벡터라 할 수 있다.The posture vector (θ) is 3 * K dimension (θ = [θ _0,1 , θ _0,2 , θ _0,3 , θ _1,1 , …, θ _{K,1 ) when the number of joints to be estimated is K.} , θ _K,2 , θ _K,3 ] ∈

^3K ). Here, an SMPL model in which the number of joints is 24 is assumed using the hip joint as the root joint, and the posture vector (θ) can be set as a parameter of 72 dimensions (3 * 24). 3 * Among the K-dimensional posture vectors (θ), the initial three-dimensional posture vectors (θ ₀ ) = [θ _0,1 , θ _0,2 , θ _0,3 ] ∈

³ _{) is the global rotation vector (θ 0} ) representing the rotation of the entire human model around the root joint. _{And the posture vector (θ J} = [θ ₁ , θ ₂ , …, θ _K-1 ]) for the remaining K-1 joints except for the global rotation vector (θ _{0 ) for the root joint from the posture vector (θ) is} It can be called a body rotation vector indicating the position of each joint as a relative rotation angle between each joint connected to the root joint as a starting point.

Therefore, when the body shape vector (β) and the posture vector (θ) are obtained, a human model corresponding to the body shape vector (β) and the posture vector (θ) can be obtained from the SMPL model template.

However, the 3D human model restoration apparatus of the present embodiment needs to restore the 3D human model corresponding to each person to an accurate position in the 3D virtual space even when a plurality of people are included in the 2D image. Therefore, it is necessary to estimate the position in the 3D space together with the 3D human model. However, it is not easy to estimate the positions of a plurality of people disposed at different arbitrary positions in a two-dimensional image. In addition, although in the position vector (θ) estimated the global rotation vector (θ ₀₎ of the root joint, and to assume a rest position vector (θ _J) from the estimated global rotation vector (θ ₀₎ since even if relatively easy , the global rotation vector (θ ₀ ) is not constrained to the position of other joints, so the degree of freedom is relatively high.

Accordingly, the two-dimensional posture feature acquisition unit 260 is converted to a scale corresponding to the normalized image plane coordinates, and normalizes the two-dimensional joint vector so that the hip joint, which is the root joint, is disposed at the origin, thereby determining the position of the root joint in three dimensions in the distance direction. can be limited to estimating only from That is, the degree of freedom of the global rotation vector θ _{0 can be reduced.}

²))이 추정된 경우, N개의 2차원 관절(j)의 위치는 2차원 영상의 해상도에 따른 종속성을 제거하기 위해 정규화된 이미지 평면 상의 좌표(

)로 변환되어야 한다.In this embodiment, in order to prevent inconsistency when different two-dimensional images are input, it is assumed that all images were taken with the same camera having a focal length f and a sensor width w, and a specific position (J ₀ = The width (W) and height (H) of the 2D image where the SMPL model located at [x _t , y _t , z _{t ]) is taken is given, and N 2D joints (j = [} j ₀ , j ₁ , … j _N ], j _i ∈

² )) is estimated, the positions of the N two-dimensional joints (j) are coordinates on the normalized image plane (

) should be converted to

)로 변환하기 위해서는 우선 2차원 영상의 넓이(W)와 높이(H)의 비를 기지정된 비율로 조절한다. 일예로 넓이(W)와 높이(H)의 비를 1로 설정할 수 있으며, 넓이(W)가 높이(H)보다 큰 경우, 정규화된 이미지 평면의 y 좌표에 오프셋((W-H)/2)을 추가하고, 높이(H)가 넓이(W)보다 큰 경우 x 좌표에 오프셋((H-W)/2)을 추가할 수 있다. 그리고 좌표의 x와 y 좌표를 넓이(W)와 높이(H) 중 더 큰 값(max(W, H))으로 나누고, 정규화된 이미지 좌표의 원점으로 기준 위치를 맞추기 위해 더 넓이(W)와 높이(H) 중 더 큰 값의 절반(max(W, H)/2)의 절반으로 모든 좌표를 차감한다. 여기서 정규화된 이미지 평면의 초점 거리(f_n)는 카메라 초점 거리(f)를 센서 폭(w)으로 나누어 결정될 수 있다.coordinates on the normalized image plane (

), the ratio of the width (W) to the height (H) of the 2D image is first adjusted at a predetermined ratio. For example, the ratio of the width (W) to the height (H) can be set to 1, and when the width (W) is greater than the height (H), an offset ((WH)/2) is added to the y-coordinate of the normalized image plane. In addition, if the height (H) is greater than the width (W), you can add an offset ((HW)/2) to the x-coordinate. Then, divide the x and y coordinates of the coordinates by the larger value (max(W, H)) of the width (W) and the height (H). Subtract all coordinates by half (max(W, H)/2) of the greater of the height (H). Here, the focal length f _n of the normalized image plane may be determined by dividing the camera focal length f by the sensor width w.

³)에서 정규화 이미지 평면 좌표의 2차원 관절 벡터에 따라 3차원 공간으로 투영된 인간 모형의 위치(J₀ = [x_t, y_t, z_t])는 2차원 관절 벡터의 루트 관절의 위치가 정규화 이미지 평면 좌표의 원점(0, 0)으로 이동되면, 투영된 인간 모형의 위치(J₀ ^b = [0, 0, z_t])가 되며, 이는 루트 관절의 위치의 위치를 Z 축 방향의 거리(z_t)만으로 추정할 수 있음을 의미한다. 그리고 3차원 공간에 투영된 인간 모형의 원위치(J₀)는 정규화 이미지 평면 좌표에서 루트 관절 위치(

)와 정규화된 루트 관절의 위치(

= [0, 0]) 사이의 위치 차와 이에 대응하는 인간 모형의 위치(J₀, J₀ ^b) 차 사이의 비례 관계가 성립해야 서로 닮은 형상으로 나타나는 특성을 이용하여 추정될 수 있다.As shown in Fig. 5, the three-dimensional camera coordinate system (t ∈

³ ), the position of the human model projected into 3D space according to the 2D joint vector of the normalized image plane coordinates (J ₀ = [x _t , y _t , z _t ]) is the position of the root joint of the 2D joint vector When it is moved to the origin (0, 0) of the normalized image plane coordinates, it becomes the position of the projected human model (J ₀ ^b = [0, 0, z _t ]), which changes the position of the root joint position in the Z-axis direction. This means that only the distance (z _t ) can be estimated. _{And the original position (J 0} ) of the human model projected in the three-dimensional space is the root joint position (J 0 ) in the normalized image plane coordinates.

) and the position of the normalized root joint (

= [0, 0]) and the corresponding human model position (J ₀ , J ₀ ^b ) difference must be established in order to be estimated using the characteristics appearing similar to each other.

)와 정규화된 2차원 루트 관절 위치(

)가 확인되고, 위치 벡터(Z_t)가 추정되면, 3차원 공간 상에서의 인간 모형의 루트 관절의 위치(J₀)는 수학식 1에 따라 용이하게 추정될 수 있음을 알 수 있다.That is, the root joint position (

) and the normalized two-dimensional root joint position (

) is confirmed and the position vector (Z _{t ) is estimated, it can be seen that the position (J 0} ) of the root joint of the human model in the three-dimensional space can be easily estimated according to Equation (1).

(Where │ is the L2-norm function.)

)를 2차원 자세 특징에 포함하여 다단 인간 모델 추정부(300)로 전달할 수 있다.However, in order to re-convert the converted position due to normalization, the 2D posture feature acquisition unit 260 determines the root joint position (

) may be included in the two-dimensional posture feature and transmitted to the multi-stage human model estimator 300 .

6 shows a schematic configuration of the multi-stage human model estimator of FIG. 1 , FIG. 7 shows a schematic configuration of each of a plurality of model estimators of FIG. 6 , and FIG. 8 is an example of a detailed configuration of each model estimator of FIG. 7 indicates And FIG. 9 shows another example of the detailed configuration of the model posture estimation unit of FIG. 7 .

As shown in FIG. 6 , the multi-stage human model estimator 300 includes a plurality of model estimators 3001 , 3002 , ... 300S connected in series. The plurality of model estimators 3001, 3002, ... 300S _{receive feature authorizations for the state vectors β, θ, Z t} obtained in the previous stage and the two-dimensional posture obtained from the two-dimensional posture feature acquisition unit 260, respectively. , a combined state vector is obtained by combining the applied state vector with the two-dimensional posture, and extracting the feature, and a new state vector is estimated from the combined state vector.

Here, the first model estimator 3001 among the plurality of model estimators 3001, 3002, ... 300S has an initial body shape (β ⁰ ) and an initial posture (θ ⁰ ) set separately in the distance direction along with the initial position (Z _t ^{0 ).} ) can be authorized as a state vector or stored in advance. In addition, the initial position (Z _t ⁰ ) in the distance direction along with the initial body shape (β ⁰ ) and the initial posture (θ ⁰ ) may be set to an arbitrary value.

7 illustrates an s-th model estimator 300s as an example of a plurality of model estimators 3001, 3002, ... 300S having the same structure. 7 and 8 , each of the plurality of model estimators may include a feature aggregator 310 , a position estimator 320 , and a model posture estimator 330 .

The feature collecting unit 310 receives and combines the state vectors (β ^s-1 , θ ^s-1 , Z _t ^s-1 ) obtained in the previous stage and the two-dimensional posture feature to generate a combined state vector, and a position weight It is transmitted to the government 320 and the model posture estimation unit 330 .

The feature collecting unit 310 receives and combines the body vector (β ^s ), the position vector (Z _t ^s ), the root rotation vector (θ ₀ ^s ), and the joint rotation vector (θ _J ^s ) to combine the state vector (β ^s , θ ^s, Z _t ^s) of the state vector (β ^s, θ ^s, Z _t ^s) and the two-dimensional position characteristic extraction unit (200 acquired in the state vector acquiring unit 311 and the state vector obtaining unit 311 to obtain the ) may include a state feature combining encoding unit 312 that combines and encodes the applied two-dimensional posture feature to obtain a combined state vector.

^{The position estimation unit 320 estimates the body shape vector (β s} ) and the position vector (Z _t ^s ) according to the pattern estimation method previously learned from the coupling state vector applied by the feature collection unit 310 , and the posture (θ) Estimate the _{global rotation vector (θ 0} ^s ) representing the rotation of the middle root joint.

The position estimator 320 includes a body estimator 321 for estimating the ^{body vector β s} , and a depth/rotation estimator 322 for estimating the _{position vector Z t} ^s and the global rotation vector θ ₀ ^{s .} may include. The body shape estimator 321 and the depth/rotation estimator 322 may be implemented as independent artificial neural networks, respectively. Here, separately configuring the body shape estimator 321 and the depth/rotation estimator 322 _{has a similar tendency to estimate the position vector (Z t} ^s ) and the global rotation vector (θ ₀ ^s ), whereas the body shape vector ( β ^s ) because the estimation tendency is different.

³)를 추정한다.In addition, the model posture estimator 330 receives the combined state vector applied in the previous stage and the root rotation vector (θ ₀ ^s ) estimated by the position estimator 320 , and according to the pre-learned pattern estimation method, for the remaining joints. Joint rotation vector(θ ^s _J = [θ ^s ₁ , θ ^s ₂ , … θ ^s _K-1 ], θ ^s _i ∈

³ ) is estimated.

As shown in FIG. 8 , the model posture estimation unit 330 may include a plurality of joint rotation estimation units 331 to 33 (K-1) having a multi-stage structure. Here, the model posture estimator 330 includes a number of joint rotation estimation units 331 to 33 (K-1) corresponding to the number of joints K of the human model to be estimated, and the rotation vector θ _{J of each joint.} ^s = θ ₁ ^s , θ ₂ ^s , …, θ _K-1 ^s ) can be sequentially estimated. The model posture estimation unit 330 includes a plurality of joint rotation estimation units 331 to 33 (K-1)), and each joint rotation estimation unit 331 to 33 (K-1)) is a joint estimated in the previous stage. _{Sequentially estimating the rotation vector (θ J} ^s = θ ₁ ^s , θ ₂ ^s , …, θ _K-1 ^s ) of each joint from the rotation vector and the corresponding two-dimensional joint position included in the combined state vector is described above. As shown, since human joints have a kinematic tree structure from the root joint based on their interdependence, it is estimated as the relative rotation angle of each joint sequentially connected in the direction of the end of the body based on the root joint (here, hip joint as an example). case, it can be estimated more accurately.

However, in the case of the model posture estimator 330 shown in FIG. 8 , the plurality of joint rotation estimators 331 to 33 (K-1) have a multi-stage structure in which all are connected in series, but the human joint is are separated For example, the joints of both arms and legs can rotate independently. Accordingly, the model posture estimation unit 330 has a structure in which a plurality of joint rotation estimation units 331 to 33 (K-1) are branched into structures corresponding to human body parts, as shown in FIG. 9 , each It is also possible to estimate the rotation vector of the joint (θ _J ^s = θ ₁ ^s , θ ₂ ^s , …, θ _K-1 ^{s ).}

For example, in the model posture estimator 330 of FIG. 9 , the multiple joint rotation estimator in the first row estimates the rotation of each joint of the left leg, and the multiple joint rotation estimator in the second row includes the right leg and the third row and a plurality of joint rotation estimation units in the fourth row branched from the third row may estimate the rotation of each joint of the left arm and the right arm via the vertebral joint.

)에 따라 결정하여 배치한다.The three-dimensional space arrangement unit 400 calculates a body shape vector (β) and a rotation vector (θ) among _{the state vectors (β, θ, Z t} ) estimated by the multi-stage human model estimator 300 in a three-dimensional virtual space. Create a human model based on the position vector (Z _t ) and the root joint position (

) to determine and place

Meanwhile, the learning unit 500 calculates the loss of the body shape vector (β) (L _β ) and the loss of the position vector (Z _{t ) among the state vectors (β, θ, Z t} ) estimated by the multi-stage human model estimator 300 . It can be obtained by calculating according to Equations 2 and 3.

Here, N is the number of samples in the mini-batch, and the variable indicated by ^ represents the truth value labeled in the training data.

And the learning unit 500 converts the expression of the axis-angle form of each joint into a rotation matrix using Rodrigues's rotation formula, and then applies the L2 loss, and the posture loss is expressed in Equation 4 It can be calculated and obtained accordingly.

where θ _j,i is the posture parameter of body part j of sample i, R is the transformation from the axial angle representation to the rotation matrix, where the rotation matrix is flattened to 1D before the L2 loss is applied.

However, since the size of the axial angle expression represents a rotation that is a multiple of 2π, the hinge loss must be applied to the standard square of the attitude parameter to limit the size between -π and π, so the attitude loss is again expressed by Equation 5 can be expressed

In addition, the body posture loss is a loss _{excluding the global rotation vector (θ 0} ), and may be calculated by Equation (6).

), 위치 손실(

), 전역 회전 손실(

), 바디 손실(

)에 가중치(

)를 적용하고 바디 회전 손실(

) 추가로 고려하여 총 손실을 수학식 7에 따라 획득한다.The learning unit 500 loses body shape calculated according to Equations 2 to 7 (

), position loss (

), global rotation loss (

), body loss (

) to weight(

) and lose body rotation (

), the total loss is obtained according to Equation (7) with additional consideration.

And by backpropagating the acquired total loss L, the multi-stage human model estimator 300 may be trained.

)는 일예로 (1, 5, 10, 1)로 설정하였다.where the weight (

) is set to (1, 5, 10, 1) as an example.

10 shows a 3D human model reconstruction method according to an embodiment of the present invention.

Referring to FIG. 10 , in the 3D human model restoration method according to the present embodiment, a 2D image including at least one person to be modeled as a 3D model is first obtained ( S11 ). Then, the joint vector of each person included in the obtained 2D image is estimated according to the pre-learned pattern estimation method (S12). When the joint vector is estimated, the estimated joint vector is normalized to obtain a two-dimensional posture characteristic for each of at least one person included in the two-dimensional image (S13). In this case, the skeletal vector may be estimated together with the joint vector, and when the skeletal vector is estimated, the skeletal vector may be normalized and combined together with the joint vector to obtain a two-dimensional posture characteristic.

)가 포함될 수 있다.Here, normalization converts the joint position estimated according to the size and resolution of the input 2D image into coordinates of a predetermined normalized image plane, and normalizes the position of the root joint so that the root joint among the transformed joint positions becomes an anchor point on the image plane It can be performed in the form of moving to the origin of the coordinates of . At this time, the 2D posture feature includes the position of the root joint (

) may be included.

^{When the two-dimensional posture feature is obtained, initial state vectors (β 0} , θ ⁰ , Z _t ⁰ ) for generating and locating a human model in a three-dimensional space are set based on the SMPL human model template (S14). Then, the combined state vector is obtained by combining the initial state vectors (β ⁰ , θ ⁰ , Z _t ⁰ ) with the two-dimensional posture feature and encoding according to a pre-learned pattern estimation method (S15). When the binding state vector is obtained, a body shape vector (β) representing the external features of the human model is estimated according to a pre-learned pattern estimation method using the acquired binding vector (S16). _{A position vector (Z t} ) and a global rotation vector (θ ₀ ) representing the position and rotation state of the human model are obtained together according to a pre-learned pattern estimation method using the joint vector obtained in parallel (S17).

When the global rotation vector (θ ₀ ) is obtained, the joint rotation vector (θ _J = θ ₁ , θ ₂ , …, θ _K-1 ) for each joint is obtained from the combined state vector and the obtained global rotation vector (θ _{0 ).} Estimate (S18). Here, the joint rotation vector (θ _J ) for each joint is sequentially estimating the rotation vector of the joint from the joint adjacent to the root joint in the direction of the end of the body, or estimating in parallel for each part according to the body structure, and simultaneously estimating the joint in each part. Depending on the connection structure, it can be estimated sequentially.

When the body vector (β), position vector (Z _t ), global rotation vector (θ ₀ ), and joint rotation vector (θ _J ) are estimated and new state vectors (β, θ, Z _t ) are estimated, the number of state vector estimations It is determined whether is more than a predetermined number of times (S19). In the present embodiment, as shown in FIG. 6 , the number of state vector estimation may be determined according to the number of model estimators included in the multi-stage human model estimator 300 . If the number of state vector estimation is less than the number specified by the number of model estimators, the re-estimated state vector (β, θ, Z _t ) and the two-dimensional posture feature are combined and encoded according to the pre-learned pattern estimation method to combine the state vector to obtain (S15).

)로부터 3차원 공간 상에 인간 모델이 배치될 위치를 결정하여 배치한다(S21).However, if the number of state vector estimation is greater than or equal to the specified number of times, a human model is generated based on the body shape vector (β) and the posture vector (θ) among the _{obtained state vectors (β, θ, Z t ) ( S20 ).} And the position vector (Z _t ) and the position of the root joint obtained during normalization (

) to determine and arrange a position where the human model is to be placed on the three-dimensional space (S21).

On the other hand, when the human model is arranged in the 3D space, it is determined whether the human models for all people in the 2D image are arranged in the 3D space (S22). If there is a person who is not placed, the joint position of the person who is not placed again is estimated (S12). However, when it is determined that all people have been placed, the 3D human model restoration is terminated.

), 위치 손실(

), 전역 회전 손실(

), 바디 손실(

)을 계산하고 대응하는 가중치(

)를 적용하며, 바디 회전 손실(

) 추가로 고려하여 총 손실을 수학식 7에 따라 획득한다. 그리고 획득된 총 손실을 역전파하여 학습을 수행할 수 있다.Although not shown, the 3D human model reconstruction method according to the present embodiment may further include a learning step for learning a pattern estimation method for estimating the _{state vectors (β, θ, Z t ) from the combined state vector.} The learning stage is the loss of body shape (

), position loss (

), global rotation loss (

), body loss (

) and calculate the corresponding weights (

), and the body rotation loss (

), the total loss is obtained according to Equation (7) with additional consideration. Then, learning can be performed by backpropagating the acquired total loss.

11 shows an example of a human model reconstructed in a 3D space from a 2D image using the 3D human model reconstruction method of the present invention.

11 is a result of reconstructing a human model in a 3D space from a 2D image using the MS COCO data set. 11 , in the 3D human model restoration apparatus and method according to the present embodiment, when a 2D image including a plurality of people is applied, not only the posture and body type of each person included in the image but also the 3D space It can be seen that even the relative position on the image can be very accurately restored.

The method according to the present invention may be implemented as a computer program stored in a medium for execution by a computer. Here, the computer-readable medium may be any available medium that can be accessed by a computer, and may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data, and read dedicated memory), RAM (Random Access Memory), CD (Compact Disk)-ROM, DVD (Digital Video Disk)-ROM, magnetic tape, floppy disk, optical data storage, and the like.

Although the present invention has been described with reference to the embodiment shown in the drawings, which is merely exemplary, those skilled in the art will understand that various modifications and equivalent other embodiments are possible therefrom.

Accordingly, the true technical protection scope of the present invention should be defined by the technical spirit of the appended claims.

100: image acquisition unit 200: two-dimensional posture feature extraction unit
300: multi-stage human model estimator 400: 3D space arrangement unit
500: study

Claims (17)

According to the pre-learned pattern estimation method, a 2D joint vector corresponding to the joint position of each of at least one person included in the 2D image is estimated, and the estimated 2D joint vector is normalized by a predetermined method to characterize the 2D posture. A two-dimensional posture feature extraction unit to obtain;
In order to receive the two-dimensional posture characteristic for each of the at least one person included in the two-dimensional image, and to reconstruct a human model of a body type and posture corresponding to the two-dimensional posture characteristic according to a pre-learned pattern estimation method, the human a multi-stage human model estimator for estimating a body shape vector and a posture vector specified by a model template, and estimating a position vector for arranging the restored human model at a position corresponding to the two-dimensional posture feature on a three-dimensional space; and
A state vector composed of the body shape vector, the posture vector, and the position vector and the two-dimensional posture feature are applied, the human model corresponding to the body vector and the posture vector is restored, and the human restored according to the position vector A three-dimensional human model restoration apparatus including a three-dimensional space arrangement unit for arranging a model on a three-dimensional space. According to claim 1, wherein the multi-stage human model estimator
It includes a plurality of model estimators sequentially connected in a multi-stage structure,
Each of the plurality of model estimators generates a combined state vector by receiving and combining the two-dimensional posture feature and the state vector estimated by the model estimator of the previous stage, and according to the pattern estimation method learned in advance from the generated combined state vector. In order to reconstruct a human model of a body shape and posture corresponding to the two-dimensional posture characteristic, a body shape vector and a posture vector specified by a human model template are estimated, and the restored human model corresponds to the two-dimensional posture feature in a three-dimensional space. A three-dimensional human model restoration apparatus that estimates a position vector to be placed in a position to generate a new state vector including a body vector, a posture vector, and a position vector, and transmits it to the model estimator of the next stage. The method of claim 2, wherein each of the plurality of model estimators
a feature assembling unit for generating a combined state vector by receiving and combining the two-dimensional posture characteristic and the state vector estimated by the model estimator of the previous stage;
According to the pattern estimation method learned in advance from the combined state vector, the body shape vector corresponding to the two-dimensional posture feature, the position vector corresponding to the position where the human model should be placed in the three-dimensional space, and the rotation angle of each of a plurality of human joints a position estimator for estimating a global rotation vector representing a rotation angle of the entire human model based on a predetermined root joint among posture vectors composed of a plurality of corresponding joint rotation vectors; and
The two-dimensional posture characteristic and the global rotation vector are applied, and the relative rotation angle of a joint far from an adjacent joint in the root joint is estimated in a predetermined sequence based on the global rotation vector to obtain the global rotation vector among the posture vectors. A three-dimensional human model restoration apparatus including a model posture estimator for obtaining a body posture vector including a joint rotation vector of the remaining joints. The method of claim 3, wherein the position estimator
a body shape estimator trained in advance to estimate a body shape vector corresponding to the two-dimensional posture characteristic from the combined state vector; and
and a depth/rotation estimator for simultaneously estimating the position vector and the global rotation vector corresponding to the two-dimensional posture feature from the combined state vector by being independently learned from the body shape estimator. The method of claim 4, wherein the model posture estimating unit
Among the plurality of joints of the human model, the number of joints corresponding to the remaining joints except for the root joint is provided, and the joint rotation corresponding to the global rotation vector or the joint rotation vector obtained in the previous stage and the two-dimensional posture characteristic are applied, respectively. A three-dimensional human model restoration apparatus including a plurality of joint rotation estimators for estimating vectors. According to claim 5, wherein the plurality of joint rotation estimation unit
In response to the joint connection structure for each part of the human model, a three-dimensional human model that is connected in parallel to each of a plurality of parts from the root joint and is connected in series according to the joint connection structure of each part to estimate a joint rotation vector restore device. According to claim 1, wherein the two-dimensional posture feature extraction unit
The two-dimensional joint vector is transformed to have a size and position corresponding to the normalized image plane of a predetermined scale and resolution, and the position of the joint vector corresponding to the root joint in the transformed two-dimensional joint vector is placed at the origin of the normalized image plane A 3D human model restoration device that moves as much as possible. The method of claim 7, wherein the two-dimensional posture feature extraction unit
A 3D human model restoration apparatus including the position of a joint vector corresponding to the root joint before moving to the origin of the normalized image plane in the 2D posture characteristic. The method of claim 7, wherein the two-dimensional posture feature extraction unit
3D comprising: estimating a 2D skeletal vector corresponding to each skeletal direction of at least one person included in the 2D image, normalizing the estimated 2D skeletal vector in a predetermined manner, and further including in the 2D posture characteristic Human model restoration device. According to claim 3, wherein the three-dimensional human model restoration apparatus
Loss of body shape from the state vector estimated by the multi-stage human model estimator from the two-dimensional learning image and the truth value of the state vector labeled in the two-dimensional learning image (

), position loss (

), global rotation loss (

), body loss (

) and calculate the corresponding weights (

), and the body rotation loss (

), further taking into account the total loss in the formula

3D human model restoration apparatus further comprising a learning unit for calculating and backpropagating the calculated total loss to train the multi-stage human model estimator. According to the pre-learned pattern estimation method, a 2D joint vector corresponding to the joint position of each of at least one person included in the 2D image is estimated, and the estimated 2D joint vector is normalized by a predetermined method to characterize the 2D posture. obtaining a;
In order to receive the two-dimensional posture characteristic for each of the at least one person included in the two-dimensional image, and to reconstruct a human model of a body type and posture corresponding to the two-dimensional posture characteristic according to a pre-learned pattern estimation method, the human obtaining a state vector by estimating a body shape vector and a posture vector specified by a model template, and estimating a position vector for placing the reconstructed human model at a position corresponding to the two-dimensional posture feature on a three-dimensional space; and
A state vector composed of the body shape vector, the posture vector, and the position vector and the two-dimensional posture feature are applied, the human model corresponding to the body vector and the posture vector is restored, and the human restored according to the position vector A method of reconstructing a three-dimensional human model comprising the step of placing the model on a three-dimensional probability. 12. The method of claim 11, wherein obtaining the state vector comprises:
generating a combined state vector by receiving and combining the two-dimensional posture characteristic and the previously estimated state vector;
estimating a body shape vector and a posture vector specified by a human model template to reconstruct a human model of a body shape and a posture corresponding to the two-dimensional posture feature according to a pattern estimation method learned in advance from the generated combined state vector;
estimating a position vector for arranging the reconstructed human model at a position corresponding to the two-dimensional posture feature on a three-dimensional space; and
A method of restoring a three-dimensional human model, comprising generating and transmitting a new state vector including a body vector, a posture vector, and a position vector. The method of claim 12, wherein estimating the posture vector comprises:
According to the pattern estimation method learned in advance from the combined state vector, the body shape vector corresponding to the two-dimensional posture feature, the position vector corresponding to the position where the human model should be placed in the three-dimensional space, and the rotation angle of each of a plurality of human joints estimating a global rotation vector representing a rotation angle of the entire human model based on a predetermined root joint among posture vectors composed of a plurality of corresponding joint rotation vectors; and
The two-dimensional posture characteristic and the global rotation vector are applied, and the relative rotation angle of a joint far from an adjacent joint in the root joint is estimated in a predetermined sequence based on the global rotation vector to obtain the global rotation vector among the posture vectors. 3D human model restoration method comprising the step of obtaining a body posture vector including a joint rotation vector of the remaining joints. 14. The method of claim 13, wherein estimating the global rotation vector comprises:
estimating a body vector corresponding to the two-dimensional posture feature from the combined state vector; and
and simultaneously estimating the position vector and the global rotation vector corresponding to the two-dimensional posture feature from the combined state vector independently of the step of estimating the body shape vector. The method of claim 11, wherein the acquiring of the two-dimensional posture characteristic comprises:
converting the two-dimensional joint vector to have a size and a position corresponding to a normalized image plane of a predetermined scale and resolution; and
A 3D human model restoration method comprising the step of moving the position of the joint vector corresponding to the root joint in the transformed 2D joint vector to be placed at the origin of the normalized image plane. The method of claim 15, wherein the acquiring of the two-dimensional posture characteristic comprises:
The 3D human model restoration method further comprising the step of including the position of the joint vector corresponding to the root joint before moving to the origin of the normalized image plane in the 2D posture characteristic. The method of claim 13 , wherein the three-dimensional human model reconstruction method comprises:
Loss of body shape from the truth value of the state vector estimated from the two-dimensional learning image and the state vector labeled in the two-dimensional learning image (

), position loss (

), global rotation loss (

), body loss (

) and calculate the corresponding weights (

), and the body rotation loss (

), further taking into account the total loss in the formula

3D human model reconstruction method further comprising the step of calculating and learning by backpropagating the calculated total loss.

Images