KR102463778B1 - Method and Apparatus for Reconstruction of Holistic 3D Body from a Blurred Single Image - Google Patents

Abstract

Disclosed are a method and apparatus for reconstructing a three-dimensional (3D) body model from a blurred image, which generate a new large-scale dataset to reconstruct a 3D human body model from a blurry image. According to the present invention, the method for reconstructing a 3D body model from a blurred image comprises the steps of: generating, by a dataset generation module, a synthetic dataset for motion deblurring and 3D human body model reconstruction; using the dataset, by a motion deblurring module, to generate a motion deblurred image and using a loss function to learn the motion deblurred image; using, by a structure recognition module, the learned motion deblurred image to provide structural information for integrated human body reconstruction; and using, by a human body reconstruction module, the deblurred image with the structural information to reconstruct a 3D human body model.

Description

Method and apparatus for restoring 3D human body model from image with blur {Method and Apparatus for Reconstruction of Holistic 3D Body from a Blurred Single Image}

The present invention relates to a method and apparatus for reconstructing a 3D human body model from a blurred image.

Human body pose and shape reconstruction have been widely used in various applications such as human action detection, augmented/virtual reality, motion capture, and online games, drawing attention from the research community. In addition to the recent success of human body mesh reconstruction, the shift from single sharp image to integrated body mesh reconstruction yields impressive results. Unlike the non-integrative model, the integrative anatomy includes detailed facial expressions and finger-level hand poses with more parameters.

1 is a diagram for explaining an example of integrated 3D human body reconstruction according to the prior art.

From the left, human motion blur image, 3D integrated body mesh of ground truth, 3D integrated human body mesh of ExPose, 3D integrated human body mesh of 'Deblur + Exposure', and the result of the proposed joint framework. The proposed method reconstructs the general shape as well as the posture of the head and hands more accurately than the current state-of-the-art (ExPose).

Recent work on human body mesh reconstruction or even integrated body mesh reconstruction considers intrinsically sharp images as input. However, in many cases, images captured from wild scenes are often blurred by human and camera movements. As a result, an accurate human body mesh as shown in FIG. 1 cannot be reconstructed.

Although state-of-the-art image deblurring methods have achieved remarkable performance, they cannot be used as a preprocessing step prior to reconstructing the human body mesh. This is because human motion is often accompanied by a peculiar perturbation, which increases the uncertainty of the motion blur effect.

To solve the aforementioned problems, we introduce a joint and unified framework for deblurring and integrated 3D human body reconstruction, called D2R, to solve both problems simultaneously. As a single integrated framework, the deblurring module utilizes the human reconstruction module to leverage human reconstruction loss to improve performance. Deblurring performance is improved while reconstruction performance is also improved. This is because you can get sharper results, which will give you more accurate results. Even so, it is still difficult to regress the human body mesh details directly from the deblur image because the information about the specific part of the human is inadequate. As a result, a structure recognition module for emphasizing human body details in the form of a structure map is required.

The technical problem to be achieved by the present invention is to utilize a clear image as an input in the existing deep 3D integrated human body pose and shape reconstruction method, leading to an inaccurate human body mesh when a blurry image is given, and a satisfactory result with a simple cascade approach. In order to improve the problem of unobtainable image deblurring method, we propose a new joint framework of human motion deblurring and 3D integrated human body reconstruction, D2R (Deblurring-to-Reconstruction). In addition, a method and apparatus for restoring a 3D human body model from a blurry image by generating a new large-scale dataset including a clear/blurred image pair and a corresponding 3D human body pose/shape are provided.

In one aspect, the method for restoring a 3D human body model from an image with blur proposed in the present invention generates a synthetic dataset for motion deblurring and reconstruction of a 3D human body model through a dataset generation module. Generating, generating a motion deblurred image through a motion deblurring module using the dataset, and learning the motion deblurred image using a loss function, the learned motion deblurring Providing structural information for integrated human body reconstruction through a structure recognition module using a ringed image and reconstructing a 3D human body model through a human body reconstruction module using an image deblurred with the structure information do.

Generating a synthetic dataset for motion deblurring and 3D human body model reconstruction through the dataset generation module applies video frame interpolation to increase the frame rate before an average image is calculated, During average image calculation, the human body region is segmented to create a human body mask to reduce the blur effect on the background region, pseudo ground truth 2D keypoints are created to obtain integrated 3D parameters, and pseudo ground truth parameters and By generating 3D keypoints and filtering the data, synthetic datasets are created for motion deblurring and 3D human body model reconstruction.

The steps of generating a motion deblurred image through a motion deblurring module using the dataset and learning the motion deblurred image using a loss function include a pixel-wise mean error loss function and a perceptual loss function. , the adversarial loss function and the body reconstruction loss function for both global and local discriminators are used to generate motion deblurred images.

The step of reconstructing a 3D human body model through a human body reconstruction module using the image deblurred with the structure information is to estimate a projection matrix and integrated 3D parameters from the deblurred image, and to regress the integrated 3D parameters. Concatenate structure features and image features as inputs, the image features are generated from the deblur image by a convolution block, and use a feature fusion block to dynamically fuse the connection features between the structure features and image features.

In another aspect, the apparatus for restoring a 3D human body model from an image with blur proposed in the present invention generates a synthetic dataset for motion deblurring and reconstruction of a 3D human body model. A generation module, a motion deblurring module that generates a motion deblurred image using the dataset, and learns the motion deblurred image using a loss function, the learned motion deblurred image and a structure recognition module that provides structural information for integrated human body reconstruction using the structure information, and a human body reconstruction module that reconstructs a three-dimensional human body model using the image deblurred with the structure information.

Through D2R (Deblurring-to-Reconstruction), a new joint framework of human motion deblurring and 3D integrated human body reconstruction according to embodiments of the present invention, a clear image is input from the existing deep 3D integrated human body pose and shape reconstruction method It is possible to improve the problem of inaccurate human body meshing when a blurry image is given and the problem of image deblurring methods that cannot obtain satisfactory results with a simple cascade approach. A 3D human body model can be reconstructed from a blurred image by creating a new large-scale dataset containing sharp/blurred image pairs and corresponding 3D human body poses/shapes.

1 is a diagram for explaining an example of integrated 3D human body reconstruction according to the prior art.
2 is a flowchart illustrating a method of restoring a 3D human body model from an image with blur according to an embodiment of the present invention.
3 is a diagram showing the configuration of an apparatus for restoring a 3D human body model from an image with blur according to an embodiment of the present invention.
4 is a diagram for explaining a concept of a pipeline of a network according to an embodiment of the present invention.
5 is an algorithm illustrating how replacement training is performed according to one embodiment of the present invention.

Integrated human body pose and shape reconstruction is receiving great attention by restoring detailed human body posture and shape, such as facial expressions and finger-level hand shapes. Existing deep 3D integrated body pose and shape reconstruction methods use clear images as input, leading to inaccurate body meshes when blurry images are given. There are various image deblurring methods, but a simple step-by-step approach cannot produce satisfactory results.

In the present invention, we propose a new joint framework of human motion deblurring and 3D integrated human body reconstruction, Deblurring-to-Reconstruction (D2R), to solve the two problems at the same time. It also creates a new large-scale dataset containing sharp/blurred image pairs and corresponding 3D human body poses/shapes.

The present invention proposes a method and apparatus for restoring a 3D human body model incorporating a structure recognition module for emphasizing human body details in the form of a structure map. According to an embodiment of the present invention, the proposed joint framework is learned in an alternative way to improve the performance of each module by utilizing an additional structure recognition module.

Although the network is designed as a single unified framework, solving common problems end-to-end is not effective because the problems belong to different domains. Therefore, the proposed D2R is solved by learning each module alternately. In addition, due to the lack of appropriate datasets, we propose a new dataset for integrated 3D body reconstruction that includes large-scale blurred human body images and 3D human body parameters of corresponding real-world data. Experimental results show that the proposed method outperforms integrative human body reconstruction qualitatively as well as qualitatively while the input image is deblurred. Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Recently, by applying the excellence of deep neural networks, several techniques have made significant progress in the field of 3D human body posture and shape reconstruction from monocular images. Most prior art parametric human body models (SMPL [9]) were utilized to reconstruct human body meshes.

Another prior art directly regressed the SMPL parameters on the input image. On the other hand, instead of using highly non-linear image features, some tasks have utilized additional representations as inputs, such as human masks or high-density correspondences. While the aforementioned techniques focus only on clear images as input, another prior art introduces a new method of reconstructing a human body mesh from low-resolution images.

Comparing to existing poses and human bodies, it is not an easy task to reconstruct an integrated human body because details of the face and hands, such as more accurate head posture, facial expression, and finger-level hand posture, must be reconstructed.

Another prior art has adopted an optimization approach, where an anatomical model is fit to 2D points to obtain an integral anatomical mesh. This approach yields notable results, but inference is slow and prone to noise.

Another prior art proposed ExPose, which predicts SMPL-X parameters directly from a single RGB image and has faster inference time and improved performance. The D2R framework proposed in the present invention reconstructs an integrated human body mesh from a blurry image. The method proposed in the present invention also restores the deblurred image by using the information of the human body reconstruction network.

In many techniques of general image deblurring, deep learning technology is used to handle the image deblurring task.

In the prior art, a recurrent neural network architecture has been proposed to solve non-uniform motion blur. Another prior art introduced a hierarchical structure, and the aforementioned approaches mostly focused on dynamic scene blur and struggled to deal with foreground motion blur. To address these issues, a multi-head decoder architecture was introduced to simultaneously address dynamic scene and object motion blur with improved results.

Meanwhile, the success of GAN-based architectures in other image restoration fields also inspired the field of image deblurring. Another prior art proposed a coarse-to-fine approach through adversarial learning. Another prior art proposed a GAN-based method with perceptual loss and adversarial loss. Recent work has designed an approach to deblurring an image by learning how to create blur.

A major factor in this success of the prior art is the availability of large datasets. For the deblurring application, a dynamic blur dataset was created by averaging multiple consecutive frames captured by a high-frame video camera, referred to as the GoPro dataset. Another prior art followed a similar strategy to create a HIDE dataset containing human pedestrian activity.

For the field of human body reconstruction, 2D human keypoint annotations were used to generate sharp image pairs in the wild. Another prior art provided millions of sharp images captured in a confined environment through 3D and 2D human body keypoint annotation. Recently, we proposed an extension of the human body MS-COCO dataset to include additional annotations on face and hand details.

To bridge the gap between human body reconstruction and deblurring, it is necessary to have a user-defined and sophisticated dataset. However, the available datasets are not suitable for the purposes of the present invention. Although a human body motion blur dataset has been proposed in the prior art, this dataset is still insufficient for use because of the absence of human body annotation, restrictions on human activities, and most of the blur effect caused by camera shake motion. In this context, it is necessary to build a new dataset containing various aspects of human body motion effects, such as blurred images, sharp ground truth, 3D human body, face, and hand annotations.

2 is a flowchart illustrating a method of restoring a 3D human body model from an image with blur according to an embodiment of the present invention.

The proposed method of restoring a 3D human body model from an image with blur includes generating a synthetic dataset for motion deblurring and 3D human body model reconstruction through a dataset generation module (210). Generating a motion deblurred image through a motion deblurring module using a dataset, and learning the motion deblurred image using a loss function (220). Providing structural information for integrated human body reconstruction through a structure recognition module using an image (230) and reconstructing a 3D human body model through a human body reconstruction module using the deblurred image with the structure information ( 240).

In step 210, a synthetic dataset for motion deblurring and 3D human body model reconstruction is generated through the dataset generation module.

Video frame interpolation is applied to increase the frame rate before the average image is calculated, and body regions are segmented during average image calculation to create a body mask to reduce the blur effect on the background region.

Then, synthetic data for motion deblurring and 3D human body model reconstruction by generating pseudo ground truth 2D keypoints to obtain integrated 3D parameters and filtering data by generating pseudo ground truth parameters and 3D keypoints A set (synthetic dataset), that is, a synthetic INstavariety Blur (SIN blur) dataset according to an embodiment of the present invention is generated.

In step 220, a motion deblurred image is generated through a motion deblurring module using the dataset, and the motion deblurred image is learned using a loss function. In this case, a motion deblurred image is generated using a per-pixel mean error loss function, a perceptual loss function, an adversarial loss function for both global and local discriminators, and a body reconstruction loss function.

In step 230, structural information for integrated human body reconstruction is provided through a structure recognition module using the learned motion deblurred image.

In step 240, a 3D human body model is reconstructed through a human body reconstruction module using the deblurred image with the structure information.

An anatomy reconstruction module estimates projection matrices and integral 3D parameters from the deblurred image, and concatenates the structural features and image features as inputs to regress the integral 3D parameters.

The image feature is generated from the deblur image by a convolution block, and uses the feature fusion block to dynamically fuse the structural features and the connection features between the image features.

3 is a diagram showing the configuration of an apparatus for restoring a 3D human body model from an image with blur according to an embodiment of the present invention.

The proposed apparatus for restoring a 3D human body model from an image with blur includes a dataset generation module 310, a motion deblurring module 320, a structure recognition module 330, and a human body reconstruction module 340.

The dataset generation module 310 according to an embodiment of the present invention generates a synthetic dataset for motion deblurring and 3D human body model reconstruction through the dataset generation module.

Video frame interpolation is applied to increase the frame rate before the average image is calculated, and body regions are segmented during average image calculation to create a body mask to reduce the blur effect on the background region.

Then, synthetic data for motion deblurring and 3D human body model reconstruction by generating pseudo ground truth 2D keypoints to obtain integrated 3D parameters and filtering data by generating pseudo ground truth parameters and 3D keypoints A set (synthetic dataset), that is, a synthetic INstavariety Blur (SIN blur) dataset according to an embodiment of the present invention is generated.

, 포즈

및 표현 파라미터

로 구성된다. 이러한 매개변수는 N = 10,475개의 정점을 갖는 자세와 형태의 3D 인체 메시

을 생성하는 데 사용될 수 있다. 구체적으로 포즈 파라미터

는 인체, 얼굴, 손의 관절을 나타낸다. 주요 관절 J는 몸통관절 22개, 턱 포즈 1개, 손당 손가락 관절 15개로 구성된 53개의 관절이 있다. D는 [2]에서 수정된 회전 표현으로, 여기서 D = 6이다. 형태 파라미터

및 표현 매개변수

은 해당 PCA 공간의 10개 계수로 표현된다. The integrated human body mesh according to an embodiment of the present invention is represented by the SMPLX[1] model that simultaneously captures details of the human body, face, and hands. SMPL-X parameters are of the form

, pose

and expression parameters

consists of These parameters are a 3D human mesh with pose and shape with N = 10,475 vertices.

can be used to create specifically the pose parameters

represents the joints of the human body, face, and hand. Major joint J has 53 joints, consisting of 22 torso joints, 1 jaw pose, and 15 finger joints per hand. D is the rotation expression modified from [2], where D = 6. shape parameters

and expression parameters

is expressed as 10 coefficients of the corresponding PCA space.

According to the prior art, there is no dataset available to solve integrated 3D anatomical reconstruction from blurry images with necessary annotations such as ground truth 3D anatomy, sharp images, 2D and 3D keypoints.

In generating a dataset according to an embodiment of the present invention, we propose a synthesized INstavariety Blur (SINBlur) dataset, which is the first and only dataset to solve the common problem of deblurring and integrated 3D human body reconstruction. Table 1 shows a summary and comparison of the existing dataset and the proposed dataset.

<Table 1>

In particular, it is difficult to obtain an image including human motion blur and ground truth information of integrated 3D human body parameters in a real environment. Therefore, blurring images from a series of video frames are often not composited into an averaged image.

In the present invention, video clips of the InstaVariety dataset [3] containing various human body activity videos are utilized. Nonetheless, simple averaging over successive frames results in unrealistic ghosting artifacts due to low frame rates (e.g., 30 fps).

The present invention solves the aforementioned problem by applying video frame interpolation [4] to increase the frame rate from 30 fps to 240 fps before the average image is calculated. In addition, the body mask [5] is created to reduce the blur effect on the background area by segmenting the body area during average image calculation.

To obtain integrated 3D parameters, pseudo ground truth 2D keypoints are generated using OpenPose [6], and pseudo ground truth SMPL-X parameters and 3D keypoints are generated using Smplify-X [7]. Note that clear images are used to create corresponding ground truths for blurry images. In addition, it checks similar ground truth data information from various perspectives to filter out data lacking in annotations.

Finally, according to an embodiment of the present invention, we carefully select 20,000 images divided into 18,911 training images and 1,089 test images. The selection requirements are summarized as follows: the size of the person area must be 30% of the image, details (face and hands) are visible in the image, and the occlusion area of the image is less than 10%.

Similar labels according to an embodiment of the present invention originate from the semi-map study area [8]. Briefly, first, the network is trained with a labeled dataset of limited size. Then, the trained network is used to predict similar labels for unlabeled data. Finally, the network is trained on a pseudo-labeled dataset along with a labeled dataset. This procedure is repeated until convergence. We show that pseudo-labeling improves the performance of the evaluation dataset with real-world ground truth.

Referring back to FIG. 3, the motion deblurring module 320 according to an embodiment of the present invention generates a motion deblurred image through the motion deblurring module using the dataset and uses a loss function. to learn the motion deblurred image. In this case, a motion deblurred image is generated using a per-pixel mean error loss function, a perceptual loss function, an adversarial loss function for both global and local discriminators, and a body reconstruction loss function.

4 is a diagram for explaining a concept of a pipeline of a network according to an embodiment of the present invention.

, 구조 인식 모듈

및 재구성 모듈

의 세 가지 핵심 모듈로 구성된다. 흐릿한 인체 이미지가

에 공급되어 디블러링된 결과를 생성한다. 그런 다음

은

에서 얻은 디블러 이미지와 구조 맵에서 카메라와 SMPL-X 파라미터를 생성한다. 도 4는 본 발명의 실시예에 따른 블러가 존재하는 영상으로부터의 3차원 인체 모델을 복원하기 위한 프레임워크의 구조를 보여준다.In the present invention, we introduce a new D2R framework to solve both motion deblurring and integrated 3D reconstruction. The proposed framework is a deblurring module

, the structure recognition module

and reconstruction module

It consists of three core modules. blurry human body image

to produce a deblurred result. after that

silver

Generate the camera and SMPL-X parameters from the deblur image and structure map obtained from 4 shows the structure of a framework for restoring a 3D human body model from an image with blur according to an embodiment of the present invention.

는 모션 디블러링된 이미지

를 생성한다. 여기서

와

는 256으로 설정된다. 본 발명의 실시예에 따른 접근 방식에서는 인체 영역

주변의 이미지를 자르는 패치 기반 학습을 적용하여 해당 영역의 정교화에 초점을 맞춘다. 또한 무작위로 잘라낸 패치 이미지

도 데이터 확대에 사용된다. Motion deblurring module according to an embodiment of the present invention

is the motion deblurred image

generate here

Wow

is set to 256. In the approach according to an embodiment of the present invention, the human body area

It applies patch-based learning that crops images around it, focusing on the refinement of that region. Also randomly cropped patch image

Also used for data augmentation.

를 학습시키기 위해, 먼저 다음과 같이 픽셀 단위 평균 오차 손실 L_P, 지각 손실 L_X, 글로벌 및 로컬 판별기 모두에 대한 적대적 손실 L_Adv로 구성된 기존의 디블러링 손실 함수

로 시작한다.According to an embodiment of the present invention,

To train , first a conventional deblurring loss function consisting of per-pixel mean error loss L _P , perceptual loss L _X , and adversarial loss L _Adv for both global and local discriminators as

It starts with

(1)

(One)

을 제안한다.

은 인체 재구성 네트워크

에

를 공급함으로써 얻을 수 있다. 요약하자면, 본 발명의 실시예에서는 다음에서 설명한 것과 같이 네 가지 손실 함수를 활용한다.However, the existing loss function is insufficient to deblur the human body area. Hence the new human reconstruction loss for motion deblurring module

suggests

silver human body reconstruction network

to

can be obtained by supplying In summary, the embodiments of the present invention utilize four loss functions as described below.

(2)

(2)

,

,

는 각각 0.3, 25, 0.5로 설정된다.

은 아래에서 더욱 정확하게 공식화된다. where the weight coefficient

,

,

are set to 0.3, 25, and 0.5, respectively.

is formulated more precisely below.

Referring back to FIG. 3 , the structure recognition module 330 according to an embodiment of the present invention provides structure information for integrated human body reconstruction through the structure recognition module using the learned motion deblurred image.

는 디블러링된 이미지에서 구조 정보를 제공하여 통합적 인체 재구성 모듈을 돕는다. 이것의 목표는 구조맵

에서 저차원 특징 F_S를 추출하는 것이다. S를 얻기 위해 구조 부동 함수(structure dissimilarity function) SD를 적용하여

와

의 구조 차이를 다음과 같이 계산한다.structure recognition module

assists the integrated human body reconstruction module by providing structural information from the deblurred image. Its goal is to map the structure

It is to extract low-dimensional features F _S from . Applying the structure dissimilarity function SD to obtain S

Wow

The structural difference of is calculated as:

(3)

(3)

(4)

(4)

와

는 평균값이고

와

는 각각 블러링된 인체와 복원된 인체의 분산이다. 블러링된 인체와 복원된 인체의 공분산은

로 표시된다. C₁과 C₂는 0으로의 나누기를 피하기 위한 안정화 상수이다. 식 (4)는 SD 함수가 역방향(1-SSIM)을 계산하는 SSIM 메트릭과 유사하다. 이러한 이유로 S는 배경을 무시함으로써 인체 영역의 중간 결과로 나타낼 수 있다. 이후,

는 컨볼루션 블록

에 의해 S로부터 특징

를 추출하고 다음과 같이 설명된다: here

Wow

is the average value

Wow

is the variance of the blurred human body and the restored human body, respectively. The covariance of the blurred and restored bodies is

is indicated by C ₁ and C ₂ are stabilization constants to avoid division by zero. Equation (4) is similar to the SSIM metric where the SD function computes the inverse (1-SSIM). For this reason, S can be represented as an intermediate result in the human body region by ignoring the background. after,

is the convolution block

Features from S by

is extracted and described as:

(5)

(5)

는 2D 배치 정규화, ReLU, 최대 풀링 계층으로 구성된다. here

consists of 2D batch normalization, ReLU, and max pooling layers.

Referring back to FIG. 3 , the human body reconstruction module 340 according to an embodiment of the present invention reconstructs a 3D human body model using the image deblurred with the structure information through the human body reconstruction module.

An anatomy reconstruction module estimates projection matrices and integral 3D parameters from the deblurred image, and concatenates the structural features and image features as inputs to regress the integral 3D parameters.

The image feature is generated from the deblur image by a convolution block, and uses the feature fusion block to dynamically fuse the structural features and the connection features between the image features.

은 디블러링된 이미지

로부터 투영 행렬

및 SMPL-X 파리미터 (

)를 추정한다. 파라미터를 회귀시키기 위해,

은 다음과 같이 표현되는 구조 특징

와 이미지 특징

를 입력으로 연결한다. Integrated human body reconstruction module according to an embodiment of the present invention

is a deblurred image

projection matrix from

and SMPL-X parameters (

) is estimated. To regress the parameters,

is a structural feature expressed as

with image features

connect as input.

(6)

(6)

(7)

(7)

는 컨볼루션 블록

에 의해 디블러 이미지

로부터 생성되며

와 같은 양의 계층을 갖는다. 특징 융합 블록

는

및

(

로 표시) 사이의 연결된 특징을 동적으로 융합하기 위해 도입된다. 또한 카메라와 SMPL-X 파라미터를 추정하기 위해

의 출력이

에 공급된다. 모듈을 학습하기 위해, 본 발명에서는 3D L_3D 및 2D L_2D 키포인트와 SMPL-X L_SMPL-X 손실을 사용한다: here

is the convolution block

deblur image by

is generated from

has the same amount of layers as feature fusion block

Is

and

(

) is introduced to dynamically fuse the connected features between them. Also to estimate camera and SMPL-X parameters

the output of

supplied to To learn the module, we use 3D L _3D and 2D L _2D keypoints and SMPL-X L _SMPL-X loss:

(8)

(8)

는 각 손실함수에 대한 가중치이다. 본 발명의 실시예에 따른 구현에서, 모든 항은 동일하게 가중된다. here

is the weight for each loss function. In implementations according to embodiments of the present invention, all terms are weighted equally.

5 is an algorithm illustrating how replacement training is performed according to one embodiment of the present invention.

와 통합적 인체 재구성

의 두 가지 경사도(gradient)가 있으며, 여기서 구조 인식 모듈

의 경사도는

에 포함된다. According to an embodiment of the present invention, instead of learning the framework in an end-to-end manner, we apply replacement learning in which the information of each module is used by other modules. Algorithm 1 shows how substitution learning is performed. deblurring

and integrated human body reconstruction

There are two gradients of , where the structure recognition module

the slope of

included in

The present invention is the first technique to simultaneously address human motion blur datasets using 3D anatomical parameters as well as deblurring and integrated 3D anatomical reconstruction. According to an embodiment of the present invention, the new SINBlur dataset can provide a synthetic InstaVariety blur dataset useful for solving integral 3D anatomical reconstructions in blurred images. In addition, we propose a new joint framework for integrated 3D human body reconstruction and human body motion deblurring, elaboration of a structure recognition module for reconstruction networks and human reconstruction loss functions for deblurring networks, and significant performance improvements compared to state-of-the-art methods. can be achieved.

The devices described above may be implemented as hardware components, software components, and/or a combination of hardware components and software components. For example, devices and components described in the embodiments may include, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable array (FPA), It may be implemented using one or more general purpose or special purpose computers, such as a programmable logic unit (PLU), microprocessor, or any other device capable of executing and responding to instructions. A processing device may run an operating system (OS) and one or more software applications running on the operating system. A processing device may also access, store, manipulate, process, and generate data in response to execution of software. For convenience of understanding, there are cases in which one processing device is used, but those skilled in the art will understand that the processing device includes a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it can include. For example, a processing device may include a plurality of processors or a processor and a controller. Other processing configurations are also possible, such as parallel processors.

Software may include a computer program, code, instructions, or a combination of one or more of the foregoing, which configures a processing device to operate as desired or processes independently or collectively. You can command the device. Software and/or data may be any tangible machine, component, physical device, virtual equipment, computer storage medium or device, intended to be interpreted by or provide instructions or data to a processing device. can be embodied in Software may be distributed on networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer readable media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. Program commands recorded on the medium may be specially designed and configured for the embodiment or may be known and usable to those skilled in computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - includes hardware devices specially configured to store and execute program instructions, such as magneto-optical media, and ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter, as well as machine language codes such as those produced by a compiler.

As described above, although the embodiments have been described with limited examples and drawings, those skilled in the art can make various modifications and variations from the above description. For example, the described techniques may be performed in an order different from the method described, and/or components of the described system, structure, device, circuit, etc. may be combined or combined in a different form than the method described, or other components may be used. Or even if it is replaced or substituted by equivalents, appropriate results can be achieved.

Therefore, other implementations, other embodiments, and equivalents of the claims are within the scope of the following claims.

<References>

[1] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, pages 10975- 10985, 2019.

[2] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks. In CVPR, pages 5745-5753, 2019.

[3] Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. Learning 3D human dynamics from video. In CVPR, pages 5614-5623, 2019.

[4] Huaizu Jiang, Deqing Sun, Varun Jampani, Ming-Hsuan Yang, Erik G. Learned-Miller, and Jan Kautz. Super slomo: High quality estimation of multiple intermediate frames for video interpolation. In CVPR, pages 9000-9008, 2018.

[5] Kevin Lin, Lijuan Wang, Kun Luo, Yinpeng Chen, Zicheng Liu, and Ming-Ting Sun. Cross-domain complementary learning using pose for multi-person part segmentation. IEEE TCSVT, 31(3):1066-1078, 2021.

[6] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: Realtime multi-person 2D pose estimation using part affinity fields. IEEE TPAMI, 43(1):172-186, 2019.

[7] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, pages 10975- 10985, 2019.

[8] Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop, 2013.

[9] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multiperson linear model. ACM TOG, 34(6):248:1-248:16, 2015.

Claims (8)

generating a synthetic dataset for motion deblurring and 3D human body model reconstruction through a dataset generation module;
generating a motion deblurred image through a motion deblurring module using the dataset, and learning the motion deblurred image using a loss function;
providing structural information for integrated human body reconstruction through a structure recognition module using the learned motion deblurred image; and
Reconstructing a 3D human body model through a human body reconstruction module using the deblurred image with the structural information.
including,
The step of generating a motion deblurred image through a motion deblurring module using the dataset and learning the motion deblurred image using a loss function,
Generating a motion deblurred image using a per-pixel mean error loss function, a perceptual loss function, an adversarial loss function for both global and local discriminators, and an anatomical reconstruction loss function.
A method for restoring a 3D human body model. According to claim 1,
Generating a synthetic dataset for motion deblurring and 3D human body model reconstruction through the dataset generation module,
apply video frame interpolation to increase the frame rate before the average image is computed;
segmenting the body region during average image calculation to generate a body mask for reducing the blur effect on the background region;
A synthetic dataset for motion deblurring and 3D human body model reconstruction by generating pseudo ground truth 2D keypoints to obtain integrated 3D parameters and filtering data by generating pseudo ground truth parameters and 3D keypoints ( to generate a synthetic dataset)
A method for restoring a 3D human body model. delete generating a synthetic dataset for motion deblurring and 3D human body model reconstruction through a dataset generation module;
generating a motion deblurred image through a motion deblurring module using the dataset, and learning the motion deblurred image using a loss function;
providing structural information for integrated human body reconstruction through a structure recognition module using the learned motion deblurred image; and
Reconstructing a 3D human body model through a human body reconstruction module using the deblurred image with the structural information.
including,
The step of reconstructing a 3D human body model through a human body reconstruction module using the deblurred image with the structure information,
Estimating projection matrices and integral 3D parameters from the deblurred image, concatenating structure features and image features as inputs to regress the integral 3D parameters;
The image features are generated from the deblur image by a convolution block, using feature fusion blocks to dynamically fuse structural features and connection features between image features.
A method for restoring a 3D human body model. A dataset generation module for generating a synthetic dataset for motion deblurring and 3D human body model reconstruction;
a motion deblurring module for generating a motion deblurred image using the dataset and learning the motion deblurred image using a loss function;
a structure recognition module providing structure information for integrated human body reconstruction using the learned motion deblurred image; and
A human body reconstruction module that reconstructs a 3D human body model using the deblurred image with the structural information.
including,
The motion deblurring module,
Generating a motion deblurred image using a per-pixel mean error loss function, a perceptual loss function, an adversarial loss function for both global and local discriminators, and an anatomical reconstruction loss function.
3D human body model restoration device. According to claim 5,
The dataset generation module,
apply video frame interpolation to increase the frame rate before the average image is computed;
segmenting the body region during average image calculation to generate a body mask for reducing the blur effect on the background region;
A synthetic dataset for motion deblurring and 3D human body model reconstruction by generating pseudo ground truth 2D keypoints to obtain integrated 3D parameters and filtering data by generating pseudo ground truth parameters and 3D keypoints ( to generate a synthetic dataset)
3D human body model restoration device. delete A dataset generation module for generating a synthetic dataset for motion deblurring and 3D human body model reconstruction;
a motion deblurring module for generating a motion deblurred image using the dataset and learning the motion deblurred image using a loss function;
a structure recognition module providing structure information for integrated human body reconstruction using the learned motion deblurred image; and
A human body reconstruction module that reconstructs a 3D human body model using the deblurred image with the structural information.
including,
The human body reconstruction module,
Estimating projection matrices and integral 3D parameters from the deblurred image, concatenating structure features and image features as inputs to regress the integral 3D parameters;
The image features are generated from the deblur image by a convolution block, using feature fusion blocks to dynamically fuse structural features and connection features between image features.
3D human body model restoration device.

Images