KR20230081378A - Multi-view semi-supervised learning for 3D human pose estimation - Google Patents

Abstract

을 입력받고, 다시점 영상에 대한 의사 참값(P-GT)

를 출력하는 P-GT 생성부; 상기 P-GT 생성부에서 생성된 P-GT와 상기 다시점 영상과 함께 입력되고 단시점 모델의 학습에 사용되는 단시점 모델 학습부; 그리고 상기 단시점 모델 학습부에서 학습된 단시점 모델은 입력된 단시점 영상으로부터의 3차원 휴먼 자세 추정에 사용되는 3차원 휴먼 자세 추정부를 포함하며, 상기 단시점 영상은 3차원 휴먼 자세 추정부의 학습된 단시점 모델에 입력되어 3차원 휴먼 자세 추정 결과를 출력한다.
3차원 휴먼 자세 추정 모델은 다시점 모델과 단시점 모델로 분류된다. 일반적으로 다시점 모델은 단시점 모델에 비하여 뛰어난 자세 추정 성능을 보인다. 단시점 모델의 경우 3차원 자세 추정 성능의 향상은 많은 양의 학습 데이터를 필요로 한다. 우리는 다시점 모델로부터 다시점 휴먼 자세 데이터에 대한 의사 참값을 생성하고, 이를 단시점 모델의 학습에 활용하는 방법을 제안한다. 또한, 각각의 다시점 영상으로부터 추정된 자세의 일관성을 고려하는 다시점 일관성 손실함수를 제안하여, 단시점 모델의 효과적인 학습에 도움을 준다는 것을 보인다.A multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation is disclosed. The system is a multi-view image without a true value (GT)

is input, and the pseudo true value (P-GT) for the multi-view image

a P-GT generation unit that outputs; a single-view model learning unit inputted together with the P-GT generated by the P-GT generation unit and the multi-view image and used for learning a single-view model; The single-view model learned by the single-view model learning unit includes a 3D human posture estimator used for estimating a 3D human posture from the input single-view image, and the single-view image is learned by the 3D human posture estimator. 3D human posture estimation result is output.
3D human posture estimation models are classified into multi-view models and single-view models. In general, multi-view models show superior posture estimation performance compared to single-view models. In the case of single-view models, improvement of 3D posture estimation performance requires a large amount of training data. We propose a method for generating pseudo true values for multi-view human posture data from multi-view models and using them for learning single-view models. In addition, we propose a multi-view coherence loss function that considers the coherence of postures estimated from each multi-view image, and shows that it helps effective learning of single-view models.

Description

Multi-view semi-supervised learning for 3D human pose estimation {Multi-view semi-supervised learning for 3D human pose estimation}

The present invention relates to a multi-view semi-supervised learning system of a single-view model for 3D human pose estimation.

This thesis is a study conducted with the support of the Information and Communication Planning and Evaluation Institute with financial resources from the government (Ministry of Science and ICT) in 2021 (No. 2021-0-00348, Cloud-based unmanned store environment response using 2D/3D video integrated analysis Hyungyeong video security system development)

Extracting 3D skeleton information about a user quickly and reliably is one of the most important problems in the field of computer vision. 3D skeletal information extraction technology is used for human computer interaction (HCI), motion capture for computer graphics, gesture/action recognition for video surveillance, and health care. It has various application fields such as In particular, 3D skeleton information extraction technology has recently attracted a lot of attention as a key technology enabling a more natural user interface for smart devices or interactive digital contents.

The 3D human pose estimation method can be largely divided into a multi-view model and a single-view model.

A multi-view model that uses images from multiple camera viewpoints for one posture as input can estimate a more accurate posture than a single-view model. The reason is that the multi-view model can learn a model that is robust to the depth ambiguity problem and the occlusion problem according to the video viewpoint when estimating the 3D human posture from the multi-view image.

The single-view model is a method of estimating the three-dimensional human posture from a single-view image input, and has shown a significant increase in performance with the recent development of deep learning. However, it is still vulnerable to depth ambiguity and occlusion problems compared to multi-view models. Improving the performance of the single-view model requires a large amount of refined data including various viewpoints and postures. However, obtaining ground-truth (GT) data for a three-dimensional posture generally requires a lot of time and money.

As prior art 1 related to this, Patent Publication No. 10-2015-0061488 discloses “a method and apparatus for estimating a 3D user posture”.

The 3D user posture estimation apparatus extracts a user region corresponding to the user's body from a depth image of the user, and determines a center point in the skeletal joint region by using a randomized decision forest for each pixel in the user region. Then, the user's posture is estimated from the skeleton information after generating skeleton information using the center point in the skeleton joint area.

A 3D user posture estimating method is a method of estimating a 3D user posture using a 3D user posture estimating apparatus, comprising: extracting a user region corresponding to the user's body from a depth image of the user; determining a central point in the skeletal joint region using a randomized decision forest for each pixel in the user region; and generating skeletal information using a center point in the skeletal joint region.

3D human posture estimation is to express the posture of a person existing in an image with joint coordinates in a 3D space. The 3D human posture estimation model can be classified into a single-view model receiving a single-view image input and a multi-view model receiving a multi-view image input.

1 shows a rough example of each method of a single-view model and a multi-view model. The single-view model receives an image from a single view point and outputs a 3D human posture. The short-view model is vulnerable to the problem of depth ambiguity, which makes it difficult to estimate the depth value of 3D joint coordinates, and is not robust even when a person in the image is covered by another object. On the other hand, the multi-view model is trained with multi-view images, and since images from different viewpoints for one posture are input, a model that is robust to depth ambiguity and occlusion problems is learned. Therefore, the multi-view model can estimate a more accurate posture than the single-view model. Improving the performance of the single-view model requires a large amount of data including various viewpoints and postures. However, obtaining data including a three-dimensional attitude ground-truth (GT) requires a lot of time and money.

Patent Publication No. 10-2015-0061488 (published on June 04, 2015), "Method and apparatus for estimating 3D user posture", Korea Electronics and Telecommunications Research Institute

[1] K. Iskakov, E. Burkov, V. Lempisky, and Y. Malkov, "Learnable triangulation of human pose," ICCV, 2019. [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, "Imagenet classification with deep convolutional neural networks," NIPS, 2012. [3] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CVPR, 2016. [4] X. Sun, B. Xiao, F. Wei, S. Liang, and Y. Wei, “Integral human pose regression,” ECCV, 2018. [5] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu, "Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol . 36, 2013. [6] S. Park and N. Kwak, "3D human pose estimation with relational networks," BMVC, 2018. [7] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis, "Coarse-to-fine volumetric prediction for single-image 3D human pose," CVPR, 2017. [8] J. C. Gower, "Generalized procrustes analysis," Psychometrika, vol. 40, no. 2, 1975. [9] D. P. Kingma and L. J. Ba, “Adam: A method for stochastic optimization,” ICLR, 2015. [10] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch, ” NIPS Workshops, 2017

An object of the present invention to solve the above problems is to propose a semi-supervised learning system for a single-view 3D human posture estimation system using camera-calibrated multi-view image data without a GT, and to provide a 3D human posture estimation method in a quantitative, Provides a multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation, in which the P-GT obtained from the multi-view model is qualitatively evaluated and the P-GT obtained from the multi-view model shows learning and performance improvement of the single-view model will be.

을 입력받고, 다시점 영상에 대한 의사 참값(P-GT)

를 출력하는 P-GT 생성부; 상기 P-GT 생성부에서 생성된 P-GT와 상기 다시점 영상과 함께 입력되고 단시점 모델의 학습에 사용되는 단시점 모델 학습부; 그리고 상기 단시점 모델 학습부에서 학습된 단시점 모델은 입력된 단시점 영상으로부터의 3차원 휴먼 자세 추정에 사용되는 3차원 휴먼 자세 추정부를 포함하며, In order to achieve the object of the present invention, a multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation is a multi-view image without a true value (GT).

is input, and the pseudo true value (P-GT) for the multi-view image

a P-GT generation unit that outputs; a single-view model learning unit inputted together with the P-GT generated by the P-GT generation unit and the multi-view image and used for learning a single-view model; The single-view model learned in the single-view model learning unit includes a 3-dimensional human posture estimation unit used for estimating a 3-dimensional human posture from an input single-view image,

The single-view image is input to the learned single-view model of the 3D human posture estimator to output a 3D human posture estimation result.

The multi-view semi-supervised learning system of the single-view model for 3-dimensional human posture estimation according to the present invention quantitatively and qualitatively evaluates the 3-dimensional human posture estimation method. And the P-GT obtained from the multi-view model from the evaluation results is used to learn and improve the performance of the single-view model.

This study proposes a semi-supervised learning method using a calibrated unlabeled multi-view dataset to improve the performance of a single-view model for 3D posture estimation of human objects. The proposed method generates P-GT by applying a multi-view model to multi-view data and uses it to fine-tune the single-view model. In addition, we propose a multi-view coherence loss function that considers the consistency of 3D human posture estimation for multi-view input images. Through experiments, we confirmed that the P-GT and the multi-view coherence loss function generated by the existing pretrained multi-view model quantitatively and qualitatively improve the performance of the single-view model.

1 is a multi-view model and a single-view model
2 is an overview of a 3D human posture estimation system according to the present invention
3 is a block diagram of a 3D human posture estimation system according to an embodiment of the present invention.
Figure 4 is a qualitative comparison of baseline, proposed method, and GT

Hereinafter, the configuration and operation of a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the description of the present invention, if it is determined that a detailed description of a related known technology or known configuration may unnecessarily obscure the subject matter of the present invention, the detailed description thereof will be omitted. In addition, the attached drawing numbers refer to the same drawing numbers in other drawings when indicating the same configuration.

The 3D human posture estimation model can be classified into a multi-view model and a single-view model. In general, multi-view models show superior posture estimation performance compared to single-view models. In the case of single-view models, improvement of 3D posture estimation performance requires a large amount of training data. However, it is not easy to obtain the true value of the 3D posture. To address this problem, we propose a method for generating pseudo true values for multi-view human posture data from multi-view models and using them for learning single-view models. In addition, we propose a multi-view coherence loss function that considers the coherence of the posture estimated from each multi-view image, and shows that it helps the effective learning of single-view models.

In this study, we assume a calibrated multi-view dataset in which 3D posture GT is not provided (unlabeled), and propose a method to improve the performance of a single-view model using this dataset. The basic idea is to apply a pre-learned multi-view model [1] to an unlabeled multi-view dataset, and the estimation result is a pseudo-true value (P-GT) for 3D human poses corresponding to multi-view images. As such, it is used for learning a single-point model. In addition, we propose a multi-view consistency loss function that gives consistency to the posture estimation results of a single-view model for multi-view images. This improves the 3D depth estimation performance of the single-view model and the human posture estimation performance when occlusion occurs.

We quantitatively and qualitatively evaluate the proposed 3D human posture estimation method. And from the evaluation results, it is shown that the P-GT obtained from the multi-view model can be used for learning and performance improvement of the single-view model.

The present invention proposes a semi-supervised learning system for a single-view 3D human posture estimation system using camera-calibrated multi-view image data without a true value (GT).

2 shows an overview of the semi-supervised learning system of a single-view 3D human posture estimation system using multi-view image data.

(101)을 입력받고, 다시점 영상에 대한 의사 참값(P-GT)

를 출력하는 P-GT 생성부(102); The multi-view semi-supervised learning system of the single-view model for 3-dimensional human posture estimation of the present invention is a multi-view image without a true value (GT).

(101) is input, and the pseudo true value (P-GT) for the multi-view image

a P-GT generation unit 102 that outputs;

a single-view model learning unit 103 inputted together with the P-GT generated by the P-GT generation unit 102 and the multi-view image 101 and used for learning a single-view model; and

The single-view model learned in the single-view model learning unit 103 includes a 3-dimensional human posture estimation unit 108 used for 3-dimensional human posture estimation from the input single-view image 107,

The single-view image 107 is input to the learned single-view model of the 3D human posture estimator 108 to output a 3D human posture estimation result.

는 P-GT 생성부(102)에 입력된다. P-GT 생성부(102)는 다시점 영상(101)에 대한 의사 참값(P-GT; pseudo-GT)

를 출력한다. 이 때,

는

개의 3차원 관절 좌표이며 시점

에서 촬영된 영상에 대응하는 P-GT이다. 생성된 P-GT는 다시점 영상과 함께 단시점 모델 학습부(103)로 입력되어 단시점 모델의 학습에 사용된다. 마지막으로 다시점 영상으로부터 학습된 단시점 모델은 3차원 휴먼 자세 추정부(108)로써 단시점 영상을 입력 받아 시스템의 3차원 휴먼 자세 추정 결과를 출력한다.A detailed procedure of the system proposed by the present invention is represented by a block diagram in FIG. 3 . First, a multi-view image without a true value (GT)

is input to the P-GT generation unit 102. The P-GT generating unit 102 generates a pseudo-true value (P-GT) for the multi-view image 101.

outputs At this time,

Is

The three-dimensional joint coordinates of the dog and the viewpoint

This is the P-GT corresponding to the video taken in . The generated P-GT is input to the single-view model learning unit 103 together with the multi-view image and used for learning the single-view model. Finally, the single-view model learned from the multi-view image receives the single-view image as the 3D human posture estimator 108 and outputs the 3D human posture estimation result of the system.

를 출력한다. The P-GT generation unit 102 generates P-GTs for images of each viewpoint from the multi-view image 101 . The P-GT generation unit 102 is composed of a pretrained multi-view model and a P-GT generation process through regularization. The multi-view model receives the multi-view image 101 and is implemented with a 2-dimensional joint coordinate output network composed of an artificial neural network and an algebraic triangulation method. The multi-view model has J three-dimensional joint coordinates defined in the world coordinate system from the input.

outputs

로부터 단시점 모델의 학습에 활용할 수 있는 P-GT를 생성하기 위해

를 각 시점

에 대한 영상

에 대응하는 히트맵 공간으로 정규화(normalization)한다. 이는 캘리브레이션된 카메라 파라메터를 사용한

의

에 대한 원근 투영(perspective projection), 픽셀 좌표에 대한 정규화, 그리고 깊이 좌표에 대한 정규화 과정으로 이루어진다. 먼저,

가

의

번째 관절이라고 할 때,

의

에 대한 원근 투영을 위하여

를 카메라 좌표계로 변환한다. 이 과정은 다음 식과 같이 수행된다.to the next

To generate a P-GT that can be used for learning a single-point model from

at each point

video about

Normalize to the heatmap space corresponding to . This is done using calibrated camera parameters.

of

It consists of perspective projection for , normalization for pixel coordinates, and normalization for depth coordinates. first,

go

of

When referring to the second joint,

of

for the perspective projection of

to the camera coordinate system. This process is performed as follows.

와

는 시점 c에 대한 카메라 외부 파라메터를 의미한다. 이를 통해 월드 좌표계의 3차원 좌표

는 카메라 좌표계로 변환된다. 그 후, 카메라 내부 파라메터

를 사용하여

가

에 원근 투영된 2차원 관절 좌표

를 획득한다. 이는 다음 식과 같이 수행된다.in the above expression

and

denotes an external parameter of the camera for the point in time c. Through this, the three-dimensional coordinates of the world coordinate system

is converted to the camera coordinate system. After that, camera internal parameters

use with

go

2D joint coordinates projected in perspective

Acquire This is done as follows.

Normalization of the obtained two-dimensional coordinates is performed through the following two equations.

는 입력 영상과 히트맵 크기 사이의 비율인 4로 나누어져 입력 영상보다 4배 작은 히트맵 공간에서의

좌표가 된다. 이 때 입력 영상의 크기는

, 히트맵의 크기는

이다. 깊이 좌표에 대한 정규화는 다음 식과 같이 수행된다.

is divided by 4, the ratio between the input image and the heatmap size, and in the heatmap space 4 times smaller than the input image

becomes the coordinates. In this case, the size of the input image is

, the size of the heatmap is

am. Normalization for depth coordinates is performed as follows.

이하임을 가정한다.

에서 골반 관절의 깊이 값인

를 빼서 골반 관절을 기준으로 상대적으로 정의되는 깊이 값을 얻는다. 이러한 관절 좌표의 깊이 값은

의 범위에 존재한다. 또한 정규화된 깊이 값이 히트맵의 뎁스 축 범위인

내에 존재하도록 만든다. 마지막으로 정규화된

를 합쳐서 우리는 정규화된 관절 좌표

를 얻는다.To normalize the depth coordinates of each joint, we first determine the size of the human object

Assume below

is the depth value of the pelvic joint in

is subtracted to obtain a depth value defined relative to the pelvic joint. The depth values of these joint coordinates are

exists in the range of Also, the normalized depth value is the range of the depth axis of the heatmap.

make it exist within last normalized

By summing, we get the normalized joint coordinates

get

를 출력한다.By performing the above normalization at each joint and all viewpoints, the P-GT generation unit calculates the P-GT for each viewpoint of the multi-view image.

outputs

를 구성하는

개의 영상을 각각 입력받아 자세 추정 결과인

를 출력한다. 단시점 모델은 인공 신경망으로 구성된 encoder

, decoder

로 이루어지며 입력은

와

를 거쳐 3차원 히트맵을 생성한다. 마지막으로 생성된 히트맵에 soft-argmax를 적용하여 단시점 모델은 각 관절의 3차원 좌표를 출력한다. 이제 모델의 학습을 위하여 출력된 관절 좌표와 입력 영상에 대응하는 P-GT 또는 GT에 기반하여 L1 손실함수를 계산한다. 또한,

의 각 영상에서 얻어진 모든 휴먼 자세가 같은 자세임을 보장하기 위하여 모델의 각 추정 결과들을 월드 좌표계로 변환하고 모든 추정 결과들 사이에 L1 손실함수를 계산한다. 월드 좌표계로의 변환은 정규화 과정의 역연산으로 수행된다. 자세 추정 결과들의 일관성을 보장하는 손실함수를 본 발명에서는 '다시점 일관성 손실함수'로 정의한다. 두 손실함수의 가중 합으로부터 역전파 알고리즘을 통해 단시점 모델을 학습한다.The single-view model learning unit 103 learns a single-view model using the multi-view image 101 and the generated P-GT. At this time, not only the P-GT but also multi-view image data in which available GTs exist can be utilized together. The single point model is

constituting

Each image is input and the posture estimation result is

outputs The single-point model is an encoder composed of artificial neural networks.

, decoder

and the input is

and

to generate a 3D heat map. Finally, by applying soft-argmax to the generated heatmap, the single-view model outputs the 3D coordinates of each joint. Now, for model learning, the L1 loss function is calculated based on the output joint coordinates and the P-GT or GT corresponding to the input image. also,

In order to ensure that all human postures obtained from each image of are the same posture, each estimation result of the model is converted into a world coordinate system, and an L1 loss function is calculated between all estimation results. Conversion to the world coordinate system is performed as an inverse operation of the normalization process. A loss function that ensures consistency of attitude estimation results is defined as a 'multi-view consistency loss function' in the present invention. A single-point model is learned from the weighted sum of the two loss functions through the backpropagation algorithm.

The single-view model learned in the single-view model learning unit 103 is used in the 3-dimensional human posture estimation unit 108 to estimate the 3-dimensional human posture from the single-view image 107 . The short-view image 107 is input to the learned single-view model of the 3D human posture estimator 108 to output a 3D human posture estimation result.

2. How to propose

The approximate procedure for using unlabeled multi-view image datasets to improve the performance of the single-view model proposed in this study is as follows. First, a P-GT is generated from a pretrained multi-view model. Second, we initialize a single-point model with a pre-trained model on a labeled dataset containing GT. Third, a single-view model is additionally trained using an unlabeled multi-view image dataset containing P-GT and a labeled dataset containing GT together. At this time, the single-view model is optimized with the proposed multi-view coherence loss function.

2.1. Pre-training of single-point models

를 만든다. 각 deconvolution 층의 필터 크기와 stride는 각각 4와 2로 설정한다. 단시점 모델은

의 출력 텐서에 soft-argmax[4] 연산을 적용하여 3차원 휴먼 자세를 구성하는 관절 좌표를 획득한다. The single-view model that performs 3D human posture estimation in this study is an integral regression model [4] with ResNet-50 [3] pretrained with ImageNet [2] as a backbone. To construct a single-point model, a fully convolutional network was created by replacing the global average pooling layer and the fully connected layer with three consecutive deconvolution layers and one 1x1 convolution layer in the ResNet-50 model.

makes The filter size and stride of each deconvolution layer are set to 4 and 2, respectively. The single point model is

The joint coordinates constituting the 3D human posture are obtained by applying the soft-argmax[4] operation to the output tensor of .

는 입력 영상

을 입력 받아

개 관절에 대한 3차원 히트맵(heatmap)

를 출력한다. 그 후,

에 soft-argmax를 적용하여 각 관절의 3차원 좌표

를 획득한다. network specifically

is the input image

take input

3D heatmap for dog joints

outputs After that,

3D coordinates of each joint by applying soft-argmax to

Acquire

번째 관절에 대한 3차원 히트맵

를 확률 분포

로 만들기 위해 softmax 연산을 적용하는 과정을 나타낸다:Soft-argmax operation is a method of obtaining spatial coordinates by making the heat map into a probability distribution and calculating the expected value.

3D heatmap for the joint

the probability distribution

shows the process of applying the softmax operation to make

(1)

(One)

에 다음과 같은 기댓값 연산을 적용하여 특징점 좌표

를 획득한다:After that

by applying the following expected value operation to

get:

(2)

(2)

를 획득한다. 단시점 모델의 사전 학습을 위해 우리는 모델이 출력한

에 labeled 데이터셋의 GT에 기반한 L1 손실 함수를 적용한다.We applied Equation (2) to all joints to obtain a 3D human posture.

Acquire For pre-training of the single-point model, we

We apply the L1 loss function based on the GT of the labeled dataset to .

2.2. Create P-GT dataset

로부터

개 관절들의 좌표

를 추정한다. 다시점 모델

는 [1]의 algebraic triangulation 모델을 사전 학습하여 사용한다.We use a pre-trained multi-view model to utilize an unlabeled multi-view image dataset.

from

Coordinates of Dog Joints

to estimate multi-view model

uses the pre-trained algebraic triangulation model of [1].

, (3)

, (3)

는 unlabeled 다시점 영상 데이터셋의 한 sample이다.

와

는 각각 시점

에서 촬영된 영상과 한 자세를 관찰하는 서로 다른 시점의 개수를 의미한다. 여기서 3차원 휴먼 자세

는 월드 좌표계(world coordinate system)에서 정의된다.here

is a sample of an unlabeled multi-view image dataset.

and

is each point

It means the number of different viewpoints observing the image taken in and one posture. 3D human pose here

is defined in the world coordinate system.

를 단시점 모델의 학습에 활용하기 위해 우리는

를 각 시점

의 영상

에 대응하는 히트맵 공간으로 정규화(normalization)한다. 이는

의

에 대한 원근 투영(perspective projection), 픽셀 좌표에 대한 정규화, 그리고 깊이 좌표에 대한 정규화 과정들로 이루어진다.

In order to use for the training of a single-point model, we

at each point

video of

Normalize to the heatmap space corresponding to . this is

of

It consists of perspective projection for , normalization for pixel coordinates, and normalization for depth coordinates.

를

에 투영하여 2차원 좌표를 획득하고, 픽셀 좌표에 대해 정규화 하는 과정을 나타낸다:Equation is joint coordinates

cast

It shows the process of obtaining two-dimensional coordinates by projecting on and normalizing them to pixel coordinates:

, (4)

, (4)

, (5)

, (5)

(6)

(6)

와

는 카메라

의 외부 파라미터를,

and

the camera

the external parameters of

와

는 카메라

의 외부 파라미터를, 그리고

는 내부 파라미터를 나타낸다. 식 (4)를 통해 월드 좌표계에서 정의된

는 카메라 좌표계에서 정의된 관절 좌표

로 변환된다.

는 식 (5)를 통해 픽셀 좌표

로 투영된다. 마지막으로 식 (6)을 통해 우리는 영상 크기와 히트맵 크기 사이의 비율인 4를 사용하여 정규화된 2차원 좌표

를 얻을 수 있다. 또한 우리는

의 깊이 좌표

를 다음과 같이 정규화한다:

and

the camera

the external parameters of , and

represents an internal parameter. defined in the world coordinate system through equation (4)

is the joint coordinates defined in the camera coordinate system

is converted to

is the pixel coordinate through equation (5)

projected into Finally, through Equation (6), we obtain the normalized two-dimensional coordinates using 4, the ratio between the image size and the heatmap size.

can be obtained. Also, we

depth coordinates of

is normalized as follows:

. (7)

. (7)

이하임을 가정한다.

에서 골반 관절의 깊이 값인

를 빼서 골반 관절을 기준으로 상대적으로 정의되는 깊이 값을 얻는다. 이러한 깊이 값은

의 범위에 존재하므로 우리는 추가적으로, 정규화된 깊이 값이 히트맵의 뎁스 축 범위인

내에 존재하도록 만든다. 이러한 과정은 식 (7)에 나타나 있으며, 이를 통해 우리는 정규화된 깊이 값

를 획득한다. 결국 시점

에 대응하는 히트맵 공간에서 정의되는 P-GT 관절 좌표

는 다음과 같다:To normalize the depth coordinates of each joint, we first determine the size of the human object

Assume below

is the depth value of the pelvic joint in

is subtracted to obtain a depth value defined relative to the pelvic joint. These depth values are

Since it exists in the range of we additionally, the normalized depth value is the range of the depth axis of the heatmap

make it exist within This process is shown in Equation (7), through which we obtain the normalized depth value

Acquire the point at the end

P-GT joint coordinates defined in the heat map space corresponding to

is as follows:

. (8)

. (8)

에 적용하여 우리는 unlabeled sample

에 대한 P-GT

를 획득할 수 있고, 이를 활용하여 우리는 P-GT 데이터셋

을 구성한다.Equation (4)-(8) at each time point

By applying to the unlabeled sample

For P-GT

can be obtained, and by using this, we have the P-GT dataset

make up

2.3. Multi-Point Consistency Loss Function

는 다음과 같다:The proposed method learns a single-view model using multi-view image data. When a single-view model receives a multi-view image corresponding to one posture, the human postures at each view point estimated by the model must take a consistent posture. Therefore, we propose a multi-view consistency loss function to make the trained model satisfy these conditions. The multi-view coherence loss function is defined as the sum of the L1 loss functions between the resulting postures after converting the estimated joint coordinates for each viewpoint into the world coordinate system. The process of converting the joint coordinates normalized to the heat map space into the world coordinate system is performed by the inverse operation of Eqs. (4)-(8). Now the multi-point consistency loss function

is as follows:

, (9)

, (9)

와

는 각각 시점

와

에서 단시점 모델에 의해 추정된 정규화된 관절 좌표를 나타내며,

는 정규화된 관절 좌표를 월드 좌표계 기준으로 변환하는 함수이다.here,

and

is each point

and

Represents the normalized joint coordinates estimated by the single-view model in

is a function that converts normalized joint coordinates to world coordinate system standards.

을 다음과 같이 정의한다:We use a loss function to fine-tune a pretrained single-point model.

is defined as:

, (10)

, (10)

는 단시점 모델의 출력

에 적용하는 P-GT 및 GT에 기반한 L1 손실 함수들의 합으로 정의된다.

와

는 각 손실 함수의 영향력을 결정하는 가중치이다. 단시점 모델의 미세 조정을 위해 우리는 식 (10)을 최소화한다.here

is the output of the single-point model

It is defined as the sum of L1 loss functions based on P-GT and GT applied to .

and

is a weight that determines the influence of each loss function. For fine-tuning of the single-point model, we minimize Eq.

3. Experimental results

3.1. Datasets, evaluation methods, and implementation details

This study uses the Human3.6M [5] dataset, which includes large-scale 3D human postures, to learn and evaluate the proposed method. In the Human3.6M dataset, each human object (hereinafter referred to as S) performs 15 actions, and videos of each human object performing the action are taken with four different viewpoint cameras. We use personal data of 5 out of 11 (S1, S5, S6, S7, S8) people as a learning dataset according to the learning and evaluation methods of previous studies [6, 7]. Among them, the data of 3 students (S1, S5, S6) is assumed to be a labeled dataset, and the data of 2 students (S7, S8) is assumed to be an unlabeled dataset. The data of the other two (S9, S11) is used as an evaluation dataset. We use the evaluation dataset by sub-sampling every 64 frames.

이다.We measure and report MPJPE and PA-MPJPE to quantitatively evaluate the performance of the single-point model. MPJPE represents the Euclidean distance between a joint and its GT estimated by the single-point model in the evaluation dataset. PA-MPJPE is the value obtained after performing Procrustes alignment [8] between the estimated joint and GT. The units of MPJPE and PA-MPJPE are

am.

Pre-training of multi-view and single-view models is performed with labeled datasets. For the efficiency of learning when creating the P-GT dataset, the posture estimation result by applying the pretrained multi-view model is stored offline. After that, it is used to fine-tune the single-view model.

로 설정한다. 단시점 모델의 사전 학습 또한 같은 학습 데이터셋이 사용되며, 에포크 수, 배치 크기, 학습률은 각각 20, 32,

이다. 두 모델의 사전 학습에 사용된 optimizer는 Adam[9] 이다.A labeled dataset is used for pre-training of the multi-view model, and the number of epochs, batch size, and learning rate are 6, 8, and 8, respectively.

set to The same training dataset is used for pre-training of the single-point model, and the number of epochs, batch size, and learning rate are 20, 32, and 32, respectively.

am. The optimizer used for pre-training of the two models is Adam [9].

For fine-tuning of the single-point model, we train for 9 epochs on the labeled and P-GT datasets.

로 설정한다. Optimizer로는 Adam을 사용한다. 손실 함수의 가중치는

과

로 설정한다. 제안하는 모델은 Pytorch[10] 프레임워크를 사용하여 구현되었다.In this case, the batch size and learning rate are 6 and

set to Adam is used as the optimizer. The weights of the loss function are

class

set to The proposed model was implemented using the Pytorch [10] framework.

3.2 Quantitative results

We quantitatively compare the two baseline models and the proposed method (Ours) to show that the proposed method helps improve the performance of the single-point model. As a baseline model, the following two methods are used. The first is a single-point model (Base) trained with the GT dataset, and the second is a model (L1-only) that applies only the L1 loss function to the estimation results of P-GT and each time point without applying the multi-view coherence loss function. am.

Table 1 shows the quantitative performance comparison of the baseline methods and the proposed method. We presented evaluation results for each camera view of the Human3.6M evaluation dataset to show performance improvement for multi-viewpoint data. We can see that the L1-only baseline shows higher performance than the baseline. This result shows that the P-GT generated by the existing multi-view model has an accuracy that is conducive to learning. Also, the proposed method shows higher performance than the two baseline models for all camera views. From these results, we confirmed that the P-GT dataset and the multi-view coherence loss function can substantially improve the 3D human posture estimation performance of the single-view model.

3.3 Qualitative results

로 설정한다. Optimizer로는 Adam을 사용한다. 손실 함수의 가중치는

과

로 설정한다. 제안하는 모델은 Pytorch[10] 프레임워크를 사용하여 구현되었다.In this case, the batch size and learning rate are 6 and

set to Adam is used as the optimizer. The weights of the loss function are

class

set to The proposed model was implemented using the Pytorch [10] framework.

Figure 4 is a qualitative comparison of baseline, proposed method, and GT

4. Conclusion

This study proposes a semi-supervised learning method using a calibrated unlabeled multi-view dataset to improve the performance of a single-view model for 3D posture estimation of human objects. The proposed method generates P-GT by applying a multi-view model to multi-view data and uses it to fine-tune the single-view model. In addition, we propose a multi-view coherence loss function that considers the consistency of 3D human posture estimation for multi-view input images. Through experiments, we confirmed that the P-GT and the multi-view coherence loss function generated by the existing pretrained multi-view model quantitatively and qualitatively improve the performance of the single-view model.

Embodiments according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable recording medium. The computer readable recording medium may include program instructions, data files, and data structures alone or in combination. Computer-readable recording media include storage, hard disks, magnetic media such as floppy disks and magnetic tapes, optical media such as CD-ROMs and DVDs, and magnetic media such as floptical disks. - A hardware device configured to store and execute program instructions in storage media such as magneto-optical media, ROM, RAM, flash memory, storage, etc. may be included. Examples of program instructions may include those produced by a compiler, machine language codes as well as high-level language codes that can be executed by a computer using an interpreter. The hardware device may be configured to operate as one or more software modules to perform the operations of the present invention.

As described above, the method of the present invention is implemented as a program and can be read using computer software on a recording medium (CD-ROM, RAM, ROM, memory card, hard disk, magneto-optical disk, storage device, etc.) ) can be stored in

Although described with reference to specific embodiments of the present invention, the present invention is not limited to the same configuration and operation as the specific embodiments to illustrate the technical idea as described above, and within the limit that does not deviate from the technical spirit and scope of the present invention It can be implemented with various modifications, and the scope of the present invention should be determined by the claims described later.

101: multi-view image 102: P-GT generator (multi-view model)
103: short-view model learning unit 107: short-view image
108: 3-dimensional human posture estimator (learned single-view model)
109: 3D human posture estimation result

Claims (7)

Multi-view image without true value (GT)

is input, and the pseudo true value (P-GT) for the multi-view image

a P-GT generation unit that outputs;
a single-view model learning unit inputted together with the P-GT generated by the P-GT generation unit and the multi-view image and used for learning a single-view model; and
The single-view model learned in the single-view model learning unit includes a 3-dimensional human posture estimation unit used for estimating a 3-dimensional human posture from an input single-view image,
The multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation, wherein the single-view image is input to the learned single-view model of the 3-dimensional human posture estimator and outputs a 3-dimensional human posture estimation result. According to claim 1,

is the coordinates of j three-dimensional joints and the viewpoint

It is a pseudo true value (P-GT) corresponding to the image captured in the generated P-GT together with the multi-viewpoint image and is input to the single-viewpoint model learning unit and used for learning the single-viewpoint model, 3D human posture estimation. A multi-view semi-supervised learning system of a single-view model for According to claim 1,
The P-GT generation unit includes a pretrained multi-view model and a P-GT generation process through normalization, and generates a P-GT for an image of each view from the multi-view image,
The multi-view model receives the multi-view image and is implemented with a 2-dimensional joint coordinate output network composed of an artificial neural network and an algebraic triangulation method. 3D joint coordinates

outputs,
to the next

To generate a P-GT that can be used for learning a single-point model from

at each point

video about

Normalize to the heat map space corresponding to , which uses calibrated camera parameters.

of

Consists of perspective projection for , normalization for pixel coordinates, and normalization for depth coordinates, first,

go

When the j-th joint of

of

for the perspective projection of

is converted to the camera coordinate system, and this process is performed as follows,

in the above expression

and

Means an external parameter of the camera for the viewpoint c, through which the three-dimensional coordinates of the world coordinate system

is converted to the camera coordinate system, and then, camera internal parameters

use with

go

2D joint coordinates projected in perspective

as follows

A multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation. According to claim 3,
Normalization of the acquired two-dimensional coordinates is performed through the following two equations,

is divided by 4, the ratio between the input image and the heatmap size, and in the heatmap space 4 times smaller than the input image

coordinates, and the size of the input image at this time is

, the size of the heatmap is

is,
Normalization for depth coordinates is performed as

To normalize the depth coordinates of each joint, the size of the human object is

Assuming that the following

is the depth value of the pelvic joint in

is subtracted to obtain a depth value defined relative to the pelvic joint, and the depth value of these joint coordinates is

is in the range of and the normalized depth value is the depth axis range of the heatmap

made to exist within, normalized

normalized joint coordinates by summing

get,
By performing the above normalization at each joint and all viewpoints, the P-GT generation unit calculates the P-GT for each viewpoint of the multi-view image.

A multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation, which outputs. According to claim 4,
The single-view model learning unit learns a single-view model using the multi-view image and the generated P-GT, and can utilize not only the P-GT but also multi-view image data in which available GTs exist. the model is

constituting

Each image is input and the posture estimation result is

Outputs , and the single-point model is an encoder composed of an artificial neural network.

, decoder

, and the input is

and

, a 3D heat map is generated, and finally, by applying soft-argmax to the generated heat map, the single view model outputs the 3D coordinates of each joint. semi-supervised learning system.

According to claim 1,
Train a single-view model using multi-view image data,
A labeled dataset is used for pre-training of the multi-view model, and the number of epochs, batch size, and learning rate are 6, 8, and 8, respectively.

set to
The same training dataset is used for pre-learning of the single-point model, and the number of epochs, batch size, and learning rate are 20, 32, and

has been set,
A multi-view semi-supervised learning system of a single-view model for 3-dimensional human posture estimation using Adam as the optimizer used for pre-training of the two models. According to claim 1,
Now, for the learning of the single-view model, the L1 loss function is calculated based on the output joint coordinates and the P-GT or GT corresponding to the input image,
Loss function for fine-tuning a pretrained single-point model

is defined as:

(10)
here,

is the output of the single-point model

It is defined as the sum of L1 loss functions based on P-GT and GT applied to

and

is a weight that determines the influence of each loss function,
also,

In order to ensure that all human postures obtained from each image of are the same posture, each estimation result of the model is converted into a world coordinate system, and an L1 loss function is calculated between all estimation results. , and defines a loss function that guarantees the consistency of the posture estimation results as a 'multi-view coherence loss function' and learns a single-view model through a backpropagation algorithm from the weighted sum of the two loss functions. A multi-view semi-supervised learning system of a single-view model for

Images