KR102459293B1 - Method and apparatus for generating mesh model of human or quadrupeds - Google Patents

Abstract

According to an embodiment, a method of generating a mesh model performed by an apparatus for generating a mesh model of humans or quadrupeds includes the steps of: extracting a feature vector of an object present in an observation image; classifying a class for the object based on the feature vector; obtaining a 3D linear mesh model parameter of quadrupeds as an output of an artificial neural network for estimating a first mesh model parameter by inputting the feature vector to the pre-learned artificial neural network for estimating the first mesh model parameter; obtaining a 3D linear mesh model parameter of a human as an output of an artificial neural network for estimating a second mesh model parameter by inputting the feature vector to the pre-learned artificial neural network for estimating the second mesh model parameter; selecting one of the 3D linear mesh model parameter of the quadrupeds and the 3D linear mesh model parameter of the human according to a result of classifying the species of the object; and generating a 3D mesh model by using the selected 3D linear mesh model parameter. The present invention can estimate the 3D posture and shape of as many species as possible.

Description

Apparatus and method for generating mesh models for humans or quadrupedal animals

The present invention relates to an apparatus for generating a mesh model for a human or a quadrupedal animal, and a method for such an apparatus to generate a mesh model for a human or a quadrupedal animal.

The three-dimensional posture and shape estimation for objects such as people and animals in an image aims to recognize an object appearing in the image, and to estimate a three-dimensional mesh and three-dimensional posture corresponding to each object. Existing methods are limited to three-dimensional posture and shape estimation for a single human species or three-dimensional posture and shape estimation for a single (or few) animal species. The three-dimensional pose and shape estimation for these limited species has limitations when applied to actual industry. In short, for autonomous vehicles, when driving on an unpaved road where human and animal driving routes are not separated, avoidance maneuvering technology for unexpected unexpected objects must be considered. In fact, evasive maneuvers such as 3D LiDAR-based forward collision avoidance (FCA) and crossroads forward collision avoidance (FCA-JT) have been studied and installed in recently sold autonomous vehicle models. However, existing methods have limitations in responding to unlearned objects such as wild animals other than pedestrians by using previously learned pedestrian information for learning.

Korean Patent Registration No. 10-2338491, registration date December 8, 2021.

According to an embodiment, with respect to a feature vector of an object present in an observation image, all three-dimensional linear mesh model parameters of a quadruped animal and three-dimensional linear mesh model parameters of a human are acquired and then selectively used to obtain a three-dimensional mesh A method and apparatus for generating a mesh model are provided for estimating the three-dimensional posture and shape of as many species as possible by generating the model.

The problems to be solved of the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned will be clearly understood by those of ordinary skill in the art to which the present invention belongs from the following description.

According to a first aspect, a method for generating a mesh model performed by an apparatus for generating a mesh model for a human or a quadrupedal animal includes extracting a feature vector of an object existing in an observation image, and the object based on the feature vector. Classifying a class with respect to , and inputting the feature vector into a pre-trained first mesh model parameter estimation artificial neural network as an output of the first mesh model parameter estimation artificial neural network three-dimensional linearity of a quadruped animal obtaining a mesh model parameter; and inputting the feature vector into a pre-trained second mesh model parameter estimation artificial neural network to obtain a human three-dimensional linear mesh model parameter as an output of the second mesh model parameter estimation artificial neural network step, selecting any one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human according to the result of classifying the species with respect to the object; and generating a three-dimensional mesh model using the linear mesh model parameters.

According to a second aspect, an apparatus for generating a mesh model includes an image acquisition unit for acquiring an observed image, and an artificial neural network operation unit for estimating a three-dimensional linear mesh model parameter of an object existing in the observed image through an operation using an artificial neural network; , a data processing unit for processing data including the estimated three-dimensional linear mesh model parameter, wherein the artificial neural network operation unit extracts a feature vector of an object present in the observed image obtained by the image acquisition unit, , classifying the species for the object based on the feature vector, inputting the feature vector into a pre-trained first mesh model parameter estimation artificial neural network, and as an output of the first mesh model parameter estimation artificial neural network, a quadruped animal obtain the 3D linear mesh model parameters of , and input the feature vectors to the pre-trained second mesh model parameter estimation artificial neural network to obtain the human 3-dimensional linear mesh model parameters as the output of the second mesh model parameter estimation artificial neural network. obtaining, and the data processing unit selects any one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human according to the result of classifying the species with respect to the object, A 3D mesh model is generated using the 3D linear mesh model parameters.

The computer-readable recording medium storing the computer program according to the third aspect includes instructions for causing the computer program to cause a processor to perform the mesh model generating method.

The computer program stored in the computer-readable recording medium according to the fourth aspect includes instructions for the computer program to cause a processor to perform the mesh model generating method.

According to the embodiment, with respect to the feature vector of the object present in the observation image, both the 3D linear mesh model parameter of a quadruped animal and the 3D linear mesh model parameter of a human are acquired and then selectively used as a 3D mesh model By creating , it is possible to estimate the three-dimensional posture and shape of as many species as possible. Since it is possible to estimate the three-dimensional postures and shapes of as many species as possible, it can be used for evasive maneuvers when driving an autonomous vehicle, general animal behavior analysis, and future route prediction. In addition, when a new bipedal or quadrupedal walking object that has not been learned in advance but morphologically similar to a human, dog, cow, or horse appears, it is possible to estimate the three-dimensional posture and shape using the common shape features of the pre-learned species. do. In addition, it can be applied to augmented reality (AR), virtual reality (VR), metaverse, graphics, etc., so it can be applied to many computer vision-related research.

1 is a block diagram of an apparatus for generating a mesh model for a human or a quadrupedal animal according to an embodiment of the present invention.
FIG. 2 is a configuration diagram of an artificial neural network calculating unit shown in FIG. 1 .
3 is a flowchart illustrating a mesh model generating method performed by the mesh model generating apparatus for a human or quadrupedal animal according to an embodiment of the present invention.
4 is a diagram illustrating an example of matching sub-joints (sub-keypoints) common to a quadrupedal animal mesh model and a human mesh model.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only these embodiments allow the disclosure of the present invention to be complete, and common knowledge in the technical field to which the present invention pertains It is provided to fully inform those who have the scope of the invention, and the present invention is only defined by the scope of the claims.

Terms used in this specification will be briefly described, and the present invention will be described in detail.

The terms used in the present invention have been selected as currently widely used general terms as possible while considering the functions in the present invention, but these may vary depending on the intention or precedent of a person skilled in the art, the emergence of new technology, and the like. In addition, in a specific case, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the description of the corresponding invention. Therefore, the term used in the present invention should be defined based on the meaning of the term and the overall content of the present invention, rather than the name of a simple term.

Where each film, window, panel, structure, or layer, etc. is described herein as being formed "on" or "under" each film, window, panel, structure, or layer, etc. As such, the terms “on” and “under” include both “directly” or “indirectly” being formed.

In addition, the reference for the upper / lower of each component will be described with reference to the drawings. The size of each component in the drawings may be exaggerated for explanation, and does not mean the size actually applied. Also, like reference numerals refer to like elements throughout.

In the entire specification, when a part 'includes' a certain element, this means that other elements may be further included, rather than excluding other elements, unless otherwise stated.

In the present specification, unless otherwise specified, the expression "a" is construed as meaning including the singular or the plural as interpreted in context.

In addition, it should be understood that all numbers and expressions indicating amounts of components, reaction conditions, etc. described herein are modified by the term "about" in all cases unless otherwise specified.

In this specification, terms such as first, second, etc. are used to describe various components, and the components should not be limited by the terms. The above terms are used for the purpose of distinguishing one component from another component.

In addition, the term 'unit' used in the specification means software or hardware components such as FPGA or ASIC, and 'unit' performs certain roles. However, “part” is not meant to be limited to software or hardware. The 'unit' may be configured to reside on an addressable storage medium or may be configured to refresh one or more processors. Thus, as an example, 'part' refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and ‘parts’ may be combined into a smaller number of components and ‘parts’ or further divided into additional components and ‘parts’.

Hereinafter, with reference to the accompanying drawings, the embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the embodiments of the present invention. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description will be omitted.

1 is a block diagram of an apparatus for generating a mesh model for a human or a quadrupedal animal according to an embodiment of the present invention, and FIG. 2 is a block diagram of an artificial neural network operation unit shown in FIG. 1 .

1 and 2 , the mesh model generating apparatus 100 includes an image acquisition unit 110 , an artificial neural network operation unit 120 , and a data processing unit 130 .

The image acquisition unit 110 obtains an observation image and provides it to the artificial neural network operation unit 120 .

The artificial neural network calculation unit 120 estimates the 3D linear mesh model parameter of the object present in the observed image through calculation using the artificial neural network. The artificial neural network operation unit 120 extracts a feature vector of an object existing in the observed image acquired by the image acquisition unit 110 , classifies a species for the object based on the feature vector, and uses the feature vector previously learned. estimating mesh model parameters for animals by inputting them into an artificial neural network to estimate the mesh model parameters for animals Obtaining the three-dimensional linear mesh model parameters of a quadrupedal animal as an output of the artificial neural network, and estimating the pre-trained mesh model parameters for humans using the feature vector Estimation of second mesh model parameters for humans by input to the artificial neural network Obtain the human 3D linear mesh model parameters as the output of the artificial neural network, and obtain the species classification results for objects and the obtained 3D linear mesh model parameters of the quadruped animal and human 3D linear mesh model parameters are provided to the data processing unit 130 .

Here, when learning the artificial neural network for animal mesh model parameter estimation artificial neural network and the human mesh model parameter estimation artificial neural network in the artificial neural network operation unit 120 , a loss function for evaluating the accuracy of the entire joint for quadrupedal animals and humans is reduced. It may be that the network parameters are learned in the direction. Also, the network parameters may be learned in the direction of learning morphological similarity by matching the sub-joints that the quadrupedal animal mesh model and the human mesh model have in common. In addition, the mesh model parameter estimation artificial neural network for animals and the artificial neural network for human mesh model parameter estimation may have learned network parameters in a direction to reduce a loss function calculated from the same frame of an observation image, respectively. In addition, when training an artificial neural network for estimating mesh model parameters for animals, when a three-dimensional linear mesh model of a quadruped animal using a label included in the training data set is projected as a two-dimensional image coordinate system, a silhouette and an animal mesh model When the 3D linear mesh model using the 3D linear mesh model parameters of the quadrupedal animal output by the artificial neural network is projected into the 2D image coordinate system, the network parameters may be learned in the direction where the silhouettes overlap as much as possible. . In addition, during training of the artificial neural network for estimating mesh model parameters for animals, it is regulated (regularize) that the distribution characteristics of the three-dimensional linear mesh model parameters of quadruped animals output by the artificial neural network for estimating the mesh model parameters for animals vary by animal species. Network parameters can be learned in the direction of reducing the loss function. For example, the mesh model parameter estimation artificial neural network for animals and the mesh model parameter estimation artificial neural network for humans in the artificial neural network operation unit 120 network parameters are learned in the direction of reducing the overall loss function in which one or more of the loss functions listed above are reflected. can

The data processing unit 130 selects any one of a three-dimensional linear mesh model parameter of a quadrupedal animal and a three-dimensional linear mesh model parameter of a human according to the result of classifying the species for the object, and selects the selected three-dimensional linear mesh model Create a 3D mesh model using parameters.

3 is a flowchart illustrating a mesh model generation method performed by the apparatus 100 for generating a mesh model for a human or a quadruped animal according to an embodiment of the present invention, and FIG. 4 is a mesh model and a quadrupedal animal mesh model. It is a diagram showing an example of matching sub-joints common to human mesh models.

Hereinafter, a mesh model generating method performed by the mesh model generating apparatus 100 for a human or quadrupedal animal according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4 .

First, the image acquisition unit 110 of the mesh model generation apparatus 100 obtains an observation image and provides it to the artificial neural network operation unit 120 of the mesh model generation apparatus 100 .

Then, the artificial neural network calculation unit 120 provides the species classification result for the object present in the observed image, the 3D linear mesh model parameter of the quadruped animal, and the 3D linear mesh model parameter of the human to the data processing unit 130 . .

를 추출하여 변형체 멀티태스크 브랜치에 제공할 수 있고(S310), 변형체 멀티태스크 브랜치는 인공신경망을 이용해 특징 벡터

에 대한 종 분류 결과와 4족 보행 동물의 3차원 선형 메쉬 모델 파라미터 및 사람의 3차원 선형 메쉬 모델 파라미터를 생성하여 데이터 처리부(130)에 제공할 수 있다. 예컨대, 변형체 멀티태스크 브랜치는 종 분류를 위한 인공신경망으로서 분류 알고리즘(classifier)을 포함할 수 있고, 메쉬 모델 파라미터의 추정을 위한 인공신경망으로서 동물용 회귀 알고리즘(skinned multi animal linear mesh model regressor) 및 사람용 회귀 알고리즘(skinned multi person linear mesh model regressor)을 포함할 수 있다.For example, the artificial neural network calculator 120 may include a deformable body feature extractor and a deformable body multi-task branch as illustrated in FIG. 2 . In this case, the deformable feature extractor uses an artificial neural network to obtain feature vectors of objects present in the observed image.

can be extracted and provided to the variant multi-task branch (S310), and the variant multi-task branch is a feature vector using an artificial neural network.

A species classification result for , a three-dimensional linear mesh model parameter of a quadrupedal animal, and a three-dimensional linear mesh model parameter of a human may be generated and provided to the data processing unit 130 . For example, the variant multitask branch may include a classifier as an artificial neural network for classifying a species, and a skinned multi animal linear mesh model regressor and a human as an artificial neural network for estimating mesh model parameters. It may include a skinned multi person linear mesh model regressor.

에 대한 종 분류 결과를 출력한다. 예를 들어, 사람, 개, 말, 소 등을 분류할 수 있다(S320).Here, the classification algorithm (classifier) included in the deformable multitask branch of the artificial neural network operation unit 120 is a feature vector

Output the species classification results for For example, people, dogs, horses, cattle, etc. may be classified (S320).

에 대하여 4족 보행 동물의 3차원 선형 메쉬 모델 파라미터로서 자세 파라미터

및 형상 파라미터

를 출력할 수 있다(S330).In addition, the regression algorithm for animals included in the deformable multitask branch of the artificial neural network operation unit 120 is a feature vector.

Postural parameters as three-dimensional linear mesh model parameters for quadrupedal animals

and shape parameters

can be output (S330).

에 대하여 사람의 3차원 선형 메쉬 모델 파라미터로서 자세 파라미터

및 형상 파라미터,

를 출력할 수 있다(S340).In addition, the human regression algorithm included in the deformable multitask branch of the artificial neural network operation unit 120 is a feature vector

With respect to the human three-dimensional linear mesh model parameters, posture parameters

and shape parameters;

can be output (S340).

Here, steps S320, S330 and S340 may be processed in parallel as shown in FIG. 3 . Alternatively, two of steps S320, S330, and S340 may be serially processed and the other step may be processed in parallel with the serially processed steps. Alternatively, steps S320, S330, and S340 may all be serially processed.

로 표현할 수 있고, 사람의 3차원 메쉬 모델은

로 표현할 수 있다(S360).Then, the data processing unit 130 of the mesh model generating apparatus 100 selects any one of the three-dimensional linear mesh model parameter of a quadrupedal animal and the three-dimensional linear mesh model parameter of a human according to the result of classifying the species for the object. Select (S350), and generate a 3D mesh model using the selected 3D linear mesh model parameter. A three-dimensional mesh model of a quadrupedal animal is

It can be expressed as , and the human 3D mesh model is

It can be expressed as (S360).

In addition, the data processing unit 130 may project the generated 3D mesh model onto a 2D image, and as an incidental result, 3D joint coordinates and 2 Dimensional joint coordinates can also be obtained.

를 줄이는 방향으로 네트워크 파라미터가 학습될 수 있다.Meanwhile, the artificial neural network operation unit 120 may be pre-trained for species classification and estimation of the 3D linear mesh model parameter of a quadrupedal animal and the 3D linear mesh model parameter of a human. In the learning process of the artificial neural network operation unit 120, the total loss function exemplified in Equation 1

Network parameters can be learned in a direction to reduce .

[Equation 1]

는 사람, 4족 보행 동물에 대한 전체 관절의 정확성을 평가하는 손실 함수이다. 사람의 3차원 메쉬 모델

, 4족 보행 동물의 3차원 메쉬 모델

의 정점(vertex)들의 선형 결합으로 3차원 관절을 구할 수 있고, 이를 영상으로부터 예측된 카메라 파라미터를 통해 사영하면 2차원 관절 좌표를 얻을 수 있다. 데이터셋으로부터 주어진 정답(ground-truth) 2차원 관절 좌표를 사람, 동물에 대하여 각각

,

라고 하고, 3차원 메쉬 모델로부터 예측된 2차원 관절 좌표를 사람, 동물 각각에 대하여

,

라고 할 경우, 정답 관절 좌표와 예측한 관절 좌표의 유클리드 거리(euclidean distance)로 계산된다.

is a loss function that evaluates the overall joint accuracy for humans and quadrupeds. 3D mesh model of the human

, a three-dimensional mesh model of a quadrupedal animal

A 3D joint can be obtained by linear combination of the vertices of The ground-truth two-dimensional joint coordinates given from the dataset are calculated for humans and animals, respectively.

,

, and the two-dimensional joint coordinates predicted from the three-dimensional mesh model are calculated for each person and animal.

,

, it is calculated as the Euclidean distance between the correct joint coordinates and the predicted joint coordinates.

,

는 사람, 동물의 전체 관절 중 보이는 관절의 개수

,

는 입력 영상에서

번째 관절이 보이는지 여부를 나타내는 플래그 이다.

,

is the number of visible joints among all joints in humans and animals.

,

is from the input image.

A flag indicating whether the second joint is visible.

[Equation 2]

[Equation 3]

은 위와 같이 정답과 예측한 관절 좌표 간의 거리를 나타내지만, 전체 관절이 아니라 도 4에 예시한 바와 같이 정의한 부-관절 사이의 거리를 측정한다. 사람 이미지의 경우, 동물의 관절에 대한 정답이 존재하지 않고, 반대로 동물 이미지의 경우 사람 관절에 대한 정답이 존재하지 않는다. 실시예에 따르면, 사람과 4족 보행 동물의 형태학적 유사성을 학습하기 위해 한 영상 프레임으로부터 사람, 동물의 3차원 메쉬 모델을 모두 구성하게 되는데, 도 2의 예시에서는 말의 관측 프레임이 들어왔으므로, 사람의 3차원 메쉬 모델에 대한 구성을 잘 하였는지 판단하기 힘들다. 따라서 적어도 사람의 3차원 메쉬 모델이 동물과 공통적으로 가지고 있는 머리, 손 끝, 발 끝 등의 부-관절을 맞추도록 네트워크가 학습되면, 하나의 영상 프레임으로부터 모든 멀티태스크 브랜치에서 손실 함수를 계산할 수 있고, 따라서 모든 멀티태스크 브랜치로의 경사(gradient) 흐름이 생기게 되므로, 버리는 학습 데이터 없이 데이터 효율적인 지도학습이 가능하다.

represents the distance between the correct answer and the predicted joint coordinates as above, but measures the distance between the sub-joints defined as illustrated in FIG. 4 rather than the entire joint. In the case of a human image, there is no correct answer for the joint of an animal. According to the embodiment, in order to learn the morphological similarity between humans and quadrupeds, all three-dimensional mesh models of humans and animals are constructed from one image frame. , it is difficult to judge whether the human 3D mesh model is well constructed. Therefore, if the network is trained to fit the sub-joints such as the head, fingertips, and toes that at least the human 3D mesh model has in common with animals, it is possible to calculate the loss function in all multitask branches from one image frame. As a result, there is a gradient flow to all multi-task branches, so data-efficient supervised learning is possible without discarding training data.

은 수학식 4와 같이 나타낼 수 있고, 동물 이미지에 대하여, 예측한 3차원 메쉬 모델을 구성한 후 이를 2차원 영상 좌표계로 사영했을 때 3차원 메쉬 모델이 차지하는 실루엣이 실제 영상의 실루엣과 같아야 함을 지도해 주는 손실 함수이다. 동물 데이터셋에 대하여 주어진 정답 실루엣

과 3차원 메쉬 모델을 통해서 예측된 실루엣

의 겹치는 정도가 클수록 값이 작아지도록 설계되었다.

can be expressed as Equation 4, and when a predicted 3D mesh model is constructed for an animal image and then projected as a 2D image coordinate system, the silhouette occupied by the 3D mesh model should be the same as the silhouette of the actual image. It is a loss function that gives Correct answer silhouette given for animal dataset

and the silhouette predicted through the 3D mesh model

It was designed so that the larger the degree of overlap, the smaller the value.

[Equation 4]

가 다른 분포를 가지게 설계 되었는데, 입력 영상에 들어온 동물의 종에 따라 네트워크로부터 예측된 형상 파라미터를 규제하는(regularize) 종-선택적 형상 파라미터 손실 함수를 디자인 하였다. 동물 종을

, 그리고 각

종에 대한 형상 파라미터 분포의 평균과 공분산 행렬을

,

라고 했을 때, 종-선택적 형상 파라미터 손실 함수는 수학식 5와 같은 마할라노비스 거리(mahalanobis distance) 개념으로 계산된다. 이때

는 영상에 대한 정답 종, 그리고

는 참이면 1, 거짓이면 0을 반환하는 함수이다.In addition, in the case of a three-dimensional mesh model of a quadrupedal animal, the shape parameters for the animal species

was designed to have a different distribution, and a species-selective shape parameter loss function was designed to regularize the shape parameters predicted from the network according to the species of animals that entered the input image. animal species

, and each

Calculate the mean and covariance matrices of the distribution of shape parameters for the species.

,

, the species-selective shape parameter loss function is calculated with the concept of mahalanobis distance as in Equation 5. At this time

is the correct species for the image, and

is a function that returns 1 if true and 0 if false.

[Equation 5]

The artificial neural network operation unit 120 may learn the network parameters in a direction to reduce the overall loss function in which one or more or all of the loss functions listed above are reflected.

Meanwhile, according to the above-described embodiment, each step included in the method for generating a mesh model may be implemented in a computer-readable recording medium for recording a computer program including instructions for performing such a step.

As described so far, according to an embodiment of the present invention, after obtaining both the 3D linear mesh model parameters of a quadruped animal and the 3D linear mesh model parameters of a human with respect to the feature vector of an object present in an observation image, By generating a three-dimensional mesh model using alternatively, the three-dimensional posture and shape of as many species as possible can be estimated.

Combinations of each step in each flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be embodied in a processor of a general purpose computer, special purpose computer, or other programmable data processing equipment, such that the instructions performed by the processor of the computer or other programmable data processing equipment provide the functions described in each step of the flowchart. It creates a means to do these things. These computer program instructions may also be stored in a computer-usable or computer-readable medium that may direct a computer or other programmable data processing equipment to implement a function in a particular manner, and thus the computer-usable or computer-readable medium. The instructions stored in the recording medium are also capable of producing an article of manufacture including instruction means for performing the functions described in each step of the flowchart. The computer program instructions may also be mounted on a computer or other programmable data processing equipment, such that a series of operational steps are performed on the computer or other programmable data processing equipment to create a computer-executed process to create a computer or other programmable data processing equipment. It is also possible that instructions for performing the processing equipment provide steps for performing the functions described in each step of the flowchart.

Further, each step may represent a module, segment, or portion of code comprising one or more executable instructions for executing the specified logical function(s). It should also be noted that in some alternative embodiments it is also possible for the functions recited in the steps to occur out of order. For example, it is possible that two steps shown one after another may in fact be performed substantially simultaneously, or that the steps may sometimes be performed in the reverse order depending on the function in question.

The above description is merely illustrative of the technical idea of the present invention, and various modifications and variations will be possible without departing from the essential quality of the present invention by those skilled in the art to which the present invention pertains. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical spirit of the present invention, but to explain, and the scope of the technical spirit of the present invention is not limited by these embodiments. The protection scope of the present invention should be construed by the following claims, and all technical ideas within the scope equivalent thereto should be construed as being included in the scope of the present invention.

100: mesh model generator
110: image acquisition unit
120: artificial neural network computation unit
130: data processing unit

Claims (14)

A mesh model generating method performed by a mesh model generating apparatus for a human or a quadrupedal animal, comprising:
extracting a feature vector of an object existing in an observation image;
classifying a class for the object based on the feature vector;
inputting the feature vector into a pre-trained first mesh model parameter estimation artificial neural network to obtain a three-dimensional linear mesh model parameter of a quadrupedal animal as an output of the first mesh model parameter estimation artificial neural network;
obtaining a human 3D linear mesh model parameter as an output of the second mesh model parameter estimation artificial neural network by inputting the feature vector into a pre-trained second mesh model parameter estimation artificial neural network;
selecting one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human according to the result of classifying the species for the object;
generating a three-dimensional mesh model using the selected three-dimensional linear mesh model parameter;
How to create a mesh model. The method of claim 1,
When learning the first mesh model parameter estimation artificial neural network and the second mesh model parameter estimation artificial neural network,
The network parameters are learned in the direction of reducing the loss function that evaluates the accuracy of the entire joint for the quadruped animal and the person.
How to create a mesh model. The method of claim 1,
When learning the first mesh model parameter estimation artificial neural network and the second mesh model parameter estimation artificial neural network,
The network parameters are learned in the direction of learning morphological similarity by matching the sub-keypoints that the quadruped animal mesh model and the human mesh model have in common.
How to create a mesh model. 4. The method of claim 3,
The first mesh model parameter estimating artificial neural network and the second mesh model parameter estimating artificial neural network learn network parameters in the direction of reducing the loss function calculated from the same frame of the observed image, respectively.
How to create a mesh model. The method of claim 1,
When learning the first mesh model parameter estimation artificial neural network,
When a three-dimensional linear mesh model of a quadruped animal using a label included in the training data set is projected as a two-dimensional image coordinate system, the silhouette and the first mesh model parameter estimation 3D output by the artificial neural network When the 3D linear mesh model using the linear mesh model parameters is projected in the 2D image coordinate system, the network parameters are learned in the direction where the silhouettes overlap as much as possible.
How to create a mesh model. The method of claim 1,
When learning the first mesh model parameter estimation artificial neural network,
3D linear mesh model parameters of a quadrupedal animal output from the first mesh model parameter estimation artificial neural network are learned in a way that reduces a loss function that regularizes distribution characteristics for each animal species.
How to create a mesh model. An image acquisition unit for acquiring an observation image;
An artificial neural network calculation unit for estimating a three-dimensional linear mesh model parameter of an object existing in the observed image through an operation using an artificial neural network;
and a data processing unit for processing data including the estimated three-dimensional linear mesh model parameters,
The artificial neural network calculation unit,
Extracting a feature vector of an object existing in the observed image acquired by the image acquisition unit, classifying a class for the object based on the feature vector, and applying the feature vector to a pre-trained first mesh Input to the model parameter estimation artificial neural network to obtain the 3D linear mesh model parameters of the quadruped animal as the output of the first mesh model parameter estimation artificial neural network, and use the feature vector to estimate the second mesh model parameter pre-trained artificial neural network to obtain a human three-dimensional linear mesh model parameter as an output of the second mesh model parameter estimation artificial neural network,
The data processing unit,
According to the result of classifying the species for the object, one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human is selected, and the selected three-dimensional linear mesh model parameter is to create a 3D mesh model using
Mesh model generator. 8. The method of claim 7,
When learning the first mesh model parameter estimation artificial neural network and the second mesh model parameter estimation artificial neural network,
The network parameters are learned in the direction of reducing the loss function that evaluates the accuracy of the entire joint for the quadruped animal and the person.
Mesh model generator. 8. The method of claim 7,
When learning the first mesh model parameter estimation artificial neural network and the second mesh model parameter estimation artificial neural network,
The network parameters are learned in the direction of learning morphological similarity by matching the sub-keypoints that the quadruped animal mesh model and the human mesh model have in common.
Mesh model generator. 10. The method of claim 9,
The first mesh model parameter estimating artificial neural network and the second mesh model parameter estimating artificial neural network learn network parameters in the direction of reducing the loss function calculated from the same frame of the observed image, respectively.
Mesh model generator. 8. The method of claim 7,
When learning the first mesh model parameter estimation artificial neural network,
When a three-dimensional linear mesh model of a quadruped animal using a label included in the training data set is projected as a two-dimensional image coordinate system, the silhouette and the first mesh model parameter estimation 3D output by the artificial neural network When the 3D linear mesh model using the linear mesh model parameters is projected in the 2D image coordinate system, the network parameters are learned in the direction where the silhouettes overlap as much as possible.
Mesh model generator. 8. The method of claim 7,
When learning the first mesh model parameter estimation artificial neural network,
3D linear mesh model parameters of a quadrupedal animal output from the first mesh model parameter estimation artificial neural network are learned in a way that reduces a loss function that regularizes distribution characteristics for each animal species.
Mesh model generator. As a computer-readable recording medium storing a computer program,
The computer program, when executed by a processor,
extracting a feature vector of an object existing in an observation image;
classifying a class for the object based on the feature vector;
inputting the feature vector into a pre-trained first mesh model parameter estimation artificial neural network to obtain a three-dimensional linear mesh model parameter of a quadrupedal animal as an output of the first mesh model parameter estimation artificial neural network;
obtaining a human 3D linear mesh model parameter as an output of the second mesh model parameter estimation artificial neural network by inputting the feature vector into a pre-trained second mesh model parameter estimation artificial neural network;
selecting one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human according to the result of classifying the species for the object;
Generating a three-dimensional mesh model using the selected three-dimensional linear mesh model parameter,
A computer-readable recording medium comprising instructions for causing the processor to perform a mesh model generation method. As a computer program stored in a computer-readable recording medium,
The computer program, when executed by a processor,
extracting a feature vector of an object existing in an observation image;
classifying a class for the object based on the feature vector;
inputting the feature vector into a pre-trained first mesh model parameter estimation artificial neural network to obtain a three-dimensional linear mesh model parameter of a quadrupedal animal as an output of the first mesh model parameter estimation artificial neural network;
obtaining a human 3D linear mesh model parameter as an output of the second mesh model parameter estimation artificial neural network by inputting the feature vector into a pre-trained second mesh model parameter estimation artificial neural network;
selecting one of the three-dimensional linear mesh model parameter of the quadruped animal and the three-dimensional linear mesh model parameter of the human according to the result of classifying the species for the object;
Generating a three-dimensional mesh model using the selected three-dimensional linear mesh model parameter,
A computer program comprising instructions for causing the processor to perform a method for generating a mesh model.

Images