JP7432330B2 - Posture correction network learning device and its program, and posture estimation device and its program - Google Patents

Description

The present invention relates to a posture correction network learning device and program for learning a posture correction network of a neural network that corrects the posture of a CG model, and a posture estimation device and program for estimating the posture of a CG model using the posture correction network. Regarding.

In recent years, devices have been put into practical use that reflect the joints and bones of the human body estimated from video (hereinafter referred to as human body bones) in a CG model to make CG characters move in real time or to operate virtual avatars in virtual reality. I'm doing it.
Conventionally, there are various methods for recognizing human body bones from images.
For example, there is a method of estimating the positions and connection states of joints of human body bones on a two-dimensional image using a neural network (see Non-Patent Document 1). In addition to this method, there is a method that uses a stereo camera to estimate the three-dimensional position of a joint based on the principle of triangulation (see Non-Patent Document 2). In this method, for example, as shown in FIG. 8, the three-dimensional coordinates of predetermined joints p of the human body and the connection state of each joint p are estimated.
In addition, as a method for estimating the posture of a joint of a human body bone on a two-dimensional image, there is a method that uses a latent variable model to learn the posture of a joint that cannot be directly observed from the joint coordinates on a two-dimensional image (Patent Document 1 reference).

Japanese Patent Application Publication No. 2010-229783

Zhe Cao, et.al, “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, CVPR2017 “AI (Artificial Intelligence) Skeleton Detection System”, [online], [Retrieved September 2, 2021], Internet <URL: https://www.next-system.com/visionpose>

According to conventional methods, it is possible to estimate the positions (three-dimensional positions) of joints of human body bones and the postures (three-dimensional rotation angles) of each joint from images.
However, the conventional method estimates the three-dimensional rotation angle of each joint only from the three-dimensional position of the joint, and does not take into account the rotation (roll rotation angle) about the bones connecting each joint. Note that the roll rotation angle cannot be uniquely estimated only from the three-dimensional position. For example, as shown in FIG. 9, when joints p1 and p2 are connected, the three-dimensional positions of joints p1 and p2 can be estimated using conventional methods. The roll rotation angle is indeterminate.

Therefore, when the joint positions and postures estimated in this way are applied to a three-dimensional CG model, the rotation angles of the model's joints may be expressed unnaturally, or the connection relationships may be broken. This may occur. For example, as shown in Fig. 10(a), when a CG model is generated with an arm connected to shoulder joint p, the roll rotation angle for the axis indicated by the dotted line in the posture of joint p is unstable. Therefore, as shown in FIG. 10(b), twisting may occur in the arm and torso at the joint p.

The present invention has been made in view of these problems, and it learns a posture correction network composed of a neural network that corrects the posture estimated from the three-dimensional position information of the joints of a CG model to a more correct posture. An object of the present invention is to provide a posture correction network learning device and its program, and a posture estimation device and its program that estimate the posture of a CG model using the posture correction network.

In order to solve the above problems, a posture correction network learning device according to the present invention provides a posture correction network that learns a posture correction network constituted by a neural network that corrects postures corresponding to positions of joints of a CG model in a three-dimensional space. The network learning device is configured to include posture correction means, rendering means, identification means, and parameter updating means.

In this configuration, the posture correction network learning device uses the posture correction network by the posture correction means to calculate corrected learning data from the posture indefinite learning data, which is learning data with an indefinite posture. Note that although the initial values of the parameters of the attitude correction network are initially undefined, they are sequentially updated by the parameter updating means.

Then, the posture correction network learning device uses a rendering means to create a three-dimensional dummy model in which a predetermined pattern texture is set in each part of the CG model for the correct learning data and the corrected learning data, which are learning data with correct postures. Rendering is performed to generate a two-dimensional rendered image. As a result, if the posture is incorrect due to twisting or the like, this will be expressed as a difference in the arrangement of pattern textures on the rendered image.

The posture correction network learning device also includes an identification network configured by a neural network that uses an identification means to identify a rendered image generated from correct learning data as a true value, and identifies a rendered image generated from corrected learning data as a false value. The identification result is calculated from the rendered image using . Note that although the initial values of the parameters of the identification network are initially undefined, they are sequentially updated by the parameter updating means.

Then, the posture correction network learning device uses the parameter updating means to reduce the cross entropy between the identification result by the identification means of the rendered image generated from the correct learning data and the true value, and identifies the rendered image generated from the corrected learning data. The parameters of the identification network are updated so as to reduce the cross entropy between the identification result by the means and the false value.
Further, the posture correction network learning device uses the parameter updating means to update the parameters of the posture correction network so as to increase the cross entropy between the identification result by the identification means of the rendered image generated from the correction learning data and the false value.

In this way, the identification network is trained to correctly identify the truth or falsehood of a rendered image, and the posture correction network is trained to make the identification network incorrectly identify a rendered image using corrected learning data as a true value. It will be learned.
Thereby, the posture correction network learning device learns the posture correction network as a neural network that correctly corrects the posture of the CG model.

In addition, the posture-indeterminate learning data, which is data with an indeterminate posture, uses inverse kinematics calculation, which is a method of controlling joints, to calculate a posture in which the rotation angle of the joint is indeterminate from the joint positions of the CG model in three-dimensional space. It may be generated by an attitude calculation means.
Further, the present invention can also be realized by a posture correction network learning program for causing a computer to function as the posture correction network learning device.

Moreover, in order to solve the above-mentioned problem, a posture estimation device according to the present invention is a posture estimation device that estimates the posture of a CG model, and is configured to include posture correction means.

In such a configuration, the posture estimation device uses a posture correction network constituted by a neural network that corrects the posture corresponding to the joint position of the CG model in the three-dimensional space, using the posture correction means to correspond to the joint position. Correct your posture.
Note that the posture to be corrected is generated from the position of the joint of the CG model in a three-dimensional space by a posture calculation means that calculates a posture in which the rotation angle of the joint is uncertain by inverse kinematics calculation, which is a method of controlling the joint. You can.
Moreover, the present invention can also be realized by a posture estimation program for causing a computer to function as the posture estimation device.

The present invention has the following excellent effects.
According to the present invention, it is possible to learn a posture correction network constituted by a neural network that accurately corrects the postures of joints of a CG model in a three-dimensional space.
Further, according to the present invention, by using a learned posture correction network, it is possible to correct the postures of joints that are unstable based only on the joint positions of the CG model, and to estimate the correct postures.

FIG. 2 is an explanatory diagram for explaining the processing content of the posture correction network learning device according to the embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of a posture correction network learning device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram for explaining the structure of a dummy model, in which (a) is an example of pattern texture installed in each part, (b) is an example of an image rendered with the dummy model in the correct pattern arrangement, and (c) is an example of an image rendered with the correct pattern arrangement. An example image is shown in which a dummy model is rendered with an incorrect pattern arrangement. FIG. 6 is an explanatory diagram for explaining the processing content of the posture calculation means. 3 is a flowchart showing the operation of the posture correction network learning device according to the embodiment of the present invention. FIG. 3 is an explanatory diagram for explaining the processing content of the posture estimation device according to the embodiment of the present invention. FIG. 1 is a block diagram showing the configuration of a posture estimation device according to an embodiment of the present invention. FIG. 3 is a diagram showing the positions of joints of a CG model in a three-dimensional space estimated from an image. FIG. 2 is an explanatory diagram for explaining the indeterminacy of the posture of a joint obtained by a conventional method. Examples of CG generated with joint postures determined by conventional methods, (a) shows the state when the joint postures are correctly connected, and (b) shows the state when twisting occurs in the joint. be.

Embodiments of the present invention will be described below with reference to the drawings.
<Processing of posture correction network learning device>
With reference to FIG. 1, processing of the posture correction network learning device 1 (FIG. 2) according to the embodiment of the present invention will be described.

The posture correction network learning device 1 is configured to learn a posture correction network Ng formed by a neural network that corrects postures associated with joint positions of a CG model in a three-dimensional space.
The posture correction network learning device 1 uses a plurality of pieces of joint position information P and a plurality of pieces of joint position and posture information Q as learning data.

The joint position information P is a set {p} of three-dimensional position coordinates (hereinafter simply referred to as positions) of joints p for the number of joints. This joint position information P may be, for example, a joint position estimated from an image using a conventional general human body bone recognition method, or may be a joint position estimated from a video using a conventional general human body bone recognition method, or a marker is attached to the joint position of the human body, and the marker is It may be obtained by motion capture in which the position is measured as the position of a joint.

Joint position/posture information Q is a set of three-dimensional position coordinates (positions) q of joints and rotation angles (hereinafter simply referred to as postures) φ of each joint based on each three-dimensional axis for the number of joints {q ,φ}. This joint position/posture information Q is information on the known (correct) posture of the joint. This joint position and posture information Q includes human body bone information from CG data in which joint positions and postures are correctly generated in advance, and information obtained by selecting only existing CG data that accurately represents joint postures. can be used.
Note that the number of joints in the joint position/posture information Q is the same as the number of joints in the joint position information P.

The posture correction network learning device 1 sequentially learns two neural networks, a posture correction network Ng and a discrimination network Nd, from joint position information P and joint position and posture information Q, which are learning data, to create a highly accurate posture correction network. Generate Ng.
Note that the posture correction network Ng and the discrimination network Nd correspond to a generator and a discriminator in a generative adversarial network (GAN) of deep learning.

The posture correction network learning device 1 calculates a set of postures θ^ {θ^} by inverse kinematic calculation for each joint position set {p} that is joint position information P, and calculates a set {θ^} of posture parameters {p, Generate θ^}.
Then, the posture correction network learning device 1 uses the posture correction network Ng to correct the posture set {θ^} corresponding to the joint position set {p} to {θ ^~ }. Then, the posture correction network learning device 1 generates a two-dimensional image (rendered image) Ig by rendering the dummy model Dm, which is a 3D model, using the corrected posture parameter set {p, θ ^~ }.
Further, the posture correction network learning device 1 generates a two-dimensional image (rendered image) Ir by rendering the dummy model Dm with a set of posture parameters {q, φ} that is the joint position and posture information Q.

The posture correction network learning device 1 learns the identification network Nd to identify the image Ir as “true” as an image with a correct posture.
Additionally, the posture correction network learning device 1 learns the identification network Nd to identify the image Ig as a "false" image as an image with an incorrect posture. The posture correction network learning device 1 also learns the posture correction network Ng so that the identification network Nd identifies the image Ig as "true."
Thereby, the posture correction network learning device 1 can learn a highly accurate posture correction network Ng.

<Configuration of posture correction network learning device>
Next, the configuration of the posture correction network learning device 1 according to the embodiment of the present invention will be described with reference to FIG. 2 (see FIG. 1 as appropriate).
As shown in FIG. 2, the attitude correction network learning device 1 includes a storage means 10, an attitude calculation means 11, a learning data selection means 12, an attitude correction means 13, a rendering means 14, an identification means 15, and a parameter update means 16.

The storage means 10 stores various data used in the posture correction network learning device 1. The storage means 10 can be composed of a general storage medium such as a hard disk or a semiconductor memory.
Here, the storage means 10 is configured separately as a neural network storage means ₁₀₁ , a learning data storage means ₁₀₂ , and a dummy model storage means ₁₀₃ , but it may be configured as a single storage medium. .

The neural network storage means ₁₀₁ stores the posture correction network Ng and the identification network Nd.
The posture correction network Ng is a neural network that corrects the postures of predetermined joints of the CG model.
The posture correction network Ng takes as input data a predetermined number of joints (n sets) in which three-dimensional postures θ^ (=(θx^, θy^, θz^)) of the joints are one set. Further, the posture correction network Ng outputs n sets of three-dimensional postures θ ^~ (=(θx ^~ , θy ^~ , θz ^~ )) of the corrected joints, the same number as the input data. Note that the number of joints is the same as the number of joints in the joint position information P.

The model structure of this posture correction network Ng is not particularly limited, but may be a structure in which multiple affine layers that perform weighted summation operations are connected, which is the structure of a general neural network, and an activation function is inserted between the layers. can do.
Note that the inter-layer weighting coefficients, which are parameters to be learned by the posture correction network Ng, are set to initial values such as random numbers in advance, and are updated by the parameter updating means 16.

The identification network Nd is a neural network that identifies whether the postures of the joints of the CG model of the image generated by the rendering means 14 are correct or incorrect.
The identification network Nd takes a two-dimensional image as input data. Further, the identification network Nd outputs a probability value greater than 0 and less than 1 that identifies "true value: 1" and "false value: 0".

The model structure of this identification network Nd is not particularly limited, but it connects multiple convolution layers that perform convolution operations using kernels of a predetermined size, and outputs a value greater than 0 and less than 1 at the final stage. The structure can be provided with an activation function (sigmoid function) that is normalized to .
Note that the weighting coefficient of the kernel, which is a learning target parameter of the identification network Nd, is set to an initial value such as a random number in advance, and is updated by the parameter updating means 16.

The learning data storage means ₁₀₂ stores posture indefinite learning data Si and correct answer learning data Sc as learning data for learning the posture correction network Ng and the identification network Nd.

The posture indeterminate learning data Si includes joint positions (three-dimensional position coordinates) p and postures of each joint (rotation angles with respect to each three-dimensional axis) calculated from the joint position information P by the posture calculation means 11. ) θ^ is the set {p, θ^}. Note that, as explained in FIG. 9, the posture in the posture indefinite learning data Si has an indefinite roll rotation angle.

The correct learning data Sc consists of joint positions (three-dimensional position coordinates) q, which is joint position and posture information Q that correctly represents the joint positions and postures, and the postures of each joint (based on each three-dimensional axis). rotation angle) φ and the set {q, φ}.

The dummy model storage means ₁₀₃ stores in advance a dummy model Dm, which is a 3D model.
The dummy model Dm is data representing the shape of the CG model as a 3D model (polygon data, etc.). This dummy model Dm is used by the rendering means 14 to generate a two-dimensional image from the positions and postures of the joints.
In the dummy model Dm, it is assumed that each part of the CG model is set with a pattern texture that identifies each part. Note that the pattern texture is different depending on the rotational direction of the axis so that the rotational state of the axis (bone) between the joints can be identified at least.

For example, as shown in FIG. 3(a), when the wrist p ₁ , elbow p ₂ , and shoulder p ₃ of the dummy model Dm are the joints, the biceps brachii BC, triceps brachii TC, and forearm flexor FF of the arm , different pattern textures are pasted on each part corresponding to the forearm extensor muscles FE.
As a result, when the dummy model Dm is rendered into an image, if the postures of the joints are correct, the image will have a correct pattern arrangement as shown in FIG. 3(b). On the other hand, if the posture of the shoulder _p3 is not correct, each part of the biceps BC and triceps TC will be twisted, resulting in an incorrect image, as shown in FIG. 3(c).

The posture calculation means 11 calculates rotation angles with respect to three axes in a three-dimensional space as the posture of each joint from the joint positions (three-dimensional position coordinates) that are joint position information P. A general inverse kinematics calculation may be used to calculate the posture of the joint from the three-dimensional position coordinates.

For example, as shown in FIG. 4, when the wrist p ₁ , elbow p ₂ , and shoulder p ₃ are joints, the posture calculation means ₁₁ initially calculates the An arbitrary posture θ ₁ is set as the value. Then, the posture calculation means 11 calculates the posture θ ₂ at the elbow p ₂ by inverse kinematics calculation according to the amount of displacement of the three-dimensional position coordinates of the wrist p ₁ and the elbow p ₂ . Further, the posture calculation means 11 calculates the posture θ ₃ at the shoulder p ₃ by inverse kinematics calculation according to the amount of displacement of the three-dimensional position coordinates of the elbow p ₂ and the shoulder p ₃ . Note that in FIG. 4, the positions p ₁ , p ₂ , and p ₃ of the joints serve as constraint conditions, and the postures are determined sequentially, but the postures θ ₁ , θ ₂ , and θ ₃ are undefined on the dotted line axes. This means that they have a sexual nature. Further, in FIG. 4, the number of joints is reduced to simplify the explanation.
In this way, the posture calculation means 11 calculates the posture of each joint from the three-dimensional position coordinates of the joint, which is the joint position information P.
The posture calculation means 11 stores the position p and the posture θ^ in the learning data storage means ₁₀₂ as posture indefinite learning data Si for each predetermined number of joints of the CG model.

The learning data selection means 12 selects and reads out the learning data stored in the learning data storage means ₁₀₂ .
The learning data selection means 12 may alternately select the posture indefinite learning data Si or the correct answer learning data Sc, or may select the other learning data when learning using one of the learning data is completed. You can also use it as The learning data selection means 12 notifies the parameter updating means 16 which of the posture indefinite learning data Si and the correct learning data Sc has been selected. Further, the learning data selection means 12 selects the next learning data at a timing instructed by the parameter updating means 16.

When the learning data selection means 12 selects the posture indefinite learning data Si, the learning data selection means 12 outputs the learning data to the posture correction means 13.
Further, when the learning data selection means 12 selects the correct learning data Sc, the learning data selection means 12 outputs the learning data to the rendering means 14.

The learning data selection means 12 selects new learning data when instructed to input the next learning data from the parameter updating means 16.
Note that the learning data selection means 12 ends the selection of learning data when a predetermined learning end condition is reached. For example, when the learning data selection means 12 selects the entire learning data a predetermined number of times of learning, when the parameter update means 16 notifies that the amount of change in the parameter update has fallen below a predetermined threshold, etc. It is.

The posture correction means 13 corrects the postures of the joints of the posture indefinite learning data Si, which is the learning data calculated by the posture calculation means 11, using the posture correction network Ng.
The posture correction means 13 inputs the postures of the number of joints of the posture indefinite learning data Si into the posture correction network Ng stored in the neural network storage means ₁₀₁ , and calculates the weighting coefficient between layers, which is a parameter to be learned. This is used to perform neural network calculations.
The posture correction means 13 outputs the joint positions and corrected postures of the posture indefinite learning data Si to the rendering means ₁₄ as corrected learning data Si2.

The rendering means 14 generates a two-dimensional image (rendered image) by rendering the dummy model Dm stored in the dummy model storage means ₁₀₃ using the positions and postures of the joints of the CG model.
The rendering means 14 outputs joint positions and postures whose postures inputted from the posture correction means 13 are corrected (corrected learning data Si ₂ ), or joint positions and postures inputted from the learning data selection means 12 (correct learning data). A two-dimensional image is generated by rendering the dummy model Dm using the data Sc).
For example, the rendering means 14 maps pattern textures that are previously associated with each part in the dummy model Dm according to the positions and postures of the joints of the CG model, using a predetermined viewpoint position as a reference, and performs projection transformation. By doing so, a two-dimensional image is generated.
The rendering means 14 outputs the generated two-dimensional image to the identification means 15.

The identification means 15 uses the identification network Nd to identify whether the image generated by the rendering means 14 is an image with a correct posture.
The identification means 15 inputs the image generated by the rendering means 14 into the identification network Nd stored in the neural network storage means ₁₀₁ , and uses the weighting coefficient of the kernel, which is the parameter to be learned, to determine the neural network. Perform calculations.
The identification means 15 outputs the calculation result (identification result) of the identification network Nd, which is output in a range exceeding 0 and less than 1, to the parameter updating means 16.

The parameter updating means 16 reduces the cross entropy between the identification result of the image Ir generated from the correct learning data Sc and the true value, and reduces the cross entropy between the identification result of the image Ig generated from the corrected learning data Si ₂ and the false value. The parameters of the identification network Nd are updated to reduce the , and the parameters of the posture correction network Ng are updated to increase the cross entropy between the identification result of the image Ig generated from the corrected learning data _Si2 and the false value. be.

Here, the parameter updating means 16 updates the parameters of the attitude correction network Ng and the identification network Nd so as to maximize and minimize the loss values of the predetermined loss functions, respectively.
As the loss functions, a loss function for learning the discrimination network Nd (hereinafter referred to as a discrimination loss function) and a loss function for learning the posture correction network Ng (hereinafter referred to as a posture correction loss function) are used.

The discrimination loss function is the sum of the cross entropy between the discrimination result and the true value when the learning data is the correct learning data Sc, and the cross entropy between the discrimination result and the false value when the learning data is the corrected learning data _Si2 . (loss value).
The posture correction loss function is a function that calculates the cross entropy (loss value) between the identification result and the false value when the learning data is the corrected learning data _Si2 .

Here, D(Ig) is the value of the identification result for the image Ig input from the identification means 15 when the learning data is the corrected learning data _Si2 , and D(Ig) is the value of the identification result input from the identification means 15 when the learning data is the correct learning data Sc. Let D(Ir) be the value of the identification result for the image Ir. Also, let the true value be "Real (=1)" and the false value be "Fake (=0)". Then, let L _bce be a function that calculates binary cross entropy.
Note that the learning data selection means 12 notifies whether the learning data is the corrected learning data _Si2 obtained by correcting the attitude indefinite learning data Si or the correct learning data Sc.
In this case, the parameter updating means 16 updates the parameters of the identification network Nd so as to minimize the identification loss function L ^D shown in equation (1) below.

Note that a represents a real number [a∈(0,1)] in an open interval with “0” and “1” as both ends, and b represents “0” or “1” [b∈{0,1}].
Using the loss function of equation (1), the parameter updating means 16 trains the identification network Nd to identify the image Ir as "true" and the image Ig as "false."
Further, the parameter updating means 16 updates the parameters of the attitude correction network Ng so as to maximize the attitude correction loss function ^LG shown in the following equation (2).

Using the loss function of equation (2), the parameter updating means 16 causes the posture correction network Ng to learn so as not to cause the identification network Nd to identify the image Ig as "false".
The parameter updating means 16 updates the parameters of the identification network Nd and the attitude correction network Ng using the error backpropagation method so as to minimize and maximize the values of the loss functions of equations (1) and (2), respectively.
After updating the parameters, the parameter updating means 16 instructs the learning data selecting means 12 to select the next learning data. Furthermore, when the amount of change in parameter update is less than a predetermined threshold, the parameter update means 16 notifies the learning data selection means 12 of this fact.

As explained above, the posture correction network learning device 1 identifies the position and posture information of the joints of the CG model as correct information in the identification network Nd even if the posture is uncertain. An attitude correction network Ng that can correct the attitude can be learned as follows.
Note that the posture correction network learning device 1 can operate a computer with a posture correction network learning program for causing the computer to function as each of the above-mentioned means.

<Operation of posture correction network learning device>
Next, the operation of the posture correction network learning device 1 will be described with reference to FIG. 5 (see FIG. 2 as appropriate for the configuration).

In step S1, joint position and orientation information Q that correctly represents the positions and orientations of the joints of the CG model is stored in the learning data storage means ₁₀₂ as correct learning data Sc.
In step S2, the posture calculation means 11 calculates the posture of each joint from the joint position information P of the CG model by inverse kinematics calculation, which is a method of controlling the joints of a 3D CG model. Then, the posture calculation means 11 stores the calculated posture and joint positions in the learning data storage means ₁₀₂ as posture indefinite learning data Si.

In step S3, the learning data selection means 12 selects and reads learning data (posture indefinite learning data Si or correct answer learning data Sc) from the learning data storage _{means 102} .
In step S4, the learning data selection means 12 switches the subsequent processing depending on whether the posture indefinite learning data Si is selected as the learning data or the correct learning data Sc is selected.

If the learning data selected in this step S3 is the posture indefinite learning data Si (Yes in step S4), in step S5, the posture correction means 13 performs a neural network calculation using the posture correction network Ng. The joint postures of the posture-indeterminate learning data Si are corrected to generate corrected learning data _Si2 . The posture correction network learning device 1 then proceeds to step S6.
On the other hand, if the learning data selected in step S3 is the correct learning data Sc (No in step S4), the posture correction network learning device 1 directly proceeds to step S6.

In step S6, the rendering means 14 uses the correct learning data Sc selected in step S3 or the corrected learning data _Si2 obtained by correcting the posture indefinite learning data Si selected in step S3 and corrected in step S5. , a two-dimensional image is generated by rendering the dummy model Dm.

In step S7, the identification means 15 performs a neural network calculation using the identification network Nd to identify whether or not the image generated in step S6 is an image with a correct posture. Here, the identification means 15 calculates the identification result in a range greater than 0 and less than 1, with "0" representing "false" and "1" representing "true".

In step S8, the parameter updating means 16 updates the parameters of the identification network Nd and the attitude correction network Ng so that the predetermined loss function for identification becomes smaller and the predetermined loss function for attitude correction becomes larger.
Specifically, as shown in equation (1), if the image generated in step S6 is the image Ir generated from the correct learning data Sc, the parameter updating means 16 updates the identification result D(Ir) and the true The parameters of the identification network Nd are updated so as to reduce the cross entropy with the value "1" indicating the value. Further, when the image generated in step S6 is an image Ig generated from corrected learning data _Si2 obtained by correcting the posture indefinite learning data Si, the parameter updating means 16 indicates a false value as the identification result D(Ig). The parameters of the identification network Nd are updated so as to reduce the cross entropy with the value "0".

Further, as shown in equation (2), when the image generated in step S6 is an image Ig generated from corrected learning data _Si2 obtained by correcting the posture indefinite learning data Si, the parameter updating means 16 performs the identification result The parameters of the posture correction network Ng are updated so as to increase the cross entropy between D(Ig) and the value "0" indicating a false value.

In step S9, the learning data selection means 12 determines whether a predetermined end condition for completing learning is satisfied. For example, here, if the entire learning data has not been selected for a predetermined number of learning times (No in step S9), the learning data selection means 12 determines that learning is not completed, and the posture correction network learning device 1 returns to step S2 and continues learning.
On the other hand, if the learning data selection means 12 selects the entire learning data a predetermined number of times (Yes in step S9), the learning data selection means 12 determines that learning is completed, and the posture correction network learning device 1 ends the operation. .
Through the above-described operations, it is possible to learn a posture correction network Ng that can correct the posture even if the position and posture information of the joints of the CG model is uncertain.

<Processing of posture estimation device>
Next, with reference to FIG. 6, processing of the posture estimation device 2 (FIG. 7) according to the embodiment of the present invention will be described.
The posture estimation device 2 uses a posture correction network Ng, which is a neural network, to estimate the posture of a joint from the position of the joint in a three-dimensional space.
Here, the posture estimation device 2 inputs the joint position information P and corrects the posture.
The joint position information P is a set {p} of joint positions (three-dimensional position coordinates) p for the number of joints, and is the same as the joint position information P described in FIG. 1 .

The posture estimation device 2 calculates a set of postures θ^ {θ^} by inverse kinematic calculation for each joint position set {p} that is joint position information P, and calculates a set of posture parameters {p, θ^ } is generated.
Then, the posture estimation device 2 uses the posture correction network Ng to correct the posture set {θ^} corresponding to the joint position set {p} to {θ ^~ }.
Note that the posture correction network Ng is a neural network trained by the posture correction network learning device 1 (FIG. 2).
By rendering the 3D model C using the posture parameter set {p, θ ^~ } corrected in this way, it is possible to generate a two-dimensional image Ig in which the posture is naturally drawn.

<Configuration of posture estimation device>
Next, the configuration of the posture estimation device 2 according to the embodiment of the present invention will be described with reference to FIG. 7 (see FIG. 6 as appropriate).
As shown in FIG. 7, the posture estimation device 2 includes a storage means 20, a posture calculation means 21, and a posture correction means 22.

The storage means 20 stores the posture correction network Ng in advance. The storage means 20 can be composed of a general storage medium such as a hard disk or a semiconductor memory.
The posture correction network Ng is a neural network that corrects the postures of predetermined joints of the CG model, and is trained in advance by the posture correction network learning device 1 (FIG. 2).

The posture calculation means 21 calculates a rotation angle with respect to three axes in a three-dimensional space as a posture of each joint from the joint position (three-dimensional position coordinate) which is the joint position information P. A general inverse kinematics calculation may be used to calculate the posture of the joint from the three-dimensional position coordinates. Note that the attitude calculation means 21 can be the same as the attitude calculation means 11 of the attitude correction network learning device 1.
The posture calculation means 21 outputs the position p and the posture θ^ to the posture correction means 22 for each predetermined number of joints of the CG model.

The posture correction means 23 corrects the posture of the joint calculated by the posture calculation means 21 using the posture correction network Ng.
The posture correction means 23 inputs the postures of the joints calculated by the posture calculation means 21 into the posture correction network Ng stored in the storage means 20, and uses the learned weight coefficients between the layers of the posture correction network Ng. and perform neural network calculations.
Thereby, the posture correction means 23 can correct the posture that is unstable with respect to the axis (bone) between the joints calculated by the posture calculation means 21 to a correct posture.
The posture correction means 23 outputs joint position and posture information Q, which is a pair of joint positions and estimated joint postures, as an estimation result.
Note that the posture estimation device 2 can operate a computer with a posture estimation program for causing the computer to function as each of the above-mentioned means.

As described above, the posture estimation device 2 can estimate the correct posture of a joint from the position of the joint in the CG model.
Regarding the operation of the posture estimation device 2, since the posture calculation means 21 and the posture correction means 22 are simply operated continuously, a detailed explanation will be omitted.

Although the posture correction network learning device 1 and posture estimation device 2 according to the embodiments of the present invention have been described above, the present invention is not limited to these embodiments.
For example, the posture correction network learning device 1 inputs the joint position information P, and causes the posture calculation means 11 to calculate the posture corresponding to the joint by inverse kinematics calculation.
However, this attitude calculation may be performed externally in advance.
In that case, the joint position information P input to the posture correction network learning device 1 may be information in which a posture is added for each joint. The posture correction network learning device 1 may omit the posture calculation means 11 from the configuration.
This also applies to the posture estimation device 2.

Further, here, the positions and postures of the joints are explained using the positions and postures of the joints of a human body as an example, but the positions and postures of the joints of animals such as dogs and cats may be used as long as they have joints.

1 Attitude correction network learning device 10 Storage means 10 ₁ Neural network storage means 10 ₂ Learning data storage means 10 ₃ Dummy model storage means 11 Attitude calculation means 12 Learning data selection means 13 Attitude correction means 14 Rendering means 15 Identification means 16 Parameter updating means 2 Attitude estimation means 20 Storage means 21 Attitude calculation means 22 Attitude correction means Ng Attitude correction network Nd Identification network Si Attitude indefinite learning data Si ₂ correction learning data Sc Correct answer learning data Dm Dummy model

Claims (5)

A posture correction network learning device that learns a posture correction network configured by a neural network that corrects postures associated with joint positions of a CG model in a three-dimensional space,
Posture correction means that uses the posture correction network to calculate corrected learning data from posture indefinite learning data, which is learning data whose posture is indefinite;
A three-dimensional dummy model in which a predetermined pattern texture is set for each part of the CG model is rendered for each of the correct learning data and the corrected learning data in which the posture is correct, and a two-dimensional rendered image is generated. a rendering means;
A discrimination result is obtained from the rendered image by using an identification network constituted by a neural network that identifies the rendered image generated from the correct learning data as a true value and identifies the rendered image generated from the corrected learning data as a false value. Identification means for calculating;
reducing the cross entropy between the identification result by the identification means of the rendered image generated from the correct learning data and the true value, and the intersection between the identification result by the identification means of the rendered image generated from the corrected learning data and the false value; The parameters of the identification network are updated so as to reduce the entropy, and the parameters of the posture correction network are updated so as to increase the cross entropy between the identification result of the rendering image generated from the corrected learning data by the identification means and the false value. a means for updating parameters;
A posture correction network learning device comprising: The apparatus further comprises posture calculation means for calculating the posture of the joint from the position of the joint of the CG model in the three-dimensional space by inverse kinematics calculation, which is a method of controlling the joint, and generating the posture indefinite learning data. The posture correction network learning device according to claim 1, characterized in that: An attitude correction network learning program for causing a computer to function as the attitude correction network learning device according to claim 1 or 2. A posture estimation device that estimates the posture of a CG model,
The position of the joint is corrected using a posture correction network constituted by a neural network trained by the posture correction network learning device according to claim 1, which corrects the posture corresponding to the position of the joint of the CG model in a three-dimensional space. A posture estimation device comprising a posture correction means for correcting a posture associated with a posture. A posture estimation program for causing a computer to function as the posture estimation device according to claim 4 .

Images