JP2020107244A - Posture estimating apparatus, learning apparatus, and program - Google Patents

Abstract

To estimate a posture of a subject simply and with high accuracy.SOLUTION: A discriminating unit 10 of a posture estimating apparatus 1 inputs an image I and performs discrimination processing for discrete N angles θn using a preset parameter p in a convolution neutral network, for example, so as to calculate respective likelihood values w(θn). A weighted synthesis unit 20 performs weighted synthesis processing for the N angles θn with weighting responding to the respective likelihood values w(θn) for the N angles θn so as to generate continuous posture information θ.SELECTED DRAWING: Figure 1

Description

The present invention relates to a posture estimation device that estimates the posture of a subject from an input image, a learning device that learns the relationship between the input image and the posture of the subject, and a program.

Conventionally, there is known a posture estimation device that estimates a posture of a subject from an input image including the subject. For example, a technique of performing matching processing with an input image using a template in order to estimate the posture of a face is disclosed (for example, see Patent Document 1). Specifically, this technique uses a template of organs such as eyes that form a face to estimate the head posture from an input image, calculates the positions of organs such as eyes, and adapts the head model. The rotational displacement and translational displacement of the head are determined.

Further, for example, in order to estimate the posture of a face, a technique of integrating an identification result based on a color histogram of an input image and an identification result based on a feature amount (for example, a gradient histogram) other than the color histogram is disclosed. (For example, see Patent Document 2).

Japanese Patent No. 5016175 JP, 2008-22416, A

However, in the technique of template matching of Patent Document 1 described above, in order to estimate the posture of the face, a template for each organ such as an eye forming the face is required in advance. Therefore, there is a problem that it takes time to prepare a template for each organ constituting a face.

Further, the technique of the above-mentioned Patent Document 2 is for performing the estimation of the posture of the face with a low load based on the color histogram of the input image and the gradient histogram, for example.

However, the input image may include information useful for estimating the posture of the face, in addition to the color histogram and the gradient histogram. For example, phase information in the frequency domain, appearance of a characteristic pattern (tilt, position, size, aspect ratio, etc.) may be useful, but in the technique of Patent Document 2, these information are effective. Not used for. Therefore, there is a problem that the accuracy of estimating the posture of the face is insufficient in the processing limited to the information such as the color histogram.

Then, this invention is made in order to solve the said subject, The objective is to provide the posture estimation apparatus, the learning apparatus, and program which can estimate the posture of a to-be-photographed object easily and highly accurately.

In order to solve the problem, the posture estimation apparatus according to claim 1 is a posture estimation apparatus that estimates the posture of a subject included in an input image, identifies the angle of the subject based on the input image, and sets the preset angle. An identifying unit that obtains an accuracy value corresponding to each of the plurality of angles, by weighting according to the accuracy value corresponding to each of the plurality of angles obtained by the identifying unit, the plurality of angles are weighted and combined, And a weighted synthesis unit for obtaining posture information.

According to the first aspect of the present invention, since the identifying unit only needs to obtain the accuracy value for discrete angles, the circuit scale can be reduced as compared with the case where the accuracy values for continuous angles are obtained. In addition, the weighted composition unit obtains posture information that is continuous angle information using the accuracy values for discrete angles. The processing of the weighted synthesis unit is low in load because it is sufficient to perform the product-sum calculation. Therefore, continuous posture information can be obtained with a low-load and small-scale circuit.

A posture estimating apparatus according to a second aspect is the posture estimating apparatus according to the first aspect, characterized in that the identifying unit is configured by a neural network.

According to the second aspect of the present invention, a network for extracting a feature suitable for estimating the posture from the various features of the input image is constructed depending on the configuration and type of the neural network and the setting of the connection weighting coefficient which is a parameter. can do. As a result, the estimation accuracy can be improved as compared with the conventional method of estimating the posture by using the specific feature amount. Also, it is not necessary to explicitly give a template for each part of the subject (for example, a facial organ).

Further, the posture estimation apparatus according to claim 3 is the posture estimation apparatus according to claim 1 or 2, wherein the posture information is an angle of the subject, or the posture information is expressed as a vector value or a complex value. When the norm of is the reliability, the posture information is the angle of the subject and the reliability at the angle.

According to the third aspect of the present invention, by using the posture information as the angle and the reliability of the subject, the reliability of the angle of the subject can be quantified. Thereby, when other processing is performed using the posture information obtained by the posture estimation apparatus, the posture information having low reliability is not used for other processing. That is, reliability can be improved in the entire system including the posture estimation device and a device that uses posture information.

Furthermore, the learning device according to claim 4, wherein an image including a subject and posture information of the subject are input as learning data, a model is learned using the learning data, and parameters of the model are optimized. , An accuracy generating unit that obtains a learning accuracy value corresponding to each of a plurality of preset angles based on the posture information, the image, and the plurality of angles that are obtained by the accuracy generating unit. A learning identifying unit that learns the model for identifying the angle of the subject based on the learning accuracy value and updates the parameter used to estimate the posture of the subject. Is characterized by.

According to the invention of claim 4, the learning accuracy value corresponding to each of the plurality of angles can be obtained from the posture information, and the model can be learned using the image and the learning accuracy value. The obtained parameters can be obtained.

The learning device according to claim 5 is the learning device according to claim 4, wherein the accuracy generation unit calculates an angle formed between the vector of the posture information and each vector of the plurality of angles. A broadly monotonically decreasing and non-constant function is applied to the formed angle, and the learning accuracy value corresponding to each of the plurality of angles is obtained.

According to the invention of claim 5, the learning accuracy value increases as the angle becomes closer to the posture information. The learning identifying unit that uses such a learning accuracy value can update the parameters so that the model estimates the attitude at an angle close to the attitude information. By using this parameter in the posture estimation device, the posture of the subject can be properly estimated.

In the learning device according to claim 6, in the learning device according to claim 4, the accuracy generation unit calculates an angle formed between the vector of the posture information and each of the plurality of angles, Of the plurality of angles, a predetermined value is set to the learning accuracy value for the angle at which the formed angle is the smallest, and an angle at which the formed angle is not the smallest among the plurality of angles is smaller than the predetermined value. A value is set to the learning accuracy value.

According to the invention of claim 6, since the learning accuracy value is binary, it is sufficient to perform binary classification learning for each angle corresponding to the learning accuracy value. As a result, the circuit size of the learning identifying unit can be reduced, and the learning efficiency can be improved.

Further, the program according to claim 7 causes a computer to function as the posture estimation device according to any one of claims 1 to 3.

Further, a program according to claim 8 causes a computer to function as the learning device according to any one of claims 4 to 6.

As described above, according to the present invention, the posture of a subject can be estimated easily and with high accuracy.

It is a block diagram showing an example of composition of a posture estimating device by an embodiment of the present invention. It is a flow chart which shows an example of processing of a posture estimating device. It is a block diagram which shows an example of a structure of an identification part. It is a block diagram showing an example of composition of a learning device by an embodiment of the present invention. It is a flow chart which shows an example of processing of a learning device. It is a block diagram which shows an example of a structure of the identification part for learning.

Hereinafter, modes for carrying out the present invention will be described in detail with reference to the drawings. The posture estimation apparatus according to the embodiment of the present invention inputs an image I to be subjected to posture estimation, and uses a preset parameter p to calculate the posture information θ based on the image I and its reliability r as necessary. Estimate and output.

The image I is, for example, a human head image. The posture information θ is, for example, 0 when the person's head is directly facing the camera that captured the image I, and a positive rotation in a predetermined rotation direction (eg, counterclockwise when viewed from above the person). An angle value (angle value of 0 radian or more and less than 2π radian) is defined as an angle.

The size, shape and resolution of the image I are preferably fixed. For example, the image I is a rectangular image having horizontal W pixels and vertical H pixels. W and H are natural numbers. The image I may be, for example, a color image or a monochrome image.

The posture information θ may be an angle value, a vector value of a direction vector, or a complex value (phasor). When the posture information θ is represented by an angle value, the range of θ may be limited to a predetermined range (when θ is represented by the arc method, for example, 0≦θ<2π). When the posture information θ is represented by a vector value or a complex value, its norm value (for example, Euclidean norm value) is 1. The reliability r is, for example, a real value of 0 or more and 1 or less.

The posture information θ and the reliability r may be collectively expressed as the posture information θ as a vector value or a complex value. At this time, the deviation angle of the posture information θ is made to correspond to the posture angle, and the norm value of the posture information θ is made to correspond to the reliability r.

Further, the learning apparatus according to the embodiment of the present invention uses the image J _k and the posture information φ _k as learning data to learn the relationship between the image J _k and the posture information φ _k, and uses the optimum value in the posture estimation apparatus. A parameter p is calculated.

[Posture estimation device]
Next, the posture estimation apparatus according to the embodiment of the present invention will be described in detail. FIG. 1 is a block diagram showing an example of the configuration of a posture estimation apparatus according to an embodiment of the present invention, and FIG. 2 is a flowchart showing an example of processing of the posture estimation apparatus. The posture estimation device 1 is a device that estimates the posture of a subject included in an input image, and includes an identification unit 10 and a weighted synthesis unit 20.

(Identification unit 10)
The identification unit 10 inputs an image I that is an object of posture estimation, performs identification processing of N representative angles θ _n for a subject included in the image I using a preset parameter p, and The accuracy value w(θ _n ) of is calculated. N is a natural number of 2 or more, and n is an integer of 0 or more and N-1 or less. The N representative angles θ _n are set in advance. Then, the identification unit 10 _{weights the} accuracy value sequence (w(θ _n )) _{n ∈ {0, 1,..., N-1} including} the accuracy values w(θ _n ) for the N angles θ _n . Output to the synthesis unit 20.

That is, the discrimination unit 10 is a discriminator in which the input signal is the image I and the output signal is each accuracy value w(θ _n ) with respect to the N representative angles θ _n . In FIG. 3, which will be described later, the image I, which is an input signal, is a color image composed of three components of each pixel, and N=8.

The representative angle θ _n is preset so as to divide 2π radian into N equal parts, for example, as in the following formula.

The identification unit 10 is composed of, for example, a neural network. FIG. 3 is a block diagram showing an example of the configuration of the identifying unit 10, and shows a case where the identifying unit 10 is configured by a convolutional neural network. The identification unit 10 is configured by a convolutional neural network including convolutional layers (convolutional units) 11, 12, 13, and 14 and fully connected layers (fully connected units) 15 and 16.

The number of layers of neural network, number of elements, activation function, presence/absence of convolutional layer, size of convolution kernel, presence/absence of fully connected layer, presence/absence and step of stride (subsampling), presence/absence and type of pooling layer, drop The configuration such as presence or absence of out is arbitrary. The identification unit 10 may be a neural network other than the convolutional neural network.

In FIG. 3, for example, the image I input by the identification unit 10 is a 20×20×3 third-order tensor composed of horizontal W=20 pixels, vertical H=20 pixels, and three color components. Hereinafter, the horizontal pixel, the vertical pixel, and the component represented by the number of horizontal W pixels×the number of vertical H pixels×the number of components are referred to as “pixel components” for convenience of description.

With reference to FIG. 2 and FIG. 3, the convolutional layer 11 inputs the image I of the third-order tensor of 20×20×3 pixel components (step S201). Then, the convolutional layer 11 applies 12 types of convolution filters of 3×3×3 pixel components to the image I in a 2×2 stride, and performs convolution processing using a preset parameter p. The convolutional layer 11 generates an image T ₁ of a third-order tensor having 10×10×12 pixel components (step S202).

Since the convolution processing by the convolution layers 11, 12, 13, and 14 in the convolutional neural network is already known, detailed description thereof will be omitted here.

The convolutional layer 11 outputs to the convolutional layer 12 the image T ₁ of the third-order tensor of 10×10×12 pixel components. The image T ₁ is a third-order tensor image composed of horizontal W=10 pixels, vertical H=10 pixels, and 12 components.

The convolutional layer 12 inputs the image T ₁ of the third-order tensor of 10×10×12 pixel components from the convolutional layer 11. Then, the convolutional layer 12 applies 24 types of convolutional filters of 3×3×3 pixel components to the image T ₁ in a 2×2 stride, and performs convolutional processing using a preset parameter p. The convolutional layer 12 generates an image T ₂ of a third-order tensor of 5×5×24 pixel components (step S203).

The convolutional layer 12 outputs the image T ₂ of the third-order tensor of 5×5×24 pixel components to the convolutional layer 13. The image T ₂ is a third-order tensor image composed of horizontal W=5 pixels, vertical H=5 pixels, and 24 components.

The convolutional layer 13 inputs the image T ₂ of the third-order tensor of 5×5×24 pixel components from the convolutional layer 12. Then, the convolutional layer 13 applies 32 types of convolutional filters of 3×3×3 pixel components to the image T ₂ in a 1×1 stride, and performs convolutional processing using a preset parameter p. The convolutional layer 13 generates an image T ₃ of a third-order tensor of 3×3×32 pixel components (step S204).

The convolutional layer 13 outputs the image T ₃ of the third-order tensor of 3×3×32 pixel components to the convolutional layer 14. The image T ₃ is a third-order tensor image composed of horizontal W=3 pixels, vertical H=3 pixels, and 32 components.

The convolutional layer 14 inputs the image T ₃ of the third-order tensor of 3×3×32 pixel components from the convolutional layer 13. Then, the convolutional layer 14 applies 64 types of convolution filters of 3×3×3 pixel components to the image T ₃ in a 1×1 stride, and performs convolution processing using a preset parameter p. The convolutional layer 14 generates an image of a third-order tensor of 1×1×64 pixel components (vector V _{1 of} 64 components) (step S205). The convolutional layer 14 outputs the 64-component vector V ₁ to the fully connected layer 15.

The full connection layer 15 inputs the 64-component vector V ₁ from the convolutional layer 14, and uses a preset parameter p to connect all components forming the 64-component vector V ₁ with each other. Is performed to generate a 16-component vector V ₂ (step S206). Then, the fully connected layer 15 outputs the 16-component vector V ₂ to the fully connected layer 16. That is, the total connection layer 15 is a network that connects all the elements of the 64-component vector V ₁ that is the input signal and the respective elements of the 16-component vector V ₂ that is the output signal.

Since the total connection process by the total connection layers 15 and 16 in the convolutional neural network is known, detailed description thereof will be omitted here.

The fully-connected layer 16 inputs the 16-component vector V ₂ from the fully-connected layer 15 and uses a preset parameter p to combine all the components forming the 16-component vector V _2. Perform processing. The fully connected layer 16 generates an eight-component vector V ₃ (accuracy value w(θ _n ), n=0, 1,..., 7) (step S207).

The fully connected layer 16 has an accuracy value sequence (w(θ _n )) _{n ∈ {0, 1,} which is an accuracy vector w ₃ of 8 components and is composed of accuracy values w (θ _n ) for eight angles θ _{n . , 7}} is output to the weighted synthesis unit 20. In this case, θ ₀ =0/2π/8=0, θ ₁ =1/2π/8=π/4, θ ₂ =2/2π/8=π/2,..., θ ₇ =7.2π /8=7π/4. In other words, the total connection layer 16 is a network that connects all the elements of the 16-component vector V ₂ that is the input signal and the respective elements of the 8-component vector V ₃ that is the output signal.

As described above, since the identification unit 10 only needs to obtain the accuracy value w(θ _n ) for the discrete angle θ _n , it requires simpler processing than the case where the accuracy value for continuous angles is obtained, and the circuit The scale can be reduced.

A bias value may be set to the elements (neurons) that form the convolutional layers 11, 12, 13, and 14 and the total coupling layers 15 and 16. Further, the activation function applied to the elements forming the convolutional layers 11, 12, 13, 14 and the fully coupled layers 15, 16 is arbitrary, but for example, a half-wave rectifying function (ReLU: Rectified Linear Unit), a sigmoid (Sigmoid) ) Function, a hyperbolic tangent function, etc. can be used.

The parameter p used in the convolutional layers 11, 12, 13, 14 and the fully connected layers 15, 16 is a parameter for identifying the identification method of the identification unit 10 in the posture estimation apparatus 1 shown in FIG. When the identification unit 10 is a neural network, it is a weighting coefficient such as a weighting value, a bias value, or a filter coefficient. A value obtained in advance by the learning device 2 described later is used as the parameter p, and the parameter p may be stored in a ROM (Read Only Memory) or the like provided in the posture estimation device 1 or may be updated externally. , RAM (Random Access Memory) or flash ROM.

(Weighted combining unit 20)
Returning to FIG. 1 and FIG. 2, the weighted synthesis unit 20 detects that the identification unit 10 has accuracy values w (θ _n ) corresponding to _N ( _n =8 in the example of FIG. 3) angles θ _n . _{Input the} column (w(θ _n )) _{n ∈ {0, 1,..., N-1}} .

Weighting combining unit 20, the weighting according to the respective reliability values w (θ _n) for the N angle theta _n, performs weighting synthesis processing of the N angle theta _n, estimates the posture information theta (step S208 ). Then, the weighted composition unit 20 outputs the posture information θ (step S209).

ｊは虚数単位である。 For example, the weighted synthesis unit 20 weights the complex value of the absolute value 1 and the angle (declination) θ _{n with} the accuracy value w(θ _n ) as a weight, and gives the result of the weighted synthesis, as in the following formula. Calculate the complex number ζ.

j is an imaginary unit.

Instead of the complex number ζ, a two-dimensional vector value having a real part and an imaginary part of the complex number ζ as a component may be used to perform the calculation of the equation (2).

Further, when the posture information θ and the reliability r are collectively expressed as the vector information or the complex posture information θ, the weighted synthesis unit 20 calculates the complex number ζ according to the following equation in the equation (2). Is output as the posture information θ as it is.

前記式（４）において、ノルム

は、例えばユークリッドノルムとする。 In addition, the weighted synthesis unit 20 performs the calculation using the following equation when individually outputting the posture information θ and the reliability r.

In the formula (4), the norm

Is, for example, the Euclidean norm.

Thus, the weighted combining unit 20, reliability value w for discrete angles theta _n a (theta _n) by weighted synthesis of the angle theta _n as a weight, to estimate the posture information of the continuous angle information theta I chose As a result, the processing of weighted composition is performed by the sum of products calculation, so that the calculation load can be reduced and the posture information θ taking a continuous value can be estimated by a small-scale circuit.

As described above, according to the posture estimation apparatus 1 of the embodiment of the present invention, the identification unit 10 inputs the image I, and uses the preset parameter p in the convolutional neural network, for example, to determine the N A representative angle θ _n is identified and each accuracy value w(θ _n ) is obtained.

The weighted synthesis unit 20 weights and synthesizes the N angles θ _n by weighting the N angles θ _n according to the respective accuracy values w(θ _n ), and generates the posture information θ.

Thereby, the posture information θ can be estimated using the preset parameter p, and the parameter p can be obtained by the learning device 2 described later. Therefore, for each facial organ described in Patent Document 1 described above. There is no need to prepare a template. In other words, compared to the technique of Patent Document 1, it takes less time and effort.

Further, since the particular feature amount is not used to estimate the posture information θ, the estimation accuracy of the posture information θ can be improved as compared with the technique of Patent Document 2 that uses only the particular feature amount. .. Therefore, the posture of the subject can be estimated easily and with high accuracy.

[Learning device]
Next, the learning device according to the embodiment of the present invention will be described in detail. FIG. 4 is a block diagram showing an example of the configuration of the learning device according to the embodiment of the present invention, and FIG. 5 is a flowchart showing an example of the processing of the learning device. The learning device 2 includes an accuracy generation unit 30 and a learning identification unit 40.

The learning device 2 inputs K (group) images J _k and posture information φ _k as learning data (step S501). Then, the learning device 2 uses these learning data to learn a model for identifying the angle of the subject included in the image J _k . The learning device 2 optimizes and optimizes the parameter p of the model, that is, the parameter p used to estimate the posture of the subject, which defines the operation of the identification unit 10 of the posture estimation device 1 illustrated in FIG. 1. And output the parameter p. The parameter p is set in the identification unit 10 of the posture estimation apparatus 1 shown in FIG. K is a natural number, and k is an integer of 0 or more and less than K.

(Accuracy generation unit 30)
The accuracy generation unit 30 receives the attitude information φ _k of the learning data, and generates each learning accuracy value t _k (θ _n ) for N angles θ _n based on the attitude information φ _k (step S502). ). Then, the accuracy generation unit 30 learns a learning accuracy value sequence (t _k (θ _n )) including the learning accuracy values t _k (θ _n ) for N pieces of angle θ _n for one piece of posture information φ _k. ) _{Output n ∈ {0, 1,..., N-1}} to the learning identifying unit 40.

The posture information φ _k is an angle value (for example, by the radian method), like the posture information θ shown in FIG. 1.

Specifically, the accuracy generation unit 30 calculates an angle α (φ _k , θ _n ) formed between the vector of (the angle indicated by) the posture information φ _k and the vector of each angle θ _n , and forms the angle α. (φ _k, θ _n) to generate the learning accuracy value t _k (θ _n) corresponding to. α(φ _k , θ _n ) is a function for calculating an angle formed between the vector of (the angle indicated by) the posture information φ _k and the vector of each angle θ _n .

For example, the accuracy generation unit 30 calculates the learning accuracy value t _k (θ _n )=A (A is A for the angle θ _n when the angle α (φ _k , θ _n ) is the minimum, as shown in the following equation. A predetermined real number, for example A=1) is set. Further, when the angle α(φ _k , θ _n ) formed is not the minimum, the accuracy generation unit 30 determines the learning accuracy value t _k (θ _n )=B for the angle θ _n (B is a predetermined value smaller than A). Set a real number, for example B=0).

Further, accuracy generator 30, shown in the following formula, by applying a predetermined function f angle α (φ _k, θ _n) relative, to calculate the learning accuracy value t _k (θ _n) May be.

λは正の実定数とする。 The function f is a monotonically decreasing function in a broad sense and is a non-constant function. For example, the following Gaussian function is used as the function f.

λ is a positive real constant.

As described above, the accuracy generation unit 30 is configured to generate the learning accuracy value t _k (θ _n ) for the discrete angle θ _n from the attitude information φ _k of the continuous angle information. Thereby, the learning accuracy value t _k (θ _n ) for the discrete angle θ _n should correspond to the accuracy value w(θ _n ) generated by the identification unit 10 of the posture estimation apparatus 1 shown in FIG. You can Then, the learning identifying unit 40 corresponding to the identifying unit 10 can use this as learning data.

(Learning identification unit 40)
FIG. 6 is a block diagram showing an example of the configuration of the learning identifying unit 40. The learning identifying unit 40 includes convolutional layers 11, 12, 13, 14 and fully connected layers 15, 16.

The learning identifying unit 40 inputs the image J _{k of the} learning data, and at the same time, outputs the learning accuracy value sequence including the learning accuracy values t _k (θ _n ) for the N angles θ _n from the accuracy generation unit 30. _Input (t _k (θ _n )) _{n ∈ {0, 1,..., N-1}} . Then, the learning identifying unit 40 performs the learning process corresponding to the identifying unit 10 illustrated in FIG. 1, and the K images J _k and the learning accuracy value sequence (t _k (θ _n )) _{n ε{0, 1,..., N−1}} is used to find the optimum parameter p to be included in the identification unit 10, and the parameter p is output.

When the identifying unit 10 is configured by a neural network, the learning identifying unit 40 is also configured by a neural network similarly to the identifying unit 10, and the parameter p that is the connection weighting coefficient thereof is set to be updatable.

The learning identification unit 40 inputs the image J _{k of} learning data. Then, the learning identification unit 40 performs convolution processing on the image J _k by the convolution layers 11, 12, 13, and 14, and total combination processing by the total connection layers 15 and 16, respectively, for each of the N angles θ _n . The accuracy value w _k (θ _n ) of is calculated (step S503). As a result, a sequence of accuracy values (w _k (θ _n )) _{n ∈ {0, 1,..., N-1} including} the accuracy values w _k (θ _n ) for the N angles θ _n is obtained.

The error calculating unit (not shown) included in the learning identifying unit 40 calculates the accuracy value sequence (w _k (θ _n )) _{n ∈ {0, 1,..., N-1}} and the learning accuracy value according to the following equation. Calculate the error between the column (t _k (θ _n )) _{n ∈ {0, 1,..., N-1}} . Then, the error calculation unit calculates the error by a series of error values (d _k (θ _n )) _{n ε{0, 1,..., Which is} composed of error values d _k (θ _n ) for N angles θ _{n . N-1}} (step S504).

The back propagation unit (not shown) included in the learning identifying unit 40 _converts the error value sequence (d _k (θ _n )) _{n ε{0,1,...,N-1}} into the convolution layers 11, 12, 13, and 14. And, the full coupling layers 15 and 16 are propagated in the reverse order (back propagation) (step S505). Then, the back propagation unit updates the parameters p used in the convolutional layers 11, 12, 13, and 14 and the fully connected layers 15 and 16 by this error value back propagation method (step S506).

The learning identifying unit 40 completes the processes of steps S502 to S506 for the K images J _k and the learning accuracy value sequence (t _k (θ _n )) _{n ∈ {0, 1,..., N-1}.} It is determined whether or not (step S507). When it is determined in step S507 that the processing is not completed (step S507: N), the learning identifying unit 40 sets the next parameter k (step S508), and proceeds to step S502.

On the other hand, when the learning identifying unit 40 determines in step S507 that the processing is completed (step S507: Y), the learning identifying unit 40 outputs the parameter p updated in step S506 as the optimum parameter (step S509). The output parameter p is used by the identification unit 10 shown in FIG.

As shown in steps S502 to S508 of FIG. 5, the learning device 2 performs the processing by the error value back propagation method on the K images J _k and the posture information φ _k (learning accuracy value sequence (t _k ( θ _n )) _{n ∈ {0, 1,..., N-1}} ) may be appropriately executed. In this case, the learning device 2 _regards all of the K images J _k and the posture information φ _k (learning accuracy value sequence (t _k (θ _n )) _{n ∈ {0, 1,..., N-1}} ). , May be sequentially executed, or a predetermined number may be randomly selected from K pieces and executed. Further, the learning device 2 selects one or more from the K pieces and selects one or more images J _k and posture information φ _k (learning accuracy value sequence (t _k (θ _n )) _{n ∈ {0, 1,...,N-1}} ) may be combined into a so-called mini-batch and executed.

In this way, the learning device 2 learns and optimizes the parameter p used in the identification unit 10 shown in FIG. 1 in the accuracy generation unit 30 and the learning identification unit 40 corresponding to the identification unit 10. did. As a result, the identifying unit 10 can be operated using the optimized parameter p.

As described above, according to the learning device 2 of the embodiment of the present invention, the accuracy generating unit 30 causes the angle α (which is formed between the vector of the attitude information φ _k (the angle indicated by the angle) and the vector of each angle θ _n ( φ _k, θ _n) is calculated, the angle alpha (phi _k, to generate the learning accuracy value corresponding to _{θ n) t k (θ n} ).

When the learning identifying unit 40 is configured by a neural network like the identifying unit 10 illustrated in FIG. 1, the convolutional layers 11, 12, 13, 14 and the fully connected layer 15, with respect to the learning data image J _k . 16 is performed. Then, the learning identifying unit 40 obtains each accuracy value w _k (θ _n ) for the N angles θ _n .

Learning identification unit 40 calculates an error value d _k (θ _n) between the reliability value w _k (θ _n) the learning accuracy value t _{k (θ n),} the error value d _k (θ _n) Is propagated back to the convolutional layers 11, 12, 13, 14 and the fully connected layers 15, 16 to update the parameter p.

The accuracy generation unit 30 processes the K pieces of posture information φ _k to generate K learning accuracy values t _k (θ _n ). Then, the learning identifying unit 40 processes the K images J _k and the learning accuracy value t _k (θ _n ) to generate an optimum parameter p.

The optimum parameter p thus generated is used in the posture estimation apparatus 1 shown in FIG. 1, and the identification unit 10 of the posture estimation apparatus 1 can be operated.

With this, the posture estimation apparatus 1 can estimate the posture information θ using the parameter p, and therefore it is not necessary to prepare the template for each facial organ described in Patent Document 1 described above, which is troublesome. There is no such thing.

Further, since the particular feature amount is not used to estimate the posture information θ, the estimation accuracy of the posture information θ can be improved as compared with the technique of Patent Document 2 that uses only the particular feature amount. ..

Therefore, by using the parameter p generated by the learning device 2, the posture estimation device 1 can easily and accurately estimate the posture of the subject.

Although the present invention has been described with reference to the embodiments, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the technical idea thereof. For example, in FIG. 3, the identification unit 10 inputs the image I including horizontal W=20 pixels, vertical H=20 pixels, and three color components, but the present invention limits the number of pixels and the number of color components. Not something to do.

Further, in FIG. 3, the identification unit 10 is configured by a neural network. The present invention does not limit the identification unit 10 to a neural network, but may use a component other than the neural network. That is, the identification unit 10 inputs the image I, performs identification processing of N representative angles θ _n with respect to the subject included in the image I using the parameter p, and obtains each accuracy value w(θ _n ). It suffices as long as it is a constituent unit that obtains and outputs. The same applies to the learning identifying unit 40 shown in FIG. 6 corresponding to the identifying unit 10.

As the hardware configuration of the posture estimation device 1 and the learning device 2 according to the embodiment of the present invention, a normal computer can be used. The posture estimation apparatus 1 and the learning apparatus 2 are configured by a computer including a CPU, a volatile storage medium such as a RAM, a non-volatile storage medium such as a ROM, and an interface.

Each function of the identification unit 10 and the weighted composition unit 20 included in the posture estimation apparatus 1 is realized by causing the CPU to execute a program describing these functions. Further, each function of the accuracy generation unit 30 and the learning identification unit 40 included in the learning device 2 is also realized by causing the CPU to execute a program describing these functions.

These programs are stored in the storage medium, read by the CPU, and executed. Further, these programs can be stored and distributed in a storage medium such as a magnetic disk (floppy (registered trademark) disk, hard disk, etc.), optical disk (CD-ROM, DVD, etc.), semiconductor memory, etc., and distributed via a network. You can also send and receive.

1 Posture estimation device 2 Learning device 10 Discrimination unit 11, 12, 13, 14 Convolution layer (convolution unit)
15,16 Fully bonded layer (Fully bonded part)
20 Weighted combiner 30 Accuracy generator 40 Learning identifier

Claims (8)

In a posture estimation device that estimates the posture of a subject included in an input image,
An identification unit that identifies the angle of the subject based on the input image and obtains a certainty value corresponding to each of a plurality of preset angles;
By a weighted combination of the plurality of angles by weighting according to the certainty value corresponding to each of the plurality of angles obtained by the identification unit, a weighted combination unit that obtains posture information,
An attitude estimation device comprising: The posture estimation apparatus according to claim 1,
The posture estimating apparatus, wherein the identifying unit is configured by a neural network. In the posture estimation device according to claim 1,
When the posture information is the angle of the subject, or when the norm when the posture information is expressed as a vector value or a complex value is reliability, the posture information is the angle of the subject and the angle at the angle. A posture estimation device having reliability. In a learning device that inputs an image including a subject as learning data and posture information of the subject, learns a model using the learning data, and optimizes parameters of the model,
An accuracy generation unit that obtains a learning accuracy value corresponding to each of a plurality of preset angles based on the posture information;
Based on the learning accuracy value corresponding to each of the plurality of angles obtained by the image and the accuracy generation unit, the model for identifying the angle of the subject is learned, and the posture of the subject is estimated. A learning identification unit that updates the parameters used to
A learning device comprising: The learning device according to claim 4,
The accuracy generation unit,
The angle between the vector of the posture information and each vector of the plurality of angles is calculated, and a monotonically-decreasing and non-constant function in a broad sense is applied to the angle to correspond to each of the plurality of angles. A learning device, wherein the learning accuracy value is obtained. The learning device according to claim 4,
The accuracy generation unit,
An angle formed between the vector of the posture information and each vector of the plurality of angles is calculated, and a predetermined value is set to the learning accuracy value for an angle of the plurality of angles where the formed angle is the smallest. The learning device is characterized in that a value smaller than the predetermined value is set as the learning accuracy value for an angle that does not become the smallest among the plurality of angles. A program for causing a computer to function as the posture estimation device according to any one of claims 1 to 3. A program for causing a computer to function as the learning device according to any one of claims 4 to 6.

Images