KR20180097949A - The estimation and refinement of pose of joints in human picture using cascade stages of multiple convolutional neural networks - Google Patents

Abstract

The present invention relates to a method for predicting a joint posture in an image and, more particularly, to method for predicting a joint posture in an image by using a sequential multiple convolutional neural network capable of precisely predicting human joint coordinates in an image after optimizing training data and a convolutional neural network through a sequential multiple convolutional neural network.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for predicting joint position in an image using a sequential multiple composite neural network,

The present invention relates to a method of predicting a joint posture in an image, and more particularly, to a method for predicting a joint posture in an image by sequentially optimizing learning data and a composite neural network through a sequential multiple- The present invention relates to a method for predicting joint position in an image using a product neural network.

Neural networks are used in various fields of artificial intelligence such as pattern recognition, computer vision, natural language processing (NLP), and robotics

Among the deep learning algorithms, CNN (Convolutional Neural Network) has good performance in image and speech recognition, and it is easy to input 2D data and easy to train.

Prediction of joint position in an image through articulated neural network is applied in various fields such as surveillance, security, and sports, and efforts are made to improve the accuracy of prediction.

1. Korean Patent No. 10-1657495, 'Modularization System for Deep Learning Analysis and Image Recognition Method Using It' 2. Korean Patent No. 10-0442835, 'Face Recognition Method and Apparatus Using Artificial Neural Network'

It is an object of the present invention to provide a method for predicting intra-image joint position using a sequential multiple composite neural network capable of accurately predicting a joint position of a human in a human image.

According to an aspect of the present invention, there is provided a method of generating training data, the method comprising: (a) generating training data Xn, Yn using elements Xn and intra-image joint coordinates Yn as elements; (b) inputting arbitrary learning data (Xi, Yi) out of the learning data into a CNN (Convolutional Neural Network) and predicting joint coordinates; (c) calculating an error value between the predicted joint coordinate (Yi ') and the joint coordinate (Yi) of the arbitrary learning data, and comparing the error value with a preset reference value; (d) calculating a weight correction value using an error back-propagation algorithm when the error value is larger than the reference value, adding the weight correction value to the weight of the resultant neural network, Generating a new composite neural network < RTI ID = 0.0 > (CNN ') < / RTI > (e) generating a plurality of predicted joint coordinates (Yi ") by applying noise to the predicted joint coordinates (Yi ') and applying noise to the joint joint coordinates (Yn) of the learning data (Xn, Yn) (Yn ') from the original image (Xn) of the learning data (Xn, Yn) by cutting out a region of interest including the predicted joint coordinate (Yi " (Xn '); (g) generating new learning data Xn ', Yn' using the partial images Xn 'and noise-added original joint coordinates Yn' as elements; (h) replacing the learning data (Xn, Yn) with the new learning data (Xn ', Yn') until the error value becomes smaller than the reference value in the step (c) ) Repeating steps; (i) setting training data (Xn ", Yn ") and a composite neural network (CNN") generated when the error value becomes smaller than the reference value in the step (c) as final learning data and final concurrent neural network; And (j) inputting a target image for predicting a joint posture to the final concatenated neural network to predict the joint posture of the target image. The method of claim 1, Provides a prediction method.

In a preferred embodiment, the step of predicting joint coordinates of the composite neural network (CNN) comprises: generating a plurality of convolutional layers by passing input learning data through a convolution filter; Converting a value of '0' or less among the values of each convolution layer into '0' through a rectified linear unit (ReLU); Reducing the convolutional layers to a max operation or a mean operation to generate a pulling layer; Fully connecting the pooling radars to generate a one-dimensional feature vector; And outputting a number of predicted joint coordinates corresponding to the joint coordinates by varying the weights and multiplying the one-dimensional feature vectors.

In a preferred embodiment, the total number of the joint coordinates is 14, each joint coordinate has an x coordinate and a y coordinate, and the upper head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist , Pelvis left, pelvis right, left knee, right knee, left foot and right foot coordinates.

The present invention further provides a computer program stored in a recording medium for executing a method for predicting a joint posture in an image using the sequential multiple composite neural network by functioning as a computer.

In addition, the present invention further provides a computer provided with the computer program and performing a method of predicting a joint posture in an image using a sequential multiple compound neural network.

The present invention has the following excellent effects.

The joint posture prediction method according to an embodiment of the present invention is advantageous in that it can accurately predict the joint coordinates in the image by generating the learning data and the multi-articulated neural network newly generated sequentially so that the prediction error is less than the reference value.

In addition, the joint posture predicting method according to an embodiment of the present invention is advantageous in that the joint posture can be more accurately predicted in real human images in the future since the learning data is newly generated to include only the human image as much as possible through cropping.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a flowchart illustrating a method for predicting a joint posture in an image using a sequential multiple composite neural network according to an embodiment of the present invention; FIG.
2 is a diagram for explaining a process of generating new learning data through a sequential multiple composite neural network in a method for predicting intra-image joint position using a sequential multiple composite neural network according to an embodiment of the present invention;
3 is a diagram for explaining a process of estimating joint coordinates of a composite neural network in an intra-image joint posture prediction method using a sequential multiple composite neural network according to an embodiment of the present invention;
FIG. 4 is a diagram for explaining joint coordinate prediction results predicted by a method for predicting intra-image joint position using a sequential multiple composite neural network according to an embodiment of the present invention.

Although the terms used in the present invention have been selected as general terms that are widely used at present, there are some terms selected arbitrarily by the applicant in a specific case. In this case, the meaning described or used in the detailed description part of the invention The meaning must be grasped.

Hereinafter, the technical structure of the present invention will be described in detail with reference to preferred embodiments shown in the accompanying drawings.

However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Like reference numerals designate like elements throughout the specification.

The intra-image joint posture predicting method (hereinafter, referred to as a 'joint posture predicting method') using a sequential multiple composite neural network according to an embodiment of the present invention is a method of predicting a joint posture predicting method in a video image input through a plurality of newly- The position of the joint can be accurately predicted.

The prediction of the joint posture is the same as the prediction of the joint positions in the image.

Also, a method for predicting a joint posture according to an embodiment of the present invention is performed by a computer, and the computer stores a computer program for performing a joint posture predicting method by functioning a computer.

The computer is not only a general personal computer but also an intelligent computer including an embedded system embedded in a security device such as a server computer, a cloud system, a smart device such as a smart phone, a tablet PC, or a security camera.

In addition, the computer program may be stored in a separate recording medium, and the recording medium may be designed and configured specifically for the present invention or may be known and used by those having ordinary skill in the computer software field .

For example, the recording medium may be a magnetic medium such as a hard disk, a floppy disk and a magnetic tape, an optical recording medium such as a CD and a DVD, a magneto-optical recording medium capable of serving also as magnetic and optical recording, Or the like, or a hardware device specially configured to store and execute program instructions by itself or in combination.

In addition, the computer program may be a program consisting of program commands, local data files, local data structures, etc., alone or in combination, and may be executed by a computer using an interpreter or the like as well as machine code Lt; RTI ID = 0.0 > language code. ≪ / RTI >

Hereinafter, a joint posture prediction method according to an embodiment of the present invention will be described in detail with reference to FIG.

Referring to FIG. 1, a joint posture predicting method according to an exemplary embodiment of the present invention includes a process of sequentially generating learning data and a composite neural network sequentially (S1000), a process of finally generating the learning data and the composite neural network Thereafter, a process of predicting the joint posture of the actual image (S2000) is included.

In the step of sequentially generating the learning data and the composite neural network sequentially (S1000), training data is generated (S1100).

The learning data Xn and Yn are initial learning data and include the image Xn and the human joint coordinates Yn in the image.

Also, one image is mapped to one joint coordinate and is generated as data for thousands to tens of thousands of images.

For example, a total of 14 joint coordinates are mapped to one image, and each joint coordinate includes an x coordinate and a y coordinate in a two-dimensional image.

That is, the joint coordinate Yn includes 28 coordinate values.

The fourteen coordinates are the coordinates of the head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, pelvis left, pelvis right, left knee, right knee, Lt; / RTI >

However, the number of the coordinates and the position of the joints can be changed according to the designer.

That is, the learning data Xn, Yn is a data set including a plurality of images Xn and joint coordinates Yn mapped to respective images.

Next, arbitrary one of the learning data Xi, Yi of the learning data Xn, Yn is input to the CNN (Convolutional Neural Network), and the values output from the CNN Is predicted by the joint coordinates (Yi ') of arbitrary learning data (Xi, Yi).

In addition, the CNN is known as a deep learning pattern recognition algorithm, which is widely used in recent years, and is known to have a very good recognition rate for image or speech recognition.

More specifically, referring to FIG. 3, the CNN first passes arbitrary learning data 100 through a convolution filter having a predetermined weight to generate a plurality of convolutional layers (200).

Next, among the pixel values of each convolution layer 200 through the rectilinear linear unit (ReLU, Rectified Linear Unit), values less than '0' meaningless in the neural network are converted into '0', and the rectified linear convolution layer (300).

Next, the size of the convolution layer is reduced to a max operation or a mean operation to generate a pooling layer 400. [

In the present invention, a max pooling layer having a size reduced by half is created by selecting a maximum value among pixels in a 2x2 window to reduce the number of pixels to one half.

Next, a convolution layer is generated by using each of the pulling layers 400 as an input image, and a process of rectifying and then reducing the convolution layer is repeated 400 'to obtain a desired size and a number of pooling layers 500.

Next, the pulling layers 500 are fully connected to generate a one-dimensional feature vector 600.

Here, the one-dimensional feature vector 600 is also referred to as a fully connected layer.

Next, the process of calculating the real number 800 by calculating the weight 700 internally by adding the weight 700 to the one-dimensional feature vector 600, and calculating the real number while changing the weight 700 is repeated.

At this time, the real number 800 is the x coordinate or y coordinate of the joint coordinates predicted in the image Xi of the arbitrary learning data Xi, Yi.

In the present invention, the total number of the joint coordinates is 14, and each joint coordinate includes the x-coordinate and the y-coordinate, so that a total of 28 real numbers (800) are repeatedly calculated.

That is, the CNN outputs 28 real numbers (800) as predicted joint coordinates (Yi ').

When calculating the convolution layer 200 or integrating the convolution layer 200 in the one-dimensional feature vector 600, the weights used are arbitrary values that are not initially specified, or predetermined default values.

Referring again to FIG. 1, after the joint coordinates are predicted by the CNN, the difference between the predicted joint coordinates Yi 'and the joint coordinates Yi of the arbitrary learning data Xi and Yi (Yi-Yi ') (S1300). If the calculated error value is larger than the reference value, the weight correction value (W) is calculated through an error back-propagation algorithm (S1500).

Here, the error back propagation algorithm uses a known algorithm as a method of training the feed-forward neural network.

However, if the error value is smaller than the reference value, the learning data Xn, Yn and the CNN are set as the final learning data Xn ", Yn" and the final combined CNN " do.

This final learning data (Xn ", Yn ") and final concurrent neural network (CNN") are used for joint posture prediction of a target image to be predicted.

However, there is almost no case in which the learning data Xn and Yn generated at the beginning and the combined product neural network CNN are set as the final learning data Xn '' and Yn '' and the final concurrent neural network CNN ''.

Next, the weight correction value (W) is added to a weight used in the CNN (CNN) to generate a new weight CNN (CNN ') (S1600).

Next, the learning data (Xn, Yn) generated at the beginning is generated as new learning data (Xn ', Yn').

This process is a core process of the present invention, in which the learning data includes only the human image as much as possible, and the human image has the joint coordinates, so that when the joint posture is predicted by inputting the actual human image, .

2, a noise is given to a joint coordinate Yi '' (B) 'predicted in the CNN, and a plurality of predicted joint coordinates (Yi') ' ") '(C)'.

Separately, noises are added to the original joint coordinates Yn, which are the joint coordinates of the learning data (Xn, Yn) '(A)' to generate noise-added original joint coordinates Yn ' .

In addition, the noise-added original joint coordinates Yn 'are generated for all the original joint coordinates Yn of the learning data Xn, Yn.

Next, a region of interest (cropping) including a plurality of predicted joint coordinates (Yi ") to which noises are imparted is cut out from an original image (Xn) of the learning data (Xn, Yn) (S1800).

For example, the partial image Xn 'may be extracted as a rectangular region connecting coordinates of upper, lower, left, and rightmost outermost polygons among the plurality of predicted joint coordinates (Yi') imparted with noises, And can be extracted into a rectangular area larger by a predetermined size including the extracted rectangle.

Further, the partial image is generated for all the original images Xn of the learning data Xn, Yn.

Next, in step S1900, new learning data Xn ', Yn' '(E)' using the partial image Xn 'and the noise-added original joint coordinates Yn' as elements is generated.

The new learning data (Xn ', Yn') includes only a human region in the image, thereby improving the accuracy in predicting the joint posture in the actual human image.

Meanwhile, since the size H '× W' of the partial image Xn 'is smaller than the size H × W of the original image Xn, it is normalized to the size of the original image Xn, The new training data Xn 'and Yn' are generated by correcting the coordinate values of the original joint coordinates Yn 'by the magnitude that the partial images Xn' are normalized and mapped to each other.

Next, the arbitrary learning data (Xi, Yi) is input to the artificial neural network (CNN) to return to the step of predicting the joint coordinates (S1200). If the error value of the predicted joint coordinate (Yi ') is smaller than the reference value Repeat the process of creating new learning data and a new compound neural network until the new learning data is obtained.

Here, the arbitrary learning data Xi and Yi and the combined product neural network CNN become the previously generated learning data Xn 'and Yn' and the combined product neural network CNN '.

When the error value of the predicted joint coordinate Yi 'becomes smaller than the reference value, the learning data and the composite neural network newly generated just before are added to the final learning data Xn' ', Yn' 'and the final combined neural network CNN ") (S1400).

FIG. 4 shows a result of predicting joint coordinates while newly generating the learning data (Xi, Yi) and the composite neural network (CNN).

The initial stage 1 is the joint coordinate (Yn) of the initial learning data (Xn, Yn), and the red point is the joint coordinate (Yn) of the initial learning data (CN) (Yi ').

Stage 2 is a result of predicting the joint coordinates using the newly generated composite neural network CNN 'and the red point is the joint coordinates of the newly generated learning data Xn', Yn ' Yn ') and the green point is the predicted joint coordinate (Yi').

The stage 3 is a result of predicting the joint coordinates using the newly generated composite neural network CNN 'and the red point is the joint coordinates of the newly generated learning data Xn' and Yn ' (Yn ') and the green point is the predicted joint coordinate (Yi').

As can be seen from FIG. 4, as the composite neural network is sequentially newly generated and the prediction is repeated, the predicted joint coordinate Yi 'is set to the joint coordinate Yn' of the target learning data Xn ', Yn' It can be seen that it is getting closer.

Referring again to FIG. 1, after the final learning data (Xn ", Yn ") and final combined neural network (CNN") are determined, a target image to be actually predicted is received (S2100).

Next, the target image is input to the final combined artificial neural network (CNN "), and the joint coordinates outputted are predicted as joint positions (S2200), and the process ends.

Therefore, the method of predicting the joint posture according to an embodiment of the present invention is advantageous in minimizing the error of the joint posture prediction by using the newly generated multiple composite neural network sequentially.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is clearly understood that the same is by way of illustration and example only and is not to be taken by way of limitation, Various changes and modifications will be possible.

100: learning data 200: convolution layer
300: rectified linear convolution layer 400: polling layer
600: One-dimensional feature vector

Claims (5)

(a) generating learning data (Xn, Yn) having elements of a plurality of images (Xn) and intra-image joint coordinates (Yn) as elements;
(b) inputting arbitrary learning data (Xi, Yi) out of the learning data into a CNN (Convolutional Neural Network) and predicting joint coordinates;
(c) calculating an error value between the predicted joint coordinate (Yi ') and the joint coordinate (Yi) of the arbitrary learning data, and comparing the error value with a preset reference value;
(d) calculating a weight correction value using an error back-propagation algorithm when the error value is larger than the reference value, adding the weight correction value to the weight of the resultant neural network, Generating a new composite neural network < RTI ID = 0.0 > (CNN ') < / RTI >
(e) generating a plurality of predicted joint coordinates (Yi ") by applying noise to the predicted joint coordinates (Yi ') and applying noise to the joint joint coordinates (Yn) of the learning data (Xn, Yn) Generating an original joint coordinate Yn ';
(f) cutting out an area of interest including the predicted joint coordinates (Yi ") from the original image (Xn) of the learning data (Xn, Yn) to generate partial images (Xn ');
(g) generating new learning data Xn ', Yn' using the partial images Xn 'and noise-added original joint coordinates Yn' as elements;
(h) replacing the learning data (Xn, Yn) with the new learning data (Xn ', Yn') until the error value becomes smaller than the reference value in the step (c) ) Repeating steps;
(i) setting training data (Xn ", Yn ") and a composite neural network (CNN") generated when the error value becomes smaller than the reference value in the step (c) as final learning data and final concurrent neural network; And
(j) inputting a target image for predicting a joint posture to the final combined neural network to predict the joint posture of the target image. Way.
The method according to claim 1,
The step of predicting joint coordinates of the CNN may include:
Generating a plurality of convolution layers by passing input learning data through a convolution filter;
Converting a value of '0' or less among the values of each convolution layer into '0' through a rectified linear unit (ReLU);
Reducing the convolutional layers to a max operation or a mean operation to generate a pulling layer;
Fully connecting the pooling radars to generate a one-dimensional feature vector; And
And outputting a number of predicted joint coordinates corresponding to the joint coordinates by varying the weights and multiplying the one-dimensional feature vectors by the weights.
3. The method of claim 2,
The upper arm, the lower arm, the left arm, the right arm, the left arm, the right arm, the left arm, the right arm, the left arm, the right arm, , The left knee, the right knee, the left foot, and the coordinates of the right foot.
A computer program stored in a recording medium for executing a method of predicting a joint posture in an image using a sequential multiple composite artificial neural network according to any one of claims 1 to 4 by functioning as a computer.
A computer provided with the computer program of claim 4 and performing a joint articulated posture prediction method using sequential multiple composite neural networks.

Images