JP7230963B2 - Image processing method, device and storage medium - Google Patents

Description

The present invention belongs to the field of computer vision, and in particular relates to an image processing method, apparatus and storage medium.

With the development of modern science and technology, especially the advancement of computer technology, it became a challenging research topic for researchers to equip equipment with human visual functions, and the Department of Computer Vision was formed. rice field. The target of research on computer vision is to perceive the real world by constructing or reconstructing a world model with visual media such as images and videos. In this world, human movements bring in a large amount of information important to human society. Human-human, human-object, and human-environment interactions constitute the primary content of visible media. For this reason, it is important to study human body motion information in visual media for effective display, analysis, and understanding. Pose estimation has attracted attention as an important category in computer vision research. Of course, depending on the specific application, the subject of pose estimation may be a real person or a virtual image.

In general, pose estimation involves determining keypoints of body parts of an object to be estimated (eg, a realistic human or a virtual image) and estimating the pose based on the keypoints. However, in an image used for pose estimation, it is unavoidable that an object to be estimated is shielded (for example, shielding between one person and another, or between an object and a person). When the body part to be estimated is occluded, it is imperative to robustly detect the keypoints of each body part.
Conventional pose estimation methods exploit the relationship between adjacent keypoints to address the occlusion problem. However, the contribution by long-range keypoints has been poorly studied and in some cases neglected.

In view of the above circumstances, the present invention learns the relationship between arbitrary keypoints and develops a global relationship that can optimize the final keypoint decision result based on well-explored keypoint relationships. An object is to provide a new image processing method, apparatus, and storage medium based on the above.

First, according to the present invention, a predetermined number of keypoint feature maps corresponding to keypoints in a body part are extracted based on an input image; Extract a relationship feature map representing multiple relationships with other keypoint feature maps for each; update the keypoint feature map based on each keypoint feature map and its corresponding relationship feature map; updated An image processing method is provided, comprising determining the positions of keypoints of body parts in the input image based on the keypoint feature map.

Further, according to the image processing method according to the embodiment of the present invention, based on the predetermined number of keypoint feature maps, each keypoint feature map and another keypoint feature map are divided into a plurality of keypoint feature maps. Before extracting the relationship feature map representing the relationship, determine the center point feature map of the center point that is the average of all the key points; and updating the keypoint feature map based on the relationship feature map.

Further, according to the image processing method according to the embodiment of the present invention, based on the predetermined number of keypoint feature maps, each keypoint feature map and other keypoint feature maps are divided into a plurality of Extracting a relationship feature map representing a relationship includes sequentially taking each keypoint feature map as a current keypoint feature map, and extracting other keypoint feature maps other than the current keypoint feature map as a plurality of at least one keypoint feature map. Groups consisting of the current keypoint feature map and other keypoint feature maps are input, and convolution processing is performed to obtain the current keypoint feature map and other keypoint feature maps obtaining a relationship feature diagram representing the relationship with each of the groups consisting of .

Further, according to the image processing method according to the embodiment of the present invention, in the convolution process, each feature map in the group of other keypoint feature maps has a corresponding first weight, and the other keypoint feature map has a corresponding first weight. An initial value of a first weight is determined based on the relationship between the body part corresponding to each feature map in the group of figures and the body part corresponding to the current keypoint feature map.

Further, according to the image processing method according to the embodiment of the present invention, updating the keypoint feature map based on each keypoint feature map and the relationship feature map corresponding thereto is performed by each keypoint feature map and its Taking the corresponding relationship feature map as input, performing the convolution process; and updating each key-point feature map based on the feature map obtained by convolution.

Further, according to the image processing method according to the embodiment of the present invention, in the convolution process, each relational feature map has a corresponding second weight, and the average pixel value of all the pixel values in each relational feature map is An initial value for the second weight is determined based on the value or the maximum pixel value.

Further, according to the image processing method according to the embodiment of the present invention, a key point feature map is extracted from an input image via a feature extraction network, the relationship feature map is extracted via a relationship extraction network, and a fusion network is generated. update the keypoint feature map based on each keypoint feature map and its corresponding relational feature map through the feature extraction network, the extraction network and the fusion network using a training image as an input image to perform a feature extraction network determining rough positions of keypoints in the training images based on the keypoint feature map output from the feature extraction network, and comparing the rough positions of the keypoints with the true keypoint positions; determining a second loss function based on the deviation; and training by adjusting parameters of the feature extraction network, the relationship extraction network and the fusion network based on the first loss function and the second loss function. .

Then, the present invention comprises keypoint feature map extracting means for extracting a predetermined number of keypoint feature maps corresponding to keypoints in each body part based on an input image; , relationship feature diagram extracting means for extracting relationship feature diagrams representing a plurality of relationships with other keypoint feature diagrams for each keypoint feature diagram; and based on each keypoint feature diagram and the corresponding relationship feature diagram, the keypoint feature map updating means for updating the keypoint feature map; and keypoint feature map determining means for determining the positions of keypoints of each part of the body in the input image based on the updated keypoint feature map. An image processing device including:

Further, according to the image processing apparatus according to the embodiment of the present invention, the keypoint feature map extracting means further determines the center point feature map of the center point that is the average of all the keypoints; A relationship feature map showing a relationship with the center point feature map is extracted each time, and the key point feature map is updated based on the relationship feature map.

Further, according to the image processing apparatus according to the embodiment of the present invention, the relational feature map extracting means further sequentially sets each keypoint feature map as a current keypoint feature map, and extracts other keypoint feature maps other than the current keypoint feature map. Distributing the keypoint feature map into a plurality of groups containing at least one keypoint feature map; by taking the current keypoint feature map and each of the groups of other keypoint feature maps as input and performing a convolution process , obtaining a relationship feature map representing the relationship between the current keypoint feature map and each of a group of other keypoint feature maps.

Further, according to the image processing method according to the embodiment of the present invention, in the convolution process, each feature map in the group of other keypoint feature maps has a corresponding first weight, and the other keypoint feature map has a corresponding first weight. An initial value of a first weight is determined based on the relationship between the body part corresponding to each feature map in the group of figures and the body part corresponding to the current keypoint feature map.

Further, according to the image processing apparatus according to the embodiment of the present invention, the keypoint feature map updating means receives each keypoint feature map and the corresponding relational feature map as input, and executes convolution processing; is configured to update each keypoint feature map based on the feature map obtained by .

Further, according to the image processing apparatus according to the embodiment of the present invention, in the convolution process, each relational feature map has a corresponding second weight, and the average pixel value or An initial value of the second weight is determined based on the maximum pixel value.

Further, according to the image processing apparatus according to the embodiment of the present invention, the key point feature map is extracted from the input image via the feature extraction network, the feature map is extracted via the relationship extraction network, and the fusion network is used. based on each keypoint feature map and the corresponding relationship feature map, the feature extraction network, the extraction network and the fusion network provide training images as input images to the feature extraction network. determining the approximate positions of the keypoints in the training image based on the keypoint feature map output from the feature extraction network, and based on the deviation between the approximate positions of the keypoints and the true keypoint positions; determining a second loss function by using; and adjusting parameters of the feature extraction network, the association extraction network and the fusion network based on the first loss function and the second loss function.

Further, the present invention comprises: a memory storage device in which a computer program is stored; extracting a predetermined number of keypoint feature maps corresponding to; based on the predetermined number of keypoint feature maps, extracting relationship feature maps representing a plurality of relationships between each keypoint feature map and another keypoint feature map; updating the keypoint feature map based on each keypoint feature map and its corresponding relational feature map; based on the updated keypoint feature map, the positions of the keypoints of each part of the body in the input image. Provided is an image processing device in which processing for determining is executed.

The present invention also provides a computer-readable storage medium in which a computer program is stored and the image processing method is executed when the computer program is executed by a processor.

According to the image processing method, apparatus, and storage medium of the present invention, context information is obtained from related keypoints by considering the overall relationship, that is, the relationship between adjacent keypoints as well as the relationship with distant keypoints. By extracting and estimating the occluded keypoints, by determining the positions of all keypoints in the input image, even occluded body part keypoints can be reliably predicted.

FIG. 4 is a diagram showing a scene to which the present invention is applicable; 4 is a flow chart showing an image processing method according to an embodiment of the present invention; FIG. 4 is a diagram showing an example of setting key points of a human body; 4 is a diagram showing an example of connection relationships between key points of the human body shown in FIG. 3. FIG. FIG. 2 is a diagram showing a network configuration for updating keypoint feature maps; FIG. 4 is a flowchart for training feature extraction networks, relationship extraction networks, fusion networks; 1 is a block diagram showing the functional configuration of an image processing apparatus according to an embodiment of the present invention; FIG. FIG. 2 is a diagram showing the architecture of a computing device according to an embodiment of the example of the present invention;

Preferred embodiments of the present invention will now be described with reference to the drawings. The following description, with reference to the drawings, is intended to provide an understanding of the various details and embodiments of the invention as defined by the claims and equivalents of the invention and is illustrative. . Thus, various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the invention. For the sake of brevity, detailed descriptions of functions and configurations that are well known in the art are omitted.

First, before describing the image processing method according to the present invention, a scene to which the image processing method shown in FIG. 1 can be applied will be described. A user image is captured by a camera 101, and a plurality of key points 102 corresponding to each part of the body in the user image are specified by the image processing method of the present invention. Based on the identified keypoints 102, the pose of the virtual image 103 can be jointly controlled. For example, in an interactive game, this can track the movements of a human player to render the actions of a virtual character corresponding to the game.

Of course, the image processing method according to the embodiment of the present invention is not limited to the scene shown in FIG. 1, and can be applied to various scenes. By identifying a plurality of key points corresponding to each part of the body by the image processing method of the present invention, it can be used to identify activities, gestures and gait by tracking the changes in posture of the human body in a certain period of time. For example, possible application scenarios include, but are not limited to, intelligent video monitors, patient monitoring systems, autonomous driving, human interaction, virtual reality, human animation, intelligent rooms, intelligent security, and athlete support training.

Next, an image processing method according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 2, in the image processing method, first, in step S201, a predetermined number of keypoint feature maps corresponding to keypoints in each body part are extracted based on the input image.

Here, the corresponding body part keypoints are skeleton keypoints. Skeletal keypoints are important elements for displaying the pose of an image (eg, human, animal, virtual image) and predicting the behavior of the image. Therefore, a predetermined number of keypoints defining the skeletal part can be set in advance for a particular image.

FIG. 3 shows an example of setting human body keypoints. As shown in Figure 3, 17 human body key points are set, from top to bottom: nose P1, left eye P2, right eye P3, left ear P4, right ear P5, left shoulder P6, right shoulder P7, left hand elbow P8, right hand Elbow P9, left wrist P10, right wrist P11, left hip P12, right hip P13, left knee P14, right knee P15, left ankle P16 and right ankle P17 are shown. That is, when the keypoints of the human body are set as shown in FIG. 3, when the user image is input in step S201, a feature diagram having 17 body part keypoints shown in FIG. 3 is extracted. In the following description, the human body keypoint setting shown in FIG. 3 will be described as an example. Of course, the setting of keypoints can be optionally made according to the specific application.

The keypoint feature map extraction process in step S201 is realized by a convolutional neural network. Here, the convolutional neural network is called a feature extraction network, for example. A convolutional neural network includes an input layer, a hidden layer, and an output layer. The input layer receives an identifying image. The suppression layer performs feature analysis and extraction on the input image. The output layer finally outputs a predetermined number of extracted keypoint feature maps.

Constrained layers include convolutional layers, pooled layers, and fully connected (fully connected) layers. In convolutional neural networks, multiple convolutional layers (complex structures such as series, parallel and cross-connections) are used for feature extraction. Different hierarchies of features are extracted by multiple convolution, normalization and pooling processes with different parameters. First, the input image is checked and convolved using convolution to obtain the convolution mapping. The convolutional mapping is then normalized using the usual calibration linear unit and batch normalization method to obtain a normalized convolutional mapping. In addition, a maximum or average pooling process is performed on the normalized convolutional mappings. After feature extraction in the convolution layer, the output feature map is transferred to the pooling layer for feature selection and information filtering. A preset pooling function included in the pooling layer provides the ability to replace single point results in the feature map with the feature map statistics of adjacent regions. For example, max pooling replaces the value of a single pixel point with the maximum value of pixel points in adjacent regions. Average pooling replaces the value of a single pixel point with the average value of pixel points in adjacent regions. To obtain diverse features, the suppression layer can be set with multiple convolutional layers and multiple pooling layers, and the above process can be repeated multiple times. In addition, various multi-scale feature mappings are extracted through a plurality of sampling processes.

The output layer finally outputs a predetermined number of feature maps based on the multiscale feature mapping extracted by the suppression layer. When the input image is an actual user image, 17 keypoint feature maps are output from the output layer in the case of human body keypoint setting as shown in FIG.

Here, the 17 keypoint feature maps output from the output layer are ordered according to the types of predetermined body parts. For example, the first keypoint feature map is the left eye map, the second keypoint map is the right eye map, and the third keypoint map is the nose map. Each keypoint feature map contains only keypoints corresponding to one body part. For example, in the left eye feature map, only features associated with left eye keypoints are included. Also, if only one person appears in the input image, only the features associated with one keypoint feature are included. For example, the left eye feature map contains only features associated with one left eye keypoint. However, if multiple people appear in the input image, each keypoint feature map may include features associated with multiple keypoints. For example, a left eye feature map may contain features associated with two left eye keypoints, but the two keypoints correspond to the same body part (left eye).

Return to Figure 2. After extracting a predetermined number of key point feature maps in step S201, the process proceeds to step S202. In step S202, based on the predetermined number of keypoint feature maps, relationship feature maps are extracted that represent a plurality of relationships between each keypoint feature map and another keypoint feature map.

Specifically, each keypoint feature map is sequentially set as the current keypoint feature map, and the following steps are repeatedly executed.

First, keypoint feature maps other than the current keypoint feature map are sorted into a plurality of groups.

Next, the current keypoint feature map and each group of other keypoint feature maps are input, and convolution processing is performed. For example, the convolution processing here is realized via a convolutional neural network. Also, for example, the convolutional neural network here is called a relationship extraction network. Of course, the convolutional neural network here differs from the convolutional neural network for extracting the keypoint characteristic diagram in step S201 in specific network parameters (eg, network depth, node weight, etc.). Two or more keypoint feature maps are provided as inputs to a convolutional neural network. The convolutional neural network performs feature extraction on the input keypoint feature map and outputs a feature description of the input keypoint feature map. Since each input keypoint feature map is associated with a keypoint, the output feature map resulting from the convolution may be viewed as representing the relationship between the keypoints. In other words, by convolving two or more keypoint feature maps, the relationship between the current keypoint feature map and other keypoint feature maps is obtained. Note that the feature map as a result of the convolution process is called a relationship feature map.

For example, for each keypoint feature diagram, a relationship feature diagram representing the relationship with each feature diagram in other keypoint feature diagrams is extracted. Alternatively, it is possible to extract a relational feature diagram representing relationships with a plurality of feature diagrams in other keypoint feature diagrams. Here, keypoints corresponding to a plurality of feature maps in other keypoint feature maps are not limited to keypoints adjacent to the current keypoint. Adjacent keypoints are keypoints with a distance of 1, as described below. Keypoints corresponding to multiple feature maps in other keypoint feature maps include long-distance keypoints far away from the current keypoint (eg, keypoints with a distance greater than 1).

Also, as a possible embodiment, in said convolution process, each feature map in a group of other keypoint feature maps has a corresponding first weight.

i番目のキーポイント特徴図について、j番目キーポイント特徴図との関係特徴図を決定する場合に、j番目のキーポイント特徴図に対応する第1重みは二次元マトリックスG_17×17の要素G(I,j)に基づいて取得する。キーポイント間の距離が短いほど、その関連が大きくなると考えられるため、大きな重みを割り当てる。逆の場合にも同様である。 For example, determine the value of the first weight based on the relationship between the body part corresponding to each feature map in a group of other keypoint feature maps and the body part corresponding to the current keypoint feature map. . Taking the body keypoint setting shown in FIG. 3 as an example, the weight corresponding to each keypoint feature map is obtained from a two-dimensional matrix G _17×17 with 17 rows×17 columns. A two-dimensional matrix G _17×17 represents the distance d between keypoints. Here, the distance d is determined by the number of intermediate points connected between two keypoints. The connection relationship between key points (for example, connectable or unconnectable points, how to connect) is determined in advance based on the skeletal structure of the human body. For example, FIG. 4 shows an example of connection relationships between keypoints. As shown in FIG. 4, the nose P1 and right eye P3 can be directly connected, but the nose P1 and left buttock P12 are always connected via the center point Pc. Here, the center point Pc is the mean point based on the positions of 17 body keypoints. Thus, the distance between the keypoint nose P1 and the keypoint right eye P3 is 1, and the distance between the keypoint nose P1 and the left hip P12 is 2. Correspondingly, the elements G(1,3)=1 and G(1,12)=2 in G _17×17 . All the elements contained in G _17×17 and the corresponding element values are:
(Outside 1)

For the i-th keypoint feature map, when determining the relationship feature map with the j-th keypoint feature map, the first weight corresponding to the j-th keypoint feature map is the element G of the two-dimensional matrix G _17×17 Get based on (I,j). Smaller distances between keypoints are assumed to be more relevant and are therefore assigned higher weights. The same is true in the opposite case.

Also, as a possible embodiment, the first weight corresponding to each feature map in the other keypoint feature map is obtained by learning through a training process described below. Note that the weight determined based on the distance between the keypoints as described above is used as the initial value. Supervised training is performed based on the initial values.

In step S202, for each keypoint feature diagram, a relationship feature diagram in another keypoint feature diagram is extracted. That is, in addition to the relationships between neighboring keypoints, the relationships between distant keypoints are also considered.

More information (eg, information associated with other keypoints) is introduced by extracting relationship feature maps for each keypoint feature map with other keypoint feature maps. Also, when one keypoint feature diagram is regarded as a single channel, a plurality of relational feature diagrams corresponding to the keypoint feature diagram are generated, so the number of channels of the feature diagram is considered to increase. Thereby, the expressive power of the feature is strengthened by combining the keypoint feature map and the corresponding relational feature map, which contributes to the detection of subsequent keypoints.

Take the human body keypoint setting shown in FIG. 3 as an example. In this case, for each keypoint feature map, a relationship feature map with each of the other 16 keypoint feature maps is extracted. That is, 16 relationship feature maps are obtained for one keypoint feature map. Of course, for one keypoint feature map, any number (such as two) of relationship feature maps in 16 other keypoint feature maps may be extracted. For example, when extracting related feature maps with two other keypoint feature maps for one keypoint feature map, eight relational feature maps are obtained.

Also, as another possible implementation, the initial optimization may be performed on a predetermined number of keypoint feature maps obtained in step S201 before performing the processing in step S202. Specifically, the following steps are further included between step S201 and step S202.

First, identify the center point feature map. As shown in Figure 4 above, the center point is the average of all keypoints. A centroid feature map is obtained in two ways: The first method is to use the feature map obtained by averaging all the keypoint feature maps as the center point feature map. The second method is to construct a convolutional neural network for extracting the keypoint feature map to output a predetermined number (e.g. 17) of keypoint feature maps and also to output the center point feature map. to adjust.

Next, for each keypoint feature map, a relationship feature map with the center point feature map is extracted, and the keypoint feature map is updated based on the related feature map. Here, the specific method of extracting the relationship feature map with the center point feature map for each keypoint feature map is similar to the method of extracting the relationship feature map between keypoint feature maps. For example, a new feature map is generated by performing a convolution process on the keypoint feature map and the center point feature map. In this new feature map, the relationship between the keypoint and the center point is fused. Also, let this new feature map be the optimized keypoint feature map.

For each keypoint feature map, extract the relationship feature map with the center point feature map, and replace the keypoint feature map with the obtained relationship feature map, thereby adding centrally related features in each keypoint feature map. . Furthermore, when extracting the relationship between each keypoint in subsequent step S202, the spatial relationship between these keypoints can be extracted, and the skeletal relationship can be extracted. Such relationships are more accurate.

Return to Figure 2. After step S202, the process proceeds to step S203. In step S203, based on each keypoint feature map and its corresponding relational feature map, the keypoint feature map is updated.

Specifically, each keypoint feature map and the corresponding relationship feature map are input, and convolution processing is performed to fuse the features in the keypoint feature map and the corresponding relationship feature map. Then, each keypoint feature map is updated based on the feature map obtained by convolution. Here, a convolutional neural network is called a fusion network, for example. Of course, a specific Network parameters (eg, network depth, node weights, etc.) are different. In addition, the specific structure of the convolutional neural network is well known to those skilled in the art, and the number of layers of the convolutional neural network can be increased or decreased as appropriate according to specific functional design requirements. , the weight parameter of each node can be modified. A keypoint feature map and all corresponding relationship feature maps are provided as inputs in multiple channels to a convolutional neural network. Then, one channel feature map is output from the neural network as the convolution result of the keypoint feature map and all the corresponding relational feature maps. The keypoint feature map is updated by replacing the keypoint feature map with the output feature map.

In the convolution process, each relationship feature map has a corresponding second weight. The second weight is obtained by learning through training, which will be described later, based on the initial value. As a possible embodiment, the initial value of the second weight is obtained as follows. Determine the initial value of the second weight based on the average pixel value or the maximum pixel value of all pixel points in the relationship feature map. Specifically, the maximum value or average value among the pixel values of all pixels included in the relationship feature map is determined, and based on the maximum value or average value, the second weight corresponding to the relationship feature map confirm.

FIG. 5 is a diagram showing a network for updating keypoint feature maps. For the sake of convenience, FIG. 5 shows an example of specifying a relational feature diagram with other keypoint feature diagrams for each keypoint characteristic diagram, using three keypoint characteristic diagrams as an example.

In FIG. 5, blocks 501, 502, and 503 indicate feature maps of keypoints i, j, and k extracted in step S201, respectively. Here, keypoint i represents only one kind of keypoint, for example, the nose keypoint, but it does not mean that only keypoint i is included in the feature map of keypoint i. For example, for multiple people, there may be multiple nose keypoints.

In FIG. 5, six different convolutional neural networks are represented by circles 504-509, respectively, for which the relational feature diagram extraction process is performed in step S202, and each resulting relational feature diagram is represented by squares 510-515. For example, 504 represents a convolutional neural network whose input is two channels (ie keypoint i feature map and keypoint j feature map) and whose output is one channel (ie the relational map between i and j). Blocks 510, 511 represent the relationship feature diagram between keypoints i and j and the relationship feature diagram between keypoints i and k.

Also, in FIG. 5, circles 516 to 518 each represent three different convolutional neural networks, which perform feature fusion processing in step S203. The inputs of the three convolutional neural networks are 3-channel input (i.e. 1 keypoint feature map and 2 related relational feature maps) and 1-channel output (i.e. new feature map obtained after feature fusion). is. Also, the feature maps output by the three convolutional neural networks to be feature fused are represented by blocks 519-521, respectively, and are the updated feature maps for keypoints i, j, and k.

The dashed box 500 shown in FIG. 5 contains all convolutional neural networks. The entire network contained in the dashed frame 500 is defined as a relational code network, receives keypoint feature maps as input, encodes relationships between arbitrary keypoint feature maps, and finally outputs relationship-encoded feature maps. and the updated keypoint feature map.

Return to Figure 2. After step S203, the process proceeds to step S204. In step S204, the keypoint locations of the body parts in the input image are determined based on the updated keypoint feature map. For example, in one possible embodiment, a heatmap corresponding to each keypoint is obtained based on the updated keypoint feature map. In this heat map, the pixel value of each pixel is the probability value of whether that pixel is a keypoint or not. The approximate value above is a value between 0 and 1. Thus, keypoint locations are established in the heatmap by identifying local extrema.

For this reason, in the image processing method according to the embodiment of the present invention, by considering the relationship between distant keypoints in addition to the relationship between neighboring keypoints, the relationship between adjacent keypoints is considered. , which can locate keypoints more accurately and achieve accurate pose recognition. The present invention also reliably predicts keypoints in occluded body parts. For example, if a keypoint is occluded, the keypoint's approximate position (for example, if the right eye is occluded, right eye-nose, right eye-mouth , right-eye-right-hand relationship, etc., predicts the approximate position of the right eye).

As described above, the processing of steps S201, S202, and S203 in FIG. 2 is realized through a specific convolutional neural network. Specifically, a keypoint feature map is extracted from an input image through a feature extraction network, a relationship feature map is extracted through a relationship extraction network, and each keypoint feature map and its corresponding Update the keypoint feature map based on the relationship feature map.

Next, specific steps for training the feature extraction network, relationship extraction network, and fusion network will be described with reference to FIG.

First, in step S601, training images are supplied to the feature extraction network as input images. Here, the difference between the training image and the input image in step S201 is that the training image has marking data. That is, in the training data, the positions of key points on each part of the body are known.

Next, in step S602, the rough locations of keypoints in the training images are determined based on the keypoint feature map output from the feature extraction network. Here, since the keypoint feature map output from the feature extraction network is not subjected to feature fusion, which will be described later, the keypoint positions obtained based on such a keypoint feature map are relatively rough and unclear. Accurate. A first loss function is determined based on the deviation between the rough keypoint position and the true keypoint position.

Next, in step S603, a second loss function is determined according to the deviation between the keypoint positions updated through the feature extraction network, the relational extraction network and the fusion network and the actual keypoint positions.

Finally, in step S604, adjust the parameters of the feature extraction network, the relationship extraction network and the fusion network according to the first loss function and the second loss function. A total loss function is determined based on the first loss function and the second loss function. For example, averaging over the first loss function and the second loss function yields the total loss function. Of course, the present invention is not limited to this. A total loss function is obtained by combining the first loss function and the second loss function in another way. The training of the feature extraction network, the relationship extraction network and the fusion network is finished when the total loss function converges.

Thus, the present invention learns the relationships between keypoints by an end-to-end approach and is easily applied to keypoint relationship extraction for arbitrary targets.

As described above, according to the image processing method according to the embodiment of the present invention, the keypoint positions of each part of the body included in the input image are obtained. In addition, the keypoint positions based on each part of the body can further estimate the pose of the image (such as a virtual person or a real user) according to the specific application scene, such as monitoring or man-machine interaction. can be done. Of course, the present invention is not limited to this. Any application that requires the use of extracted keypoint locations will likewise apply the method according to the present invention.

The image processing method according to the present invention has been described in detail above with reference to FIGS. An image processing apparatus according to the present invention will be described below with reference to FIG.

The image processing apparatus 700 includes keypoint feature map extraction means 701 , relationship feature map extraction means 702 , keypoint feature map update means 703 and keypoint position determination means 704 .

The keypoint feature map extracting device 701 extracts a predetermined number of keypoint feature maps corresponding to keypoints in each body part based on the input image.

Here, the corresponding body part keypoints are skeleton keypoints. Skeletal keypoints are important elements for displaying the pose of an image (eg, human, animal, virtual image) and predicting the behavior of the image. Therefore, a predetermined number of keypoints defining the skeletal part can be set in advance for a particular image. For example, as shown in FIG. 3, key points corresponding to each part of the body can be set in advance.

The keypoint feature map extraction device 701 implements extraction processing of keypoint feature maps by, for example, a convolutional neural network. Here, the convolutional neural network is called a feature extraction network, for example. A convolutional neural network includes an input layer, a suppression layer, and an output layer. The input layer receives an identifying image. The suppression layer performs feature analysis and extraction on the input image. The output layer finally outputs a predetermined number of extracted keypoint feature maps.

Constrained layers include convolutional layers, pooled layers, and fully connected layers. In convolutional neural networks, multiple convolutional layers (complex structures such as series, parallel and cross-connections) are used for feature extraction. Different hierarchies of features are extracted by multiple convolution, normalization and pooling processes with different parameters. First, the input image is checked and convolved using convolution to obtain the convolution mapping. The convolutional mapping is then normalized using the usual calibration linear unit and batch normalization method to obtain a normalized convolutional mapping. In addition, a maximum or average pooling process is performed on the normalized convolutional mappings. After feature extraction in the convolution layer, the output feature map is transferred to the pooling layer for feature selection and information filtering. A preset pooling function included in the pooling layer provides the ability to replace single point results in the feature map with the feature map statistics of adjacent regions. For example, max pooling replaces the value of a single pixel point with the maximum value of pixel points in adjacent regions. Average pooling replaces the value of a single pixel point with the average value of pixel points in adjacent regions. To obtain diverse features, the suppression layer can be set with multiple convolutional layers and multiple pooling layers, and the above process can be repeated multiple times. In addition, various multi-scale feature mappings are extracted through a plurality of sampling processes.

The output layer finally outputs a predetermined number of feature maps based on the multiscale feature mapping extracted by the suppression layer. When the input image is an actual user image, 17 keypoint feature maps are output from the output layer in the case of human body keypoint setting as shown in FIG.

Based on the predetermined number of keypoint feature maps, the relationship feature map extracting device 702 extracts a relationship feature map representing a plurality of relationships between each keypoint feature map and another keypoint feature map.

Specifically, the above feature map extracting means 702 is configured to repeatedly execute the following steps with each keypoint feature map as the current keypoint feature map in sequence.
First, keypoint feature maps other than the current keypoint feature map are sorted into a plurality of groups.

Next, the current keypoint feature map and each group of other keypoint feature maps are input, and convolution processing is performed. For example, the convolution processing here is realized via a convolutional neural network. Also, for example, the convolutional neural network here is called a relationship extraction network. Of course, the convolutional neural network here is the convolutional neural network for the keypoint feature diagram extraction means 701 to extract the keypoint characteristic diagram, and the specific network parameters (for example, network depth, node weight, etc.) are different. Two or more keypoint feature maps are provided as inputs to a convolutional neural network. The convolutional neural network performs feature extraction on the input keypoint feature map and outputs a feature description of the input keypoint feature map. Since each input keypoint feature map is associated with a keypoint, the output feature map resulting from the convolution may be viewed as representing the relationship between the keypoints. In other words, by convolving two or more keypoint feature maps, the relationship between the current keypoint feature map and other keypoint feature maps is obtained. Note that the feature map as a result of the convolution process is called a relationship feature map.

For example, for each keypoint feature diagram, a relationship feature diagram representing the relationship with each feature diagram in other keypoint feature diagrams is extracted. Alternatively, it is possible to extract a relational feature diagram representing relationships with a plurality of feature diagrams in other keypoint feature diagrams. Here, keypoints corresponding to a plurality of feature maps in other keypoint feature maps are not limited to keypoints adjacent to the current keypoint. Adjacent keypoints are keypoints with a distance of 1, as described below. Keypoints corresponding to multiple feature maps in other keypoint feature maps include long-distance keypoints far away from the current keypoint (eg, keypoints with a distance greater than 1).

Also, as a possible embodiment, in said convolution process, each feature map in a group of other keypoint feature maps has a corresponding first weight.

For example, determine the value of the first weight based on the relationship between the body part corresponding to each feature map in a group of other keypoint feature maps and the body part corresponding to the current keypoint feature map. . For example, as described above, the corresponding weights are determined based on the distances between keypoints. Smaller distances between keypoints are assumed to be more relevant and are therefore assigned higher weights. The same is true in the opposite case.

Also, as a possible embodiment, the first weight corresponding to each feature map in the other keypoint feature map is obtained by learning through a training process described below. Note that the weight determined based on the distance between the keypoints as described above is used as the initial value. Supervised training is performed based on the initial values.

The relationship feature diagram extraction device 702 extracts relationship feature diagrams in other keypoint feature diagrams for each keypoint feature diagram. That is, the relational feature map extractor 702 considers the relation between distant keypoints in addition to the relation between adjacent keypoints.

More information (eg, information associated with other keypoints) is introduced by extracting relationship feature maps for each keypoint feature map with other keypoint feature maps. Also, when one keypoint feature diagram is regarded as a single channel, a plurality of relational feature diagrams corresponding to the keypoint feature diagram are generated, so the number of channels of the feature diagram is considered to increase. Thereby, the expressive power of the feature is strengthened by combining the keypoint feature map and the corresponding relational feature map, which contributes to the detection of subsequent keypoints.

Take the human body keypoint setting shown in FIG. 3 as an example. In this case, for each keypoint feature map, a relationship feature map with each of the other 16 keypoint feature maps is extracted. That is, 16 relationship feature maps are obtained for one keypoint feature map. Of course, for one keypoint feature map, any number (such as two) of relationship feature maps in 16 other keypoint feature maps may be extracted. For example, when extracting related feature maps with two other keypoint feature maps for one keypoint feature map, eight relational feature maps are obtained.

As another possible implementation, the keypoint feature map extracting means 701 extracts the keypoint feature map from the input image, and then extracts the center point feature map of the center point, which is the average of all the keypoints. determining; for each keypoint feature map, it is configured to extract a relationship feature map with the center point feature map, and update the keypoint feature map based on said related feature map.

As mentioned above, the centroid feature map is obtained in two ways. The first method is to use the feature map obtained by averaging all the keypoint feature maps as the center point feature map. The second method is to construct a convolutional neural network for extracting the keypoint feature map to output a predetermined number (e.g. 17) of keypoint feature maps and also to output the center point feature map. to adjust.

Further, a specific method of extracting a relationship feature map with the center point feature map for each keypoint feature map is similar to a method of extracting a relationship feature map between keypoint feature maps. For example, a new feature map is generated by performing a convolution process on the keypoint feature map and the center point feature map. In this new feature map, the relationship between the keypoint and the center point is fused. Also, let this new feature map be the optimized keypoint feature map.

For each keypoint feature map, extract the relationship feature map with the center point feature map, and replace the keypoint feature map with the obtained relationship feature map, thereby adding centrally related features in each keypoint feature map. . Furthermore, when the subsequent relationship feature diagram extraction means 702 extracts the relationship between each keypoint, it is possible to extract the spatial relationship between these keypoints as well as the skeletal relationship. Such relationships are more accurate.

The keypoint feature map updating means 703 updates the keypoint feature map based on each keypoint feature map and the relationship feature map corresponding thereto.

Specifically, the keypoint feature map updating means 703 receives each keypoint feature map and the corresponding relationship feature map as input, and executes convolution processing to convert the keypoint feature map and the corresponding relationship feature map into configured to blend features in Then, each keypoint feature map is updated based on the feature map obtained by convolution. Here, a convolutional neural network is called a fusion network, for example. Of course, this fusion network, the feature extraction network for the keypoint feature map extraction means 701 to extract the keypoint feature maps, and the relationship feature map extraction means 702 to extract the relationship feature maps between the keypoint feature maps The specific network parameters (eg, network depth, node weights, etc.) are different for relational extraction networks. In addition, the specific structure of the convolutional neural network is well known to those skilled in the art, and the number of layers of the convolutional neural network can be increased or decreased as appropriate according to specific functional design requirements. , the weight parameter of each node can be modified. A keypoint feature map and all corresponding relationship feature maps are provided as inputs in multiple channels to a convolutional neural network. Then, one channel feature map is output from the neural network as the convolution result of the keypoint feature map and all the corresponding relational feature maps. The keypoint feature map is updated by replacing the keypoint feature map with the output feature map.

In the convolution process, each relationship feature map has a corresponding second weight. The second weight is obtained by learning through training, which will be described later, based on the initial value. As a possible embodiment, the initial value of the second weight is obtained as follows. Determine the initial value of the second weight based on the average pixel value or the maximum pixel value of all pixel points in the relationship feature map. Specifically, the maximum value or average value among the pixel values of all pixels included in the relationship feature map is determined, and based on the maximum value or average value, the second weight corresponding to the relationship feature map confirm.

A keypoint position determining means 704 determines the keypoint positions of the body parts in the input image based on the updated keypoint feature map. For example, in one possible embodiment, the keypoint location determiner 704 obtains a heatmap corresponding to each keypoint based on the updated keypoint feature map. In this heat map, the pixel value of each pixel is the probability value of whether that pixel is a keypoint or not. The approximate value above is a value between 0 and 1. Thus, keypoint locations are established in the heatmap by identifying local extrema.

For this reason, the image processing apparatus according to the embodiment of the present invention provides sufficient relation sensing, that is, by considering the relation between distant keypoints in addition to the relation between adjacent keypoints, a , which can locate keypoints more accurately and achieve accurate pose recognition. The present invention also reliably predicts keypoints in occluded body parts. For example, if a keypoint is occluded, the keypoint's approximate position (for example, if the right eye is occluded, right eye-nose, right eye-mouth , right-eye-right-hand relationship, etc., predicts the approximate position of the right eye).

As described above, the keypoint feature map extraction means 701 extracts the keypoint feature map from the input image via the feature extraction network, and the relational feature map extraction means 702 extracts the relational feature map via the relation extraction network. , and the keypoint feature map updating means 703 updates the keypoint feature map based on each keypoint feature map and its corresponding relational feature map through the fusion network.

The feature extraction network, the relationship extraction network and the fusion network are trained by the training steps described with reference to FIG. 6 above.

As described above, according to the image processing apparatus according to the embodiments of the present invention, the keypoint positions of each part of the body included in the input image are obtained. In addition, the keypoint positions based on each part of the body can further estimate the pose of the image (such as a virtual person or a real user) according to the specific application scene, such as monitoring or man-machine interaction. can be done. Of course, the present invention is not limited to this. Any application that requires the use of extracted keypoint locations will likewise apply the apparatus according to the present invention.

Also, the image processing apparatus of the present invention includes a memory in which a computer program is stored, and a processor. When the processor executes the computer program stored in the memory, key points in each body part are displayed based on an input image. extracting a predetermined number of keypoint feature maps corresponding to the points; based on the predetermined number of keypoint feature maps, creating a relationship feature map representing a plurality of relationships with other keypoint feature maps for each keypoint feature map; based on each keypoint feature map and its corresponding relationship feature map, update the keypoint feature map; based on the updated keypoint feature map, keypoints of each part of the body in the input image A process of determining the position is performed.

Specifically, the method and apparatus according to embodiments of the present invention are implemented by the architecture of computing device 800 shown in FIG. As shown in FIG. 8, the computing device 800 includes a bus 810, one or more CPUs 820, read only memory (ROM) 830, random access memory (RAM) 840, a communication port 850 connected to a network, input/output components. 860, hard disk 870, etc. In computing device 800, memories such as ROM 830 and hard disk 870 store various data and files used for processing and/or communication of the image processing method of the present invention, and program instructions executed by the CPU. Of course, the architecture shown in FIG. 8 is only exemplary, and one or more elements of the computing device shown in FIG. 8 may be omitted in the actual device, if desired.

Furthermore, embodiments of the present invention are implemented with a computer-readable storage medium. A computer readable storage medium according to an embodiment of the present invention stores computer readable instructions. When the computer readable instructions are executed by the processor, the image processing method according to the embodiments of the invention described with reference to the above figures is performed. Examples of the computer-readable storage medium include, but are not limited to, volatile and/or non-volatile storage. Volatile memory includes, for example, random access memory (RAM) and cache. The non-volatile memory includes, for example, ROM (ROM), hard disk, flash memory, and the like.

So far, the image processing method, apparatus, and storage medium according to the embodiments of the present invention have been described with reference to FIGS. 1 to 8. FIG. According to the image processing method, apparatus, and storage medium of the present invention, context information is obtained from related keypoints by considering the overall relationship, that is, the relationship between adjacent keypoints as well as the relationship with distant keypoints. We determine the positions of all keypoints in the input image by extracting and estimating the occluded keypoints. Thus, even keypoints on occluded body parts are reliably predicted.

In this specification, the term "include" or variations thereof means to include non-exclusively. Thus, a process, method, article or apparatus comprising a set of elements may include those elements as well as other elements not explicitly stated or inherent. Unless specifically limited, similar elements may be present separately and concurrently in a process, method, article, or apparatus containing these elements.

Finally, the above series of processes includes processes that are performed in chronological order according to the order described above, as well as processes that are performed in parallel or separately, out of chronological order.

According to the above description of the embodiments, it is obvious to those skilled in the art that the present invention can be realized by combining hardware platforms required for software. can also Based on such an understanding, the present invention may appear in whole or in part in the form of a software product contributing to the background art. This computer software product is stored in a storage medium such as ROM/RAM, floppy disk, optical disk, etc., and computer equipment (PC, server, network device, etc.) is described in each embodiment or part of the embodiments of the present invention. contains instructions that cause the method to be executed.

The present invention has been described in detail above. Although the principles and embodiments of the present invention have been described herein by applying specific examples, these examples are intended to aid in understanding the principles and core ideas of the present invention. Also, those skilled in the art may change the specific embodiments and application range based on the idea of the present invention. Furthermore, the above should not be taken as limiting the present invention.

Claims (11)

An image processing method executed by a processor,
extracting a predetermined number of keypoint feature maps, each corresponding to a keypoint in the body part, based on the input image;
Based on the predetermined number of keypoint feature maps, for each keypoint feature map, keypoints adjacent to the keypoint corresponding to the keypoint feature map and distant keypoints not adjacent to the keypoint feature map are calculated. extracting relationship feature maps representing multiple relationships with other keypoint feature maps including distance keypoints ;
updating the keypoint feature map based on each keypoint feature map and its corresponding relationship feature map representing multiple relationships with the other keypoint feature map ; and based on the updated keypoint feature map. to determine the positions of key points of body parts in said input image. Based on the predetermined number of keypoint feature maps , before extracting relationship feature maps representing a plurality of relationships between the keypoint feature map and the other keypoint feature maps for each keypoint feature map,
determining a center point feature map of the center point that is the average of all key points; and extracting a relationship feature map showing the relationship between each key point feature map and the center point feature map, based on the relationship feature map 2. The image processing method of claim 1, comprising updating said keypoint feature map. Extracting relationship feature diagrams representing a plurality of relationships between the keypoint feature diagram and the other keypoint feature diagrams for each keypoint feature diagram based on the predetermined number of keypoint feature diagrams,
Each keypoint feature map is taken as the current keypoint feature map in turn, and the other keypoint feature maps other than the current keypoint feature map are sorted into a plurality of groups containing at least one keypoint feature map; and the current keypoint feature map. and each group consisting of other keypoint feature maps are input, and convolution processing is performed to obtain a relationship representing the relationship between the current keypoint feature map and each group consisting of other keypoint feature maps get the feature map,
2. The image processing method according to claim 1, comprising repeatedly performing: in the convolution process, each feature map in a group of other keypoint feature maps has a corresponding first weight;
determining an initial value of a first weight based on the relationship between the body part corresponding to each feature map in a group of other keypoint feature maps and the body part corresponding to the current keypoint feature map; 4. The image processing method according to claim 3. Based on each keypoint feature map and its corresponding relationship feature map, updating the keypoint feature map includes:
Taking each keypoint feature map and its corresponding relationship feature map as input, performing a convolution process; and updating each keypoint feature map based on the feature map obtained by the convolution. The described image processing method. In the convolution process, each relationship feature map has a corresponding second weight,
6. The image processing method according to claim 5, wherein the initial value of the second weight is determined based on an average pixel value or maximum pixel value of all pixels in each relationship feature map. extracting each keypoint feature map from the input image through a feature extraction network, extracting the relationship feature map through a relationship extraction network, and extracting each keypoint feature map and its corresponding relationship feature map through a fusion network; An image processing method for updating the keypoint feature map based on
The feature extraction network, the relationship extraction network and the fusion network are
providing training images with known keypoint locations for each part of the body as input images to the feature extraction network;
Determine the rough position of each keypoint in the training image based on the feature map of each keypoint output from the feature extraction network, and determine the difference between the rough position of each keypoint and the true keypoint position. determine a first loss function based on, and determine a second loss function based on the deviation between the position of each keypoint updated through the feature extraction network, the relationship extraction network and the fusion network and the true keypoint position; and adjusting parameters of the feature extraction network, the relationship extraction network and the fusion network based on the first loss function and the second loss function. Image processing method. a key point feature map extracting means for extracting a predetermined number of key point feature maps corresponding to key points in each body part based on the input image;
Based on the predetermined number of keypoint feature maps, for each keypoint feature map, keypoints adjacent to the keypoint corresponding to the keypoint feature map and distant keypoints not adjacent to the keypoint feature map relationship feature diagram extracting means for extracting relationship feature diagrams representing a plurality of relationships with other keypoint feature diagrams including distance keypoints ;
keypoint feature map updating means for updating the keypoint feature map based on each keypoint feature map and its corresponding relational feature map representing a plurality of relationships with the other keypoint feature maps;
and a keypoint feature map determining means for determining the positions of the keypoints of each part of the body in the input image based on the updated keypoint feature map. a memory in which a computer program is stored;
and a processor connected to the memory, wherein
An image processing apparatus, wherein the processor is configured to implement the image processing method according to any one of claims 1 to 7 by executing the computer program. A program for causing a computer to execute the image processing method according to any one of claims 1 to 7. 11. A computer-readable storage medium storing the program according to claim 10.

Images