TW202203212A - Key point detection method, electronic device and computer readable storage medium - Google Patents

Abstract

The present disclosure provides a key point detection method, an electronic device and a computer readable storage medium, the method includes: acquiring the image to be detected; generating image feature map and multiple key point heat map based on the image to be detected; image feature map is used to represent the relative position relationship between the target objects in the image to be detected; each key point heat map contains one type of key points of the image to be detected, and different types of key points correspond to different parts of the target object; based on the image feature map and multiple key point heat map, the initial key point graph model is generated; the initial key graph model contains the information of different types of key points in the image to be detected and the information of connecting edges, each connecting edge is the edge between two different types of key points; the initial key point graph model is pruned by multiple connected edges, until the key points in the processed key point graph model are clustered into multiple clusters, and the key point information belonging to each target object is obtained.

Description

Key point detection method, electronic device and computer-readable storage medium

The present invention relates to the technical field of image processing, and in particular, to a key point detection method, an electronic device and a computer-readable storage medium.

Human key point detection and tracking is the basis of video analysis, and has important application prospects in the field of security and motion analysis. Bottom-up multi-person pose detection technology is widely used due to its high computational efficiency. In general, the bottom-up method first predicts the position of each key point, and then clusters each key point to obtain a complete human pose.

In the current method, a graph segmentation algorithm or a heuristic clustering algorithm can be used to cluster each key point. The clustering process is only used as a post-processing operation, and does not directly supervise the clustering results, so that the key points are The clustering process is less accurate.

In view of this, the present invention provides at least a key point detection method, an electronic device and a computer-readable storage medium.

In a first aspect, the present invention provides a key point detection method, including: acquiring an image to be detected; generating an image feature map and a plurality of key point heatmaps based on the to-be-detected image; is used to characterize the relative positional relationship between each target object in the image to be detected; each of the key point heatmaps contains a type of key points of the image to be detected, and key points of different types correspond to the different parts of the target object; based on the image feature map and a plurality of the key point heatmaps, an initial key point graph model is generated; the initial key point graph model contains the keys of different categories in the image to be detected Information of points and information of connecting edges, each connecting edge is an edge between two different types of key points; the initial key point graph model is pruned multiple times for the connecting edges, until the processed Multiple key points in the key point graph model are clustered into multiple clusters to obtain key point information belonging to each target object.

Using the above method, an initial keypoint graph model corresponding to the image to be detected can be generated based on the generated image feature map and multiple keypoint heatmaps. Since the initial keypoint graph model includes image feature maps and keypoint heatmaps The image feature map can represent the relative positional relationship between different target objects in the image to be detected, so that the initial key point graph model can be pruned at the connected edges to obtain the information of each target object. The key point information can more accurately distinguish the key points of different target objects, so as to improve the accuracy of key point clustering.

In a possible implementation, the information of the key points includes location information, category information, and primitive feature information; the information of each key point in the initial key point graph model is determined according to the following steps: Figure, determine the position information of each key point; based on the position information of each key point, extract the primitive feature information of the key point from the image feature map, and based on the key point to which the key point belongs The category label of the graph determines the category information corresponding to the key point.

In a possible implementation manner, the method further includes: for each key point in the initial key point graph model, based on the information of the key point and the relationship between the key point in the key point graph model and the key point. There is information about other key points with connecting edges between them, and the fusion feature of the key points is determined; and the initial key point graph model is subjected to multiple pruning processing of the connecting edges, including: based on the initial key points. For the fusion feature of each key point included in the point graph model, the initial key point graph model is subjected to multiple times of pruning of the connecting edges.

In a possible implementation manner, the initial keypoint graph model is subjected to multiple times of pruning of the connecting edges, until multiple keypoints in the processed keypoint graph model are clustered into multiple clusters. , including: performing a first processing process for the current key point graph model: based on the fusion features of two key points corresponding to each connecting edge in the current key point graph model Perform key point clustering of the same target object on adjacent key points among the key points to obtain at least one macro node; wherein, the macro node includes a plurality of adjacent key points after clustering; and based on each The fusion feature of the key points included in the macro node, determine the fusion feature of the macro node; Based on the obtained at least one of the macro node and the current key point graph model, the current key point graph model Perform the pruning process of the current connection edge, and obtain the key point graph model after the current pruning process; after executing the current first processing process, the key points after the current pruning process are The graph model is used as the current key point graph model, and the currently determined fusion feature of the macro node and the macro node is used as the key point and the fusion feature of the key point in the current key graph model, and the execution of the The first processing procedure is described, until the multiple key points in the processed key point graph model are clustered into multiple clusters.

In a possible implementation, based on the fusion feature of two key points corresponding to each connecting edge in the current key point graph model, the adjacent key points among the multiple key points included in the current key point graph model are compared. The key point is to perform key point clustering of the same target object to obtain at least one macro node, including: determining the weight of the connection edge based on the fusion feature of the two key points corresponding to the connection edge, and the weight represents the The probability that the two key points corresponding to the connecting edge belong to the same target object; based on the weight of each connecting edge included in the current key point graph model, the adjacent key points among the multiple key points included in the current key point graph model The key point is to perform key point clustering of the same target object to obtain at least one macro node. Here, by determining the weight of each connecting edge, the weight represents the probability that the two key points corresponding to the connecting edge belong to the same target object, and then based on the weight of each connecting edge, the current key point graph model includes multiple key points. The adjacent key points in the point are clustered by the key points of the same target object to obtain at least one macro node. For example, two key points with larger corresponding weights are clustered together to obtain a macro node, which makes the macro node determine more accurate.

In a possible implementation manner, based on the obtained at least one of the macro node and the current key point graph model, the current key point graph model is subjected to the pruning process of the current connection edge, and the result is obtained. The key point graph model after the current pruning process includes: based on the obtained at least one of the macro node and the current key point graph model, determining the connection edges to be deleted, and extracting from the current key point graph model Delete the connection edge to be deleted; use at least one of the macro node and other key points other than the key points included in the macro node in the current key point graph model as pruning processing The key points of , and the remaining connection edges after deletion are used as the connection edges after pruning, and the key point graph model after the current pruning process is obtained.

In a possible implementation, the described based on the obtained at least one macro node and the current key point graph model, determine the connection edge to be deleted, including: based on the category of each key point included in the at least one macro node information, and category information of other key points in the current key point graph model except for the key points included in at least one macro node, to determine the connection edge to be deleted.

In a possible implementation, based on the category information of each key point included in the at least one macro node and other key points other than the key points included in the at least one macro node in the current key point graph model The category information, determining the connection edge to be deleted, including: for any connection edge in the current key point graph model, the two key points corresponding to the arbitrary connection edge are the key points in different macro nodes point, and if there are key points with the same category information in the two macro nodes corresponding to any connecting edge, determine that any connecting edge is the connecting edge to be deleted; In the case that the two key points are the key points in the same macro node, determine that any connecting edge is the connecting edge to be deleted; one of the two key points corresponding to the any connecting edge is a macro point. A key point in a set node, another key point is not a key point in a macro node, and the macro node corresponding to any connecting edge has a key point with the same type information as the other key point , determine that any connecting edge is the connecting edge to be deleted.

In a possible implementation manner, the key point information of each target object is generated by a pre-trained target neural network; wherein, the target neural network is a neural network to be trained including a macro node discriminator. After training, the macro node discriminator is used to discriminate whether multiple key points included in each macro node belong to the same target object.

In a possible implementation manner, the neural network to be trained is trained by the following steps to obtain a pre-trained target neural network: obtaining a sample image; The to-be-trained neural network of the node discriminator is trained to obtain a pre-trained target neural network.

In a possible implementation manner, based on the sample image, the neural network to be trained including the macro node discriminator is trained to obtain a pre-trained target neural network, including: based on the sample image , the neural network to be trained is trained to obtain a prediction result, and the prediction result includes the detection result of the macro node discriminator, the prediction category of each key point, and the prediction position information of each key point; determining a first loss value based on the detection result of the macro node discriminator; and based on the predicted category of each key point, the predicted location information of each key point, and the information carried in the sample image The labeling result is to determine a second loss value; wherein, the labeling result includes the labeling category that each key point belongs to the corresponding target object, and the labeling position information of each key point; based on the first loss value and the second The loss value is used to train the neural network to be trained to obtain a pre-trained target neural network.

In a possible implementation manner, after obtaining the key point information of each target object in the to-be-detected image, the method further includes: determining the behavior type of the target object based on the key point information corresponding to each target object. .

In a possible implementation manner, after obtaining the key point information of each target object in the image to be detected, the method further includes: determining at least one target object based on the key point information corresponding to each target object. position information of the target part, and generate special effect information for the at least one target part according to the position information of the at least one target part.

For descriptions of the effects of the following apparatuses, electronic devices, etc., reference may be made to the descriptions of the above-mentioned methods, which will not be repeated here.

In a second aspect, the present invention provides a key point detection device, comprising: an acquisition module, configured to acquire an image to be detected; a first generation module, configured to generate an image feature map based on the to-be-detected image and a multiple key point heat maps; the image feature maps are used to represent the relative positional relationship between key points of each target object in the to-be-detected image; each of the key-point heat maps contains the to-be-detected image The key points of one category of the image, the key points of different categories correspond to different parts of the target object; the second generation module is configured to generate an initial image based on the image feature map and a plurality of the key point heat maps A key point graph model; the initial key point graph model includes information of different types of key points in the to-be-detected image and information of connecting edges, and each connecting edge is an edge between two different types of key points The processing module is configured to carry out the pruning processing of the connection edges for multiple times on the initial key point graph model, until the multiple key points in the processed key point graph model are clustered into multiple clusters, obtaining The key point information belonging to each target object respectively.

In a third aspect, the present invention provides an electronic device, comprising: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the The processor and the memory communicate through a bus, and when the machine-readable instructions are executed by the processor, the steps of the method for detecting key points according to the first aspect or any one of the implementation manners are performed.

In a fourth aspect, the present invention provides a computer-readable storage medium on which a computer program is stored, and when the computer program is run by a processor, the key points described in the first aspect or any one of the above-mentioned embodiments are executed. The steps of the detection method.

In a fifth aspect, the present invention provides a computer program product, comprising computer-readable code, when the computer-readable code is executed in an electronic device, a processor in the electronic device executes the code for implementing one or more of the above In the embodiment, the server executes the above method. In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments It is only a part of the embodiments of the present invention, but not all of the embodiments. The elements of the embodiments of the invention generally described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely representative of selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present invention.

In general, the bottom-up method first predicts the position of each key point, and then clusters each key point to obtain a complete human pose. In the current method, a graph segmentation algorithm or a heuristic clustering algorithm can be used to cluster each key point. The clustering process is only used as a post-processing operation, and does not directly supervise the clustering results, so that the key points are The clustering process is less accurate.

In order to improve the accuracy of the key point clustering process, an embodiment of the present invention provides a key point detection method.

To facilitate understanding of the embodiments of the present invention, a key point detection method disclosed in the embodiments of the present invention is first introduced in detail.

Referring to FIG. 1, it is a schematic flowchart of a key point detection method provided by an embodiment of the present invention. The method includes S101-S104, wherein: S101, acquiring an image to be detected. S102, based on the image to be detected, generate an image feature map and a plurality of key point heat maps; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image; each key point heat map It contains the key points of one category of the image to be detected, and the key points of different categories correspond to different parts of the target object. S103, an initial key point graph model is generated based on the image feature map and a plurality of key point heat maps; the initial key point graph model includes information of different types of key points in the image to be detected and information of connecting edges, each connection An edge is an edge between two keypoints of different classes. S104, performing multiple pruning processing of connecting edges on the initial key point graph model, until multiple key points in the processed key point graph model are clustered into multiple clusters, to obtain key point information belonging to each target object respectively .

In the above method, an initial key point map model corresponding to the image to be detected can be generated based on the generated image feature map and a plurality of key point heat maps, since the initial key point map model includes the image feature map and the key point heat map. The image feature map can represent the relative positional relationship between different target objects in the image to be detected, so that the initial key point graph model can be pruned at the connected edges to obtain the information of each target object. The key point information can more accurately distinguish the key points of different target objects, so as to improve the accuracy of key point clustering.

For S101 and S102: the image to be detected may be any image including multiple target objects. The target object can be a person, that is, the key points of a plurality of human bodies included in the object to be detected are detected.

During the implementation process, the acquired image to be detected can be input into the trained keypoint detection neural network to generate an image feature map and multiple keypoint heatmaps; and based on the image feature map, multiple keypoints The heatmap, and the trained keypoint detection neural network, determine each keypoint of each target object.

Here, each keypoint heatmap contains keypoints of one category of the image to be detected, and keypoints of different categories correspond to different parts of the target object. For example, the categories of key points may be head, neck, hand, etc., and the key point heatmap may be an image containing head key points, or the key point heat map may be an image containing neck key points etc.; or, the categories of the key points may be the set first category, second category, etc., wherein the first category of key points may be the key points on the thumb, and the second category of key points may be the key points on the index finger, etc., Furthermore, the keypoint heatmap may be an image containing keypoints of the first category, or the keypoint heatmap may be an image containing keypoints of the second category, or the like. Among them, the categories of key points and the number of categories can be set according to actual needs. And, the number of key points corresponding to each target object may be set according to actual needs, for example, the number of key points corresponding to each target object may be 17, 105, and so on.

Here, the number of keypoint heatmaps is consistent with the set number of keypoint categories. For example, when the set number of keypoint categories is 17, the number of keypoint heatmaps generated based on the image to be detected is also 17. Among them, the number of keypoints for each category can be one.

The number of image feature maps can be one or more. Wherein, in the case where the number of image feature maps is one, the image feature map may represent the relative positional relationship between parts of each target object in the image to be detected and corresponding to the key points of various categories. When the number of image feature maps is multiple, the number of image feature maps can be the same as the number of key point heat maps, that is, each image feature map can represent a category of each target object in the image to be detected The relative positional relationship between the parts corresponding to the key points. Among them, the size of the image feature map is consistent with the size of the keypoint heatmap.

During implementation, image feature maps and multiple keypoint heatmaps can be obtained by setting different loss functions in the keypoint detection neural network.

For S103: Here, the information of each key point can be extracted from multiple key point heatmaps and image feature maps, and each key point containing information is used as a node, and the edges between different types of key points are used as The connected edges form the initial keypoint graph model. The information of the connection edge may be information corresponding to the connection relationship between two key points. For example, the information of the connection edge 1 may be that the key point P1 corresponding to the connection edge 1 has a connection relationship with the key point P2.

In some embodiments of the present invention, the information of the key point includes location information, category information, and primitive feature information. Among them, the information of each key point in the initial key point graph model can be determined according to the following steps: based on the key point heat map, determine the position information of each key point; based on the position information of each key point, extract the key point from the image feature map The feature information of the primitives of the point, and the category information corresponding to the key point is determined based on the category label of the key point heat map to which the key point belongs.

In the implementation process, the position information of each key point can be determined based on the primitive value of each primitive point in the key point heat map. Exemplarily, for each key point heat map, a primitive point whose primitive value is a maximum value may be selected as a key point, and the position information of the selected primitive point may be determined as the position information of the key point. Among them, in the case where the value of a certain primitive point in the key point heat map is greater than the primitive value of the surrounding primitive points, the primitive value of the primitive point is considered to be a maximum value, and the primitive point is a key point . After obtaining the position information of each primitive point, the primitive value of the primitive point corresponding to the position information can be extracted from the image feature map, and the extracted primitive value can be determined as the primitive feature information of the key point . At the same time, the category information corresponding to the key point can also be determined according to the category label of the key point heat map to which each key point belongs. For example, in the case where the category label of the keypoint heatmap G1 is head, the category information of each keypoint included in the keypoint heatmap G1 is the head keypoint; the category label of the keypoint heatmap G2 is neck In the case of , the category information of each key point included in the key point heatmap G2 is the neck key point.

For S104: In some embodiments of the present invention, before performing multiple pruning processes on the initial key points, the method may further include: for each key point in the initial key point graph model, information and key points based on the key point The information of other key points with connected edges in the graph model and the key points is used to determine the fusion characteristics of the key points. Further, performing pruning processing of multiple connection edges on the initial key point graph model may include: based on the fusion feature of each key point included in the initial key point graph model, performing multiple connection edges on the initial key point graph model. Pruning treatment.

Here, a corresponding fusion feature can be generated for each key point in the initial key point graph model, and then based on the fusion feature of each key point, the initial key point graph model is pruned multiple times to connect edges.

During the implementation process, for each key point, other key points in the initial key point graph model that have connecting edges with the key point can be determined, and the key point can be generated based on the information of the key point and other key points. fusion features. Exemplarily, a graph neural network (Graph Neural Network, GNN) can be used to determine the fusion feature of each key point in the initial key point graph model, and based on the fusion of each key point included in the initial key point graph model feature, the initial keypoint graph model is pruned multiple times to connect the edges.

In the above-mentioned embodiment, by for each key point, based on the information of the key point and the information of other key points that have connecting edges with the key point, the fusion feature corresponding to the key point is generated. The fusion feature can not only characterize the characteristics of the key point, but also the relationship between the key point and other key points, so that based on the fusion features corresponding to each key point, the initial key point graph model can be more accurate. The pruning process of the secondary connection edge can more accurately determine the key point information corresponding to each target object.

In some embodiments of the present invention, the initial keypoint graph model is subjected to multiple pruning processing of connecting edges, until multiple keypoints in the processed keypoint graph model are clustered into multiple clusters, including: Execute the first processing process for the current key point graph model: Step 1. Based on the fusion feature of two key points corresponding to each connecting edge in the current key point graph model, analyze the multiple key points included in the current key point graph model. Perform key point clustering of the same target object on adjacent key points to obtain at least one macro node; wherein, the macro node includes multiple adjacent key points after clustering; and based on the key points included in each macro node The fusion characteristics of the points are determined to determine the fusion characteristics of the macro nodes; step 2, based on the obtained at least one macro node and the current key point graph model, the current key point graph model is pruned for the current connection edge, and the current key point graph model is obtained. The key point graph model after the second pruning process; Step 3: After the current first processing process is completed, the key point graph model after the current pruning process is used as the current key point graph model, and the current determined The macro node and the fusion feature of the macro node are used as the key points in the current key graph model and the fusion feature of the key points, and the first processing process is performed again until multiple key points in the processed key point graph model are aggregated. Classes are clusters. Here, the initial key point graph model can be used as the current key point graph model, and the first processing process is performed for the first time to obtain the key point graph model after pruning; and the key point graph model after the first pruning process is As the current key point graph model, each macro node and the fusion feature corresponding to each macro node obtained after the first pruning process are used as the key points in the current key point graph model and the fusion feature of the key points, Perform the second first processing process until the key points in the processed key point graph model are clustered into multiple clusters, and the number of clusters obtained by clustering is the same as the number of target objects included in the image to be detected. Similarly, each cluster includes all key points corresponding to a target object, that is, each key point of each target object in the image to be detected is obtained.

Under the above-described embodiment, in each first processing process, adjacent key points are clustered once to obtain at least one macro node, and the multiple key points included in each macro node are the key points of the same target object. , by performing multiple first processing processes on the initial key point graph model until multiple key points in the processed key point graph model are clustered into multiple clusters, the key point information of each target object is obtained, so that The obtained key point information corresponding to each target object is more accurate.

The first processing procedure is described in detail as follows: In step 1, in some embodiments of the present invention, based on the fusion feature of two key points corresponding to each connecting edge in the current key point graph model, the current key point graph model The adjacent key points of the multiple key points included in the point graph model perform key point clustering of the same target object to obtain at least one macro node, including: A1, based on the fusion feature of the two key points corresponding to the connection edge, determine the connection The weight of the edge, the weight represents the probability that the two key points corresponding to the connecting edge belong to the same target object. A2, based on the weight of each connecting edge included in the current keypoint graph model, perform keypoint clustering of the same target object on adjacent keypoints among multiple keypoints included in the current keypoint graph model to obtain at least one macroset node. Here, the weight corresponding to each connecting edge can be determined, and the weight represents the probability that the two key points on both sides of the connecting edge belong to the same target object. Exemplarily, the weight of each connecting edge may be determined according to the fusion feature of the two key points corresponding to each connecting edge through the trained edge discriminator.

In some embodiments of the present invention, keypoint clustering may be performed on adjacent keypoints among multiple keypoints included in the current keypoint graph model based on the weight of each connecting edge included in the current keypoint graph model, At least one macro node is obtained, wherein the multiple key points included in each macro node are key points belonging to the same target object. For example, two key points corresponding to connecting edges with larger weights can be clustered together to obtain a macro node. Each macro node includes two key points in the current key point graph model, so that the weight sum of the connection edges included in at least one macro node obtained after clustering is larger. For example, in the case of obtaining two macro nodes after performing this key point clustering on the current key point graph model, the sum of the connection edge weights included in the two macro nodes obtained after clustering can be larger.

After the macro nodes are obtained, the fusion feature of each macro node can be determined before the next first processing process. That is, the fusion feature of each key point included in the macro node can be fused to obtain the fusion feature corresponding to the macro node. In the implementation process, the fusion feature of each key point included in the macro node may be pooled to obtain the fusion feature of the macro node.

Here, by determining the weight of each connecting edge, the weight represents the probability that the two key points corresponding to the connecting edge belong to the same target object, and then based on the weight of each connecting edge, the current key point graph model includes multiple key points. The adjacent key points in the point are clustered by the key points of the same target object to obtain at least one macro node. For example, two key points with larger corresponding weights are clustered together to obtain a macro node, which makes the macro node determine more accurate.

In step 2, based on the obtained at least one macro node and the current key point graph model, the current key point graph model is subjected to the pruning process of the current connection edge, and the key point graph model after the current pruning process is obtained, Including: B1, based on the obtained at least one macro node and the current key point graph model, determine the connection edge to be deleted, and delete the connection edge to be deleted from the current key point graph model. B2, use at least one macro node and other key points in the current key point graph model except the key points included in the macro node as the key points after pruning, and use the remaining connection edges after deletion as the pruning After processing the connected edges, the key point graph model after the current pruning process is obtained.

Here, in step B1, first, according to the obtained at least one macro node and the current key point graph model, determine the connection edge to be deleted in the current pruning process, and remove the connection edge to be deleted from the current key point graph removed from the model.

In some embodiments of the present invention, in step B1, determining the connection edge to be deleted based on the obtained at least one macro node and the current key point graph model includes: based on each key included in the at least one macro node The category information of the points and the category information of other key points in the current key point graph model except the key points included in the at least one macro node are used to determine the connection edges to be deleted. Here, considering that each target object can include only one key point of the same category, for example, each target object can only include one key point of the head category, one key point of the neck category, and one key point of the left foot category. key points, therefore, can be based on the category information of each key point included in at least one macro node, and the category information of other key points in the current key point graph model except the key points included in at least one macro node , to determine the connecting edge to be deleted.

In some embodiments of the present invention, based on the category information of each key point included in the at least one macro node, and other key points in the current key point graph model except the key points included in the at least one macro node category information to determine the connecting edges to be deleted, including: For any connecting edge in the current keypoint graph model: Condition 1, the two key points corresponding to any connecting edge are key points in different macro nodes, and the two macros corresponding to any connecting edge When there are key points with the same category information in the nodes, it is determined that any connecting edge is the connecting edge to be deleted. Condition 2: In the case that the two key points corresponding to any connecting edge are key points in the same macro node, determine that any connecting edge is a connecting edge to be deleted. Condition 3. Among the two key points corresponding to any connecting edge, one key point is the key point in the macro node, the other key point is not the key point in the macro node, and the macro point corresponding to any connecting edge is the key point in the macro node. If there is a keypoint with the same type information as another keypoint in the set node, it is determined that any connecting edge is the connecting edge to be deleted.

Here, for any connecting edge in the current key point graph model, if the connecting edge satisfies any one of condition 1, condition 2, and condition 3, the connecting edge is considered to be the connecting edge to be deleted, otherwise , the connecting edge does not belong to the connecting edge to be deleted.

In Condition 1, if the two key points corresponding to the connecting edge are key points in different macro nodes, it can be determined whether there are key points with the same category information in the two macro nodes corresponding to the connecting edge. If there are key points with the same category information, the connecting edge is the connecting edge to be deleted; if there is no key point with the same category information, the connecting edge does not belong to the connecting edge to be deleted. In Condition 2, when the two key points corresponding to the connecting edge are key points in the same macro node, the connecting edge is the connecting edge to be deleted. In condition 3, one of the key points corresponding to the connecting edge is the key point included in the macro node, and the other key point is not the key point in the macro node, that is, the other key point is the current key point graph In the case of other key points other than the key points included in the macro node in the model, it can be judged whether there is a key point with the same category information as another key point in the macro node corresponding to the connecting edge, and in the presence of the In the case of a key point, the connection edge is the connection edge to be deleted; in the absence of the key point, the connection edge does not belong to the connection edge to be deleted.

In the above embodiment, considering that the key point information of each target object includes only one key point of the same type, that is, the key point information of each target object includes a head key point, a neck key point, a left key point, and a left key point. Therefore, based on the category of the two key points corresponding to the connecting edge, the connecting edge to be deleted can be determined, and the pruned key point graph model can be generated, and then the next first processing process can be performed until Multiple keypoints in the processed keypoint graph model are clustered into multiple clusters.

In B2, after the connection edge to be deleted is deleted, at least one macro node and other key points in the current key point graph model except the key points included in the macro node can be used as the pruned Key points, the remaining connection edges after deletion are used as connection edges after part-time processing, and the key point graph model after the current pruning process is obtained. That is, in the key point graph model after the current pruning process, when the key point is a macro node, the fusion feature of the key point is the fusion feature corresponding to the macro node.

In step 3, after the current first processing process is performed, the key point graph model after the current pruning process can be used as the current key point graph model corresponding to the next pruning process, and the currently determined key point graph model The macro node and the fusion feature of the macro node are used as the key points in the current key graph model and the fusion feature of the key points, and the first processing process is performed again until multiple key points in the processed key point graph model are aggregated. The class is a plurality of clusters, that is, until there is no connecting edge in the processed keypoint graph model, then each keypoint of each target object included in the image to be detected is obtained.

Referring to Figure 2A, the figure shows a schematic diagram of the pruning process in a key point detection method; based on the image to be detected, an image feature map 22 (ie, Feature maps) and multiple key point hotspots can be generated Figure 21 (ie Heatmaps), and then based on the image feature map 22 and multiple key point heat maps 21, an initial key point map model 23 is generated, wherein the circles in the initial key point map model 23 are identified as key points, and the dotted line is Connecting edges between keypoints of different classes. Next, a corresponding fusion feature can be generated for each key point, and the weight of each connecting edge can be determined by the trained edge discriminator 24, and based on the weight of each connecting edge included in the initial key point graph model 23, the initial The adjacent key points among the multiple key points included in the key point graph model perform key point clustering of the same target object to obtain at least one macro node 25 . Then, based on the obtained at least one macro node 25 and the initial key point graph model 23, determine the connection edge to be deleted, and delete the connection edge to be deleted from the initial key point graph model, and then at least one macro The node, and other key points except the key points included in the macro node in the initial key point graph model are used as the key points after pruning processing, and the remaining connection edges after deletion are used as the connection edges after pruning processing, obtaining For the key point graph model 26 after the current pruning process, the key point graph model 26 after the current pruning process is used as the current key point graph model to perform the first processing process again, until there are more than one key point graph model in the processed key point graph model. The keypoints are clustered into multiple clusters, that is, the processing result obtained in the last step in Figure 2A. Wherein, when the neural network to be trained is trained, the macro node discriminator 27 can also be used to discriminate each generated macro node, that is, to determine whether the key points included in each macro node belong to the same macro node. target object, and train the neural network to be trained based on the detection result of the macro node discriminator 27 to obtain a trained target neural network.

In some embodiments of the present invention, the key point information of each target object is generated by a pre-trained target neural network; wherein, the target neural network is obtained by training a to-be-trained neural network including a macro node discriminator Yes, the macro node discriminator is used to discriminate whether multiple key points included in each macro node belong to the same target object. The image to be detected can be input into the pre-trained target neural network to obtain the key point information of each target object included in the image to be detected. Among them, the category of each key point corresponding to each target object and the number of key points can be set according to actual needs.

In the implementation process, the macro node discriminator may not be included in the pre-trained target neural network. That is, the macro node discriminator can judge whether the obtained multiple key points in each macro node belong to the same target object during the training process of the neural network to be trained. In the above embodiment, a target neural network is obtained by training a neural network to be trained including a macro node discriminator, wherein the macro node discriminator is used to discriminate a plurality of key points included in each macro node Whether they belong to the same target object can make the accuracy of the target neural network obtained by training higher.

In some embodiments of the present invention, the neural network to be trained is trained through the following steps to obtain a pre-trained target neural network.

Obtain a sample image; and based on the sample image, train the neural network to be trained including the macro node discriminator to obtain a pre-trained target neural network.

Based on the sample images, the neural network to be trained including the macro node discriminator is trained to obtain a pre-trained target neural network, which may include: 1. Training the neural network to be trained based on the sample images, The prediction result is obtained, and the prediction result includes the detection result of the macro node discriminator, the prediction type of each key point, and the predicted position information of each key point. 2. Determine the first loss value based on the detection result of the macro node discriminator; and determine the second loss value based on the predicted category of each key point, the predicted position information of each key point, and the annotation results carried in the sample image Loss value; among them, the labeling result includes the labeling category that each key point belongs to the corresponding target object, and the labeling position information of each key point. 3. Based on the first loss value and the second loss value, the neural network to be trained is trained to obtain a pre-trained target neural network. Here, the sample image carries the labeling result, and the labeling result includes the labeling category to which each key point belongs to the corresponding target object, and the labeling position information of each key point. Input the sample image into the neural network to be trained to obtain the prediction result, wherein the prediction result includes the detection result of the macro node discriminator, the prediction type of each key point, and the prediction position information of each key point. Then, the neural network to be trained can be trained based on the prediction result and the labeling result, and the trained target neural network can be obtained. The first loss value can be determined based on the detection results of the macro node discriminator, and the second loss can be determined based on the predicted category of each key point, the predicted location information of each key point, and the annotation results carried in the sample image value; through the sum of the first loss value and the second loss value, the neural network to be trained is trained to obtain the target neural network.

In some embodiments of the present invention, after obtaining the key point information of each target object in the image to be detected, the method further includes: determining the behavior type of the target object based on the key point information corresponding to each target object. Here, after obtaining the information of each key point of each target object, the information of each key point of each target object can be input into the behavior detection neural network to determine the behavior type of the target object, for example, the behavior type You can run, walk, raise your arms, etc.

In some embodiments of the present invention, after obtaining the key point information of each target object in the image to be detected, the method further includes: determining at least one target part of the target object based on the key point information corresponding to each target object and generate special effect information for at least one target part according to the position information of at least one target part. Here, according to the information of each key point of each target object, the position information of at least one target part of the target object can be determined, and based on the preset special effect information corresponding to the target part, the corresponding special effect information is generated at the position of the target part . The target part may be an arm, a head, a hand, or the like. For example, the arm position of the target object can be determined according to the information of each key point of the target object, and based on the preset special effect information of the arm, the special effect information corresponding to the arm is generated at the arm position of the target object.

In related technologies, human key point detection and tracking are the basis of video analysis, and have important application prospects in the field of security and motion analysis. Bottom-up multi-person pose detection technology is widely used due to its high computational efficiency. Generally, the bottom-up method is generally divided into two steps. The first step is to predict the Gaussian response map of the key points and obtain the position of each key point. In the second step, each key point is clustered to obtain the complete human pose. However, in the related art, the clustering step adopts a graph segmentation algorithm or a heuristic clustering algorithm. Clustering is only used as a post-processing operation and does not directly supervise the clustering results. It can be seen that the related art has the following shortcomings: 1) The clustering step generally adopts a graph segmentation algorithm or a heuristic clustering algorithm, which is only used as a post-processing operation, and does not directly supervise the clustering results; 2) Ordinary graph clustering algorithms cannot make full use of the prior information of the hierarchical structure of the human body. For example, the hierarchical information is: a person can be decomposed into upper body and lower body; the upper body can be decomposed into head, shoulders, arms; It consists of 5 key points of the face; 3) Ordinary graph clustering algorithm only uses local information and ignores global human body information.

In order to solve the above problem, an embodiment of the present invention provides a key point detection method. In the method, first, based on a differentiable "hierarchical graph clustering category module", the detection and clustering of key points are combined to perform end-to-end. Ground training. Then, the clustering results are supervised, and the clustering loss can be directly transmitted back to the underlying feature extraction network for overall network optimization. In this way, the network pays more attention to the key points of wrong clustering results, and can perform feature learning more effectively. On the one hand, the hierarchical graph clustering algorithm repeatedly operates to cluster the key points of the target object step by step, forming a hierarchical structure from the key points-limbs-the entire target object, which can be used for clustering structures at each level. Supervision can better preserve the prior information of the target object's hierarchy. On the other hand, by introducing the macro-node discriminator, the features inside the entire macro node can be discriminated, and the global feature information can be better preserved.

2A is a schematic flowchart of a pruning process in a key point detection method provided by an embodiment of the present invention. By judging whether two key points belong to the same target object, the key points of the same target object are gathered together.

1) Extract the key point information in the image to be detected, and construct the initial key point graph model according to the key point information; here, first, the information of the key points needs to be extracted to construct the initial key point graph model G={V, E}. The initial key point graph model G is divided into two parts: the key point V and the edge E, where the key point V is the information of each key point, that is, it includes "the type of the key point T, the coordinates of the key point X, and the feature F of the key point". The edge E represents the relationship between key points, that is, whether they belong to the same target object. After constructing the initial keypoint graph model, extract the correlation features.

2) Use GNN to learn correlation features. Use edge convolution (EdgeConv) to build a graph convolutional neural network model, convolve the constructed key point graph model (Graph), and update the features of key points.

3) Update the similarity matrix between key points: Then train an edge discriminator (Edge Discriminator) to discriminate each pair of key points to determine whether the pair of key points belong to the same target object. Use discriminant information to update the similarity matrix between keypoints.

4) Clustering: Use the similarity matrix between key points to perform a clustering algorithm to gather adjacent key points together to form a new macro node (the key points obtained after clustering become macro node). Train a Macro-Node Discriminator to judge whether the key points inside a macro node belong to the same target object.

5) Graph Pruning: According to some prior constraints of the target object structure, the key point graph model (Graph) is pruned, and some irrelevant edges are deleted.

6) Feature Aggregation: Update features for each macro node. The whole clustering process is performed repeatedly until all edges in the keypoint graph model are deleted, or all keypoints are successfully clustered into several clusters.

An embodiment of the present invention provides a method for detecting key points. Taking Online Hierarchical Graph Clustering (OHGC) as an example, the input is: an RGB image of a multi-target object (the number of target objects is assumed to be P); Output: P target object keypoint clusters (all keypoints of a target object form a cluster); the method includes: Step S1, extract key point information in the image to be detected; Step S2, construct a heat map of key points of multi-target objects; Step S3, perform feature learning based on the correlation of GNN; Step S4, repeat the operation several times until the There are no edges to be pruned in the keypoint graph model.

In some embodiments of the present invention, the step S4 is repeated several times until there are no edges to be pruned in the key point graph model, including: step S41 , using a pooling layer (avg-pooling) to perform key point feature fusion; step S42, updating the similarity matrix between key points; step S43, clustering the key points; here, the clustering may be used to realize the merging of key points; The keypoint graph model is pruned. According to the structural constraints of the target object, the unreasonable edges in the current keypoint graph model are deleted. For example, a target object has only one head vertex.

FIG. 2B is a schematic diagram of a network structure for implementing a key point detection method according to an embodiment of the present invention. As shown in FIG. 2B , the network structure includes: a GNN module 21 , an edge discriminator 22 (Edge Discriminator) and a giant Set node discriminator 23 (Macro-node Discriminator), wherein: GNN module 21, which is formed by stacking edge convolution EdgeConv layer and multi-layer neural network (Multi-Layer Perceptron, MLP). Among them, the EdgeConv layer is a differentiable neural network module that can be embedded in the existing network architecture, and contains local domain information; by stacking EdgeConv modules or loops, it can extract global shape information, etc. advantage. The edge discriminator 22 is configured to input features of a pair of key points to determine whether the two key points belong to the same target object. The macro node discriminator 23 is configured to judge whether the key points inside a macro node completely belong to the same target object.

The above method can not only be used to accurately predict the position of the key points of the target object in the Internet video, but also can be used to analyze the behavior types of the target object; and can also be used to add real-time special effects to different parts of the target object. In the above method, based on the hierarchical graph clustering category module, an online hierarchical graph clustering algorithm is realized, and the structure prior information of the target object and the global information of the target object are retained. On the one hand, through end-to-end training, more attention is paid to the errors in the clustering results, and feature learning can be performed more effectively, which can directly optimize the clustering results and improve the clustering accuracy; on the other hand, the prior information of the target object structure can be used. and global feature information, which improves the clustering accuracy of key points.

Those skilled in the art can understand that in the above method of the embodiments, the writing order of each step does not mean a strict execution order but constitutes any limitation to the implementation process, and the execution order of each step should be based on its function and possible internal logic. Sure.

Based on the same concept, an embodiment of the present invention also provides a key point detection device. Referring to FIG. 3, it is a schematic diagram of the architecture of the key point detection device provided by the embodiment of the present invention, including an acquisition module 301, a first generation module 302, the second generation module 303, the processing module 304, the determination module 305, the training module 306, the behavior type determination module 307, and the special effect generation module 308, wherein: an acquisition module 301, configured to acquire an image to be detected; The first generation module 302 is configured to generate an image feature map and a plurality of key point heat maps based on the image to be detected; the image feature map is used to represent the characteristics of each target object in the image to be detected. The relative positional relationship between key points; each of the key point heatmaps includes a type of key points of the image to be detected, and key points of different types correspond to different parts of the target object; The second generation module 303 is configured to generate an initial key point graph model based on the image feature map and a plurality of the key point heatmaps; the initial key point graph model includes different images in the image to be detected Information of key points of the category and information of connecting edges, each connecting edge is an edge between two key points of different categories; The processing module 304 is configured to perform multiple times of pruning processing of the connecting edges on the initial key point graph model, until multiple key points in the processed key point graph model are clustered into multiple clusters, obtaining The key point information belonging to each target object respectively.

In a possible implementation manner, the information of the key points includes location information, category information, and primitive feature information; the second generation module 303 is configured to determine each of the initial key point graph models according to the following steps: Information of key points: based on the heat map of the key points, determine the position information of each key point; based on the position information of each of the key points, extract the feature information of the primitives of the key points from the image feature map , and based on the category label of the key point heat map to which the key point belongs, the category information corresponding to the key point is determined.

In a possible implementation manner, the apparatus further includes: a determination module 305 configured to, for each of the key points in the initial key point graph model, based on the information of the key points and the key point graph There is information about other key points that connect edges between the model and the key points, and the fusion feature of the key points is determined; the processing module 304 performs the connection for the initial key point graph model multiple times. In the case of edge pruning processing, the configuration is: based on the fusion feature of each of the key points included in the initial key point graph model, the initial key point graph model is pruned multiple times for the connected edges. Branch processing.

In a possible implementation manner, the processing module 304 performs multiple times of pruning the connecting edges on the initial keypoint graph model until multiple keypoints in the processed keypoint graph model are In the case of clustering into multiple clusters, the configuration is as follows: perform the first processing process for the current key point graph model: based on the fusion feature of the two key points corresponding to each connecting edge in the current key point graph model, The adjacent key points in the multiple key points included in the current key point graph model perform key point clustering of the same target object to obtain at least one macro node; wherein, the macro node includes a plurality of clustered nodes. Adjacent key points; and based on the fusion features of the key points included in each of the macro nodes, determine the fusion features of the macro nodes; based on the obtained at least one of the macro nodes and the current key point graph model, perform the current pruning process of the connection edges on the current key point graph model, and obtain the key point graph model after the current pruning process; after executing the current first processing process, The key point graph model after the current pruning process is used as the current key point graph model, and the currently determined macro node and the fusion feature of the macro node are used as the key point in the current key graph model and The fusion feature of the key points is performed, and the first processing process is performed again until the multiple key points in the processed key point graph model are clustered into multiple clusters.

In a possible implementation manner, the processing module 304, based on the fusion feature of two key points corresponding to each connecting edge in the current key point graph model, performs a multiplication process on the current key point graph model. In the case of obtaining at least one macro node by performing key point clustering of the same target object on adjacent key points among the key points, the configuration is: based on the fusion features of the two key points corresponding to the connection edges, determine the connection The weight of the edge, the weight represents the probability that the two key points corresponding to the connecting edge belong to the same target object; based on the weight of each connecting edge included in the current key point graph model, the current key point graph The adjacent key points among the multiple key points included in the model perform key point clustering of the same target object to obtain at least one macro node.

In a possible implementation manner, the processing module 304, based on the obtained at least one of the macro node and the current key point graph model, performs the current time on the connection edge on the current key point graph model. In the case of obtaining the key point graph model after the current pruning process, the configuration is: based on the obtained at least one of the macro node and the current key point graph model, determine the connection edge to be deleted, And from the current key point graph model, the connection edges to be deleted are deleted; at least one of the macro nodes and the key points included in the macro node are removed from the current key point graph model. The other key points are used as the key points after the pruning process, and the remaining connection edges after deletion are used as the connection edges after the pruning process, and the key point graph model after the current pruning process is obtained.

In a possible implementation manner, the processing module 304 is configured to: based on the obtained at least one macro node and the current key point graph model to determine the connection edge to be deleted: based on at least one macro Category information of each key point included in the node, and category information of other key points in the current key point graph model except the key point included in at least one macro node, determine the connection edge to be deleted .

In a possible implementation manner, the processing module 304 is based on the category information of each key point included in the at least one macro node and the current key point graph model except that included in the at least one macro node. The category information of other key points other than the key point, in the case of determining the connection edge to be deleted, the configuration is: for any connection edge in the current key point graph model, in the corresponding connection edge of any connection edge. When two key points are key points in different macro nodes, and there are key points with the same category information in the two macro nodes corresponding to any connection edge, determine that any connection edge is the to-be-deleted Subtract the connecting edge; in the case that the two key points corresponding to any connecting edge are key points in the same macro node, determine that any connecting edge is the connecting edge to be deleted; One of the corresponding two key points is the key point in the macro node, the other key point is not the key point in the macro node, and the macro node corresponding to any connecting edge has another key point in the macro node. In the case of the key points with the same type information of the key points, it is determined that any connecting edge is the connecting edge to be deleted.

In a possible implementation manner, the key point information of each target object is generated by a pre-trained target neural network; wherein, the target neural network is a neural network to be trained including a macro node discriminator. After training, the macro node discriminator is used to discriminate whether multiple key points included in each macro node belong to the same target object.

In a possible implementation manner, the device further includes: a training module 306, configured to train the neural network to be trained through the following steps to obtain a pre-trained target neural network: obtaining sample images; Based on the sample images, the to-be-trained neural network including the macro node discriminator is trained to obtain a pre-trained target neural network.

In a possible implementation manner, the training module 306, based on the sample image, trains the neural network to be trained including the macro node discriminator, and obtains the pre-trained target neural network. In this case, the configuration is: based on the sample image, the neural network to be trained is trained to obtain a prediction result, and the prediction result includes the detection result of the macro node discriminator and the prediction of each key point. category, and the predicted position information of each key point; based on the detection result of the macro node discriminator, determine the first loss value; and based on the predicted category of each key point, the prediction of each key point The location information, and the labeling result carried in the sample image, determine the second loss value; wherein, the labeling result includes the labeling category that each key point belongs to the corresponding target object, and the labeling location information of each key point; Based on the first loss value and the second loss value, the neural network to be trained is trained to obtain a pre-trained target neural network.

In a possible implementation manner, after obtaining the key point information of each target object in the to-be-detected image, the method further includes: a behavior type determination module 307, configured to be based on the key point corresponding to each target object. information to determine the type of behavior of the target object.

In a possible implementation manner, after obtaining the key point information of each target object in the to-be-detected image, it also includes: a special effect generation module 308 configured to be based on the key point information corresponding to each target object. , determining the position information of at least one target part of the target object, and generating special effect information for the at least one target part according to the position information of the at least one target part.

In some embodiments, the functions or templates included in the apparatuses provided in the embodiments of the present invention may be used to execute the methods described in the above method embodiments. For the implementation process, reference may be made to the descriptions in the above method embodiments. I won't go into details here.

Based on the same technical concept, an embodiment of the present invention also provides an electronic device. Referring to FIG. 4 , it is a schematic structural diagram of an electronic device provided by an embodiment of the present invention, including a processor 401 , a memory 402 , and a bus bar 403 . The memory 402 is configured to store execution instructions, including an internal memory 4021 and an external memory 4022; the internal memory 4021 here is also called an internal memory, and is configured to temporarily store the operation data in the processor 401 and the data associated with the hard disk. Waiting for the data exchanged by the external memory 4022, the processor 401 exchanges data with the external memory 4022 through the internal memory 4021. When the electronic device 400 is running, the processor 401 and the memory 402 communicate through the bus bar 403. The processor 401 is caused to execute the following instructions: acquiring an image to be detected; generating an image feature map and a plurality of key point heat maps based on the to-be-detected image; the image feature map is used to represent the to-be-detected image The relative positional relationship between each target object in the image; each of the key point heatmaps contains a type of key points of the to-be-detected image, and key points of different categories correspond to different parts of the target object; based on The image feature map and a plurality of the key point heat maps generate an initial key point graph model; the initial key point graph model includes the information of different types of key points in the to-be-detected image and the connection edges. information, each connecting edge is an edge between two different types of keypoints; the initial keypoint graph model is subjected to multiple pruning processing of the connecting edges, until more than one in the processed keypoint graph model is processed. The key points are clustered into multiple clusters, and the key point information belonging to each target object is obtained. In addition, an embodiment of the present invention further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the key point detection method described in the above method embodiments are executed. .

The computer program product of the key point detection method provided by the embodiment of the present invention includes a computer-readable storage medium storing a program code, and the instructions included in the program code can be used to execute the key point detection method described in the above method embodiments. For the steps, reference may be made to the above method embodiments, which will not be repeated here.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, and details are not repeated here. In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods may be implemented in other manners. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or elements may be combined or may be Integration into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, and may be in electrical, mechanical or other forms.

The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or may be distributed to multiple network units . Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment. In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including several The instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, removable hard disk, Read-Only Memory (ROM), Random Access Memory (RAM), disk or CD, etc. that can store program codes medium. The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical scope disclosed by the present invention can easily think of changes or replacements, which should be covered in the within the protection scope of the present invention. Therefore, the protection scope of the present invention should be subject to the protection scope of the patent application.

Industrial Applicability Based on the generated image feature map and a plurality of key point heat maps, the present invention generates an initial key point map model corresponding to the image to be detected. Since the initial key point map model includes the information in the image feature map and the key point heat map , and the image feature map can represent the relative positional relationship between different target objects in the image to be detected, so that the initial key point graph model can be pruned at the connected edges, and the key point information of each target object can be obtained. , to more accurately distinguish the key points of different target objects, so as to improve the accuracy of key point clustering.

21: Keypoint Heatmap 22: Image Feature Map 23: Initial keypoint graph model 24: Edge Discriminator 25: Macro Node 26: Processed keypoint graph model 27: Macro Node Discriminator 301: Get Mods 302: The first generation module 303: Second Generation Module 304: Processing module 305: Determine the module 306: Training Module 307: Behavior type determination module 308: Special effect generation module 400: Electronic Equipment 401: Processor 402: memory 4021: Internal memory 4022: External memory 403: Busbar S101~S104: Steps

1 is a schematic flowchart of a key point detection method provided by an embodiment of the present invention; 2A is a schematic flowchart of pruning processing in a key point detection method provided by an embodiment of the present invention; 2B is a schematic diagram of a network structure for implementing a key point detection method provided by an embodiment of the present invention; FIG. 3 is a schematic structural diagram of a key point detection apparatus provided by an embodiment of the present invention; FIG. 4 is a schematic structural diagram of an electronic device 400 according to an embodiment of the present invention.

S101~S104: Steps

Claims (15)

A keypoint detection method, including: Obtain the image to be detected; Based on the to-be-detected image, an image feature map and a plurality of key point heat maps are generated; the image feature map is used to represent the relative positional relationship between each target object in the to-be-detected image; each The keypoint heatmap includes keypoints of one category of the image to be detected, and keypoints of different categories correspond to different parts of the target object; Based on the image feature map and a plurality of the keypoint heatmaps, an initial keypoint graph model is generated; the initial keypoint graph model includes information of different types of keypoints in the to-be-detected image and connecting edges information, each connecting edge is an edge between two different types of keypoints; The initial key point graph model is subjected to multiple times of pruning of the connected edges, until multiple key points in the processed key point graph model are clustered into multiple clusters, and key points belonging to each target object are obtained. Click information. The method according to claim 1, wherein the information of the key points includes location information, category information, and primitive feature information; Determine the information of each keypoint in the initial keypoint graph model according to the following steps: Determine the location information of each key point based on the heat map of the key points; Based on the position information of each of the key points, the feature information of the key points is extracted from the image feature map, and the key points are determined based on the category labels of the key point heat map to which the key points belong. corresponding category information. The method according to claim 1, wherein the method further comprises: For each of the key points in the initial key point graph model, based on the information of the key point and information of other key points in the key point graph model that have connecting edges with the key point, determine fusion features of the key points; The performing multiple times of pruning the connected edges on the initial key point graph model, including: Based on the fusion feature of each key point included in the initial key point graph model, the initial key point graph model is subjected to multiple times of pruning processing of the connecting edges. The method according to any one of claims 1 to 3, wherein the initial keypoint graph model is subjected to multiple times of pruning of the connected edges until a plurality of times in the processed keypoint graph model are performed. Keypoints are clustered into multiple clusters, including: Execute the first processing procedure for the current keypoint graph model: Based on the fusion feature of the two key points corresponding to each connecting edge in the current key point graph model, the key points of the same target object are performed on adjacent key points among the multiple key points included in the current key point graph model. Clustering to obtain at least one macro node; wherein, the macro nodes include multiple adjacent key points after the clustering; and based on the fusion features of the key points included in each of the macro nodes, determine the Describe the fusion characteristics of macro nodes; Based on the obtained at least one of the macro node and the current key point graph model, the current key point graph model is subjected to the current pruning process of the connecting edges, and the key points after the current pruning process are obtained. graph model; After the current first processing process is performed, the key point graph model after the current pruning process is used as the current key point graph model, and the currently determined macro node and the macro node The fusion feature is used as the key point in the current key point graph model and the fusion feature of the key point, and the first processing process is performed again until the multiple key points in the processed key point graph model are clustered into multiple cluster. The method according to claim 4, wherein based on the fusion feature of two key points corresponding to each connecting edge in the current key point graph model, the multiple key points included in the current key point graph model Perform key point clustering of the same target object on the adjacent key points in the points to obtain at least one macro node, including: Determine the weight of the connection edge based on the fusion feature of the two key points corresponding to the connection edge, and the weight represents the probability that the two key points corresponding to the connection edge belong to the same target object; Based on the weight of each connecting edge included in the current keypoint graph model, perform keypoint clustering of the same target object on adjacent keypoints among the multiple keypoints included in the current keypoint graph model, to obtain at least one macro node. The method according to claim 4, wherein based on the obtained at least one macro node and the current key point graph model, the current key point graph model is pruned for the current connection edge process, and get the key point graph model after the current pruning process, including: Based on the obtained at least one of the macro node and the current key point graph model, determine the connection edge to be deleted, and delete the connection edge to be deleted from the current key point graph model; Using at least one described macro node and other key points other than the key points included in the described macro node in the current key point graph model as the key points after the pruning process, the remaining connections after deletion will be The edge is used as the connecting edge after the pruning process, and the key point graph model after the current pruning process is obtained. The method according to claim 6, wherein determining the connection edge to be deleted based on the obtained at least one macro node and the current key point graph model, comprising: The determination of the Connecting edges to be deleted. The method according to claim 7, wherein based on the category information of each key point included in at least one macro node and the key points included in the current key point graph model except at least one macro node The category information of other key points to determine the connecting edges to be deleted, including: For any connection edge in the current key point graph model, the two key points corresponding to the any connection edge are key points in different macro nodes, and the two macro nodes corresponding to any connection edge In the case where there are key points with the same category information, determine that any connecting edge is the connecting edge to be deleted; In the case that the two key points corresponding to the any connecting edge are the key points in the same macro node, determine that the any connecting edge is the connecting edge to be deleted; Among the two key points corresponding to any connecting edge, one key point is a key point in the macro node, the other key point is not a key point in the macro node, and the macro corresponding to any connecting edge If there is a key point in the node with the same type information as another key point, it is determined that any connecting edge is the connecting edge to be deleted. The method according to any one of claims 1 to 3, wherein the key point information of each target object is generated by a pre-trained target neural network; The node discriminator is obtained by training the neural network to be trained, and the macro node discriminator is used to discriminate whether multiple key points included in each macro node belong to the same target object. The method according to claim 9, wherein the neural network to be trained is trained through the following steps to obtain a pre-trained target neural network: get a sample image; Based on the sample images, the to-be-trained neural network including the macro node discriminator is trained to obtain a pre-trained target neural network. The method according to claim 10, wherein, based on the sample image, a neural network to be trained including the macro node discriminator is trained to obtain a pre-trained target neural network, including: Based on the sample image, the neural network to be trained is trained to obtain a prediction result, where the prediction result includes the detection result of the macro node discriminator, the prediction category of each key point, and each key point the predicted location information of the point; determining a first loss value based on the detection result of the macro node discriminator; and based on the predicted category of each key point, the predicted location information of each key point, and the information carried in the sample image The labeling result is to determine the second loss value; wherein, the labeling result includes the labeling category that each key point belongs to the corresponding target object, and the labeling position information of each key point; Based on the first loss value and the second loss value, the neural network to be trained is trained to obtain a pre-trained target neural network. The method according to any one of claim 1 to 3, wherein after obtaining the key point information of each target object in the to-be-detected image, the method further includes: Based on the key point information corresponding to each target object, the behavior type of the target object is determined. The method according to any one of claim 1 to 3, wherein after obtaining the key point information of each target object in the to-be-detected image, the method further includes: Based on the key point information corresponding to each target object, position information of at least one target part of the target object is determined, and special effect information for the at least one target part is generated according to the position information of the at least one target part. An electronic device includes: a processor, a memory, and a bus bar, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, there is a connection between the processor and the memory. Through bus communication, the machine-readable instructions, when executed by the processor, perform the steps of the keypoint detection method according to any one of claims 1 to 13. A computer-readable storage medium, on which a computer program is stored, the computer program executes the steps of the key point detection method according to any one of claim 1 to 13 when the computer program is run by a processor.

Images