JP2022542199A - KEYPOINT DETECTION METHOD, APPARATUS, ELECTRONICS AND STORAGE MEDIA - Google Patents

Abstract

The present invention provides a keypoint detection method, apparatus, electronic device and storage medium, the method comprising: obtaining a detected image; based on the detected image, an image feature map; wherein said image feature map is used to represent the relative positional relationship between each target object in said detected image, each said keypoint heat map comprising said based on the image feature map and a plurality of the keypoint heatmaps, comprising keypoints of one category of a detected image, wherein keypoints of different categories correspond to different parts of the target object; generating a keypoint map model, wherein the keypoint map model includes information of keypoints of different categories in the detected image and information of connecting edges, each connecting edge consisting of two being edges between keypoints of different categories; and determining each keypoint belonging to the same target object based on the keypoint map model. [Selection drawing] Fig. 1

Description

This application is filed based on a Chinese patent application numbered 202010622132.3 filed with the Chinese Patent Office on June 30, 2020, and claims priority to the Chinese patent application; The entire content of the Chinese patent application is incorporated herein by reference.

TECHNICAL FIELD The present invention relates to the field of image detection technology, and more particularly to a keypoint detection method, device, electronic device and storage medium.

Human body keypoint detection provides support for higher-layer information for analyzing human behavior in video, and is the basis for realizing video body motion recognition and human-computer interaction. . In recent years, the detection method of human body keypoints based on deep neural network has become a research hotspot because it is easy to acquire into the input image and the detection effect is efficient and accurate.

Usually, in human body keypoint detection methods based on deep neural networks, keypoint detection and keypoint clustering are two independent steps, i.e. keypoint clustering is usually a non-differentiable post-processing It is action. However, in such a method, the accuracy of the keypoint clustering process is low, i.e., clustering errors may occur, and different people's keypoints are clustered as the same person's keypoints, so that the detection result an error occurs.

In view of this, embodiments of the present invention provide at least a keypoint detection method, apparatus, electronic device and storage medium.

In a first aspect, embodiments of the present invention provide a keypoint detection method, the method comprising:
obtaining an image to be detected;
generating an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map represents the relative positional relationship between each target object in the detected image. wherein each said keypoint heatmap contains keypoints of one category of said detected image, with keypoints of different categories corresponding to different parts of said target object;
generating a keypoint map model based on the image feature map and a plurality of the keypoint heatmaps, wherein the keypoint map model contains information of different categories of keypoints in the detected image; and connecting edge information, each connecting edge being an edge between two different categories of keypoints;
determining each keypoint belonging to the same target object based on the keypoint map model.

By adopting the above method, a keypoint map model corresponding to the detected image can be generated based on the generated image feature map and multiple keypoint heatmaps, and the keypoint map model is contains the information in the image feature map and the keypoint heat map, and the image feature map can represent the relative positional relationship between different target objects in the detected image, so it is based on the keypoint map model. to correctly distinguish keypoints of different target objects to improve the accuracy of keypoint clustering.

In one possible embodiment, determining each keypoint belonging to the same target object based on said keypoint map model comprises:
determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edge;
determining each keypoint belonging to the same target object based on the determined correlation.

In one possible embodiment, determining each keypoint belonging to the same target subject based on the determined correlation is performed by identifying each keypoint for which the corresponding correlation is greater than a set threshold as a key point of the same target subject. Including using it as a point.

In one possible embodiment, determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edges. determining, for each keypoint, a fusion feature of the keypoint based on information of the keypoint and information of other keypoints having a connection relationship with the keypoint in the keypoint map model; and determining the correlation between two keypoints with a connection relationship in the keypoint map model based on the fusion features respectively corresponding to each keypoint.

In one possible embodiment, the keypoint information includes location information, category information, and pixel feature information, and based on the keypoint heatmap, determining location information for each keypoint; Extract pixel feature information of the keypoint from the image feature map based on the position information of the point, and determine category information corresponding to the keypoint based on the category label of the keypoint heatmap to which the keypoint belongs. Determine the information of each keypoint in the keypoint map model according to what is done.

In one possible embodiment, based on the category information corresponding to each keypoint, each keypoint is connected with other keypoints that are different from the category to which the keypoint belongs, in the keypoint map model. Generate each connecting edge in the keypoint map model according to forming connecting edges.

In one possible embodiment, based on the category information corresponding to each keypoint and the matching relationship between different categories of presets, each keypoint is identified by a key corresponding to a target category that matches the category to which the keypoint belongs. generating each connecting edge in said keypoint map model according to connecting points to form connecting edges in said keypoint map model. In one possible embodiment, based on said keypoint map model: After determining each keypoint belonging to the same target object, further comprising determining a behavior type of the target object based on the information of each keypoint corresponding to each target object.

In one possible embodiment, after determining each keypoint belonging to the same target object based on the keypoint map model, based on the information of each keypoint corresponding to each target object, a special effect for the target object Further including generating the information.

The description of the effects of the following devices, electronic devices, etc. can refer to the description of the above method, and will not be repeated here.

In a second aspect, an embodiment of the present invention provides an apparatus for detecting keypoints, said apparatus comprising:
an acquisition module configured to acquire the detected image;
A first generating module configured to generate an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map is for each target of the detected image. used to represent the relative positional relationship between objects, each said keypoint heatmap contains keypoints of one category of said detected image, and keypoints of different categories correspond to different positions of said target object; a first generation module corresponding to the part;
A second generation module configured to generate a keypoint map model based on the image feature map and a plurality of the keypoint heat maps, wherein the keypoint map model includes: a second generating module comprising information of keypoints of different categories and information of connecting edges of
a determination module configured to determine each keypoint belonging to the same target object based on the keypoint map model.

In a third aspect, an embodiment of the present invention provides an electronic device comprising a processor, a memory and a bus, the memory storing machine-readable instructions executable by the processor, when the electronic device executes and said processor and said memory are in communication via a bus, and said machine-readable instructions, when executed by said processor, perform a keypoint detection method according to the first aspect or any embodiment above. Execute the step.

In a fourth aspect, an embodiment of the invention provides a computer readable storage medium storing a computer program, said computer program, when executed by a processor, as described in the first aspect or any embodiment above. , perform the keypoint detection method steps.

An embodiment of the invention provides a computer program product comprising computer readable code, the computer readable code, when executed in an electronic device, causing a processor in the electronic device to perform one or more of the above implementations. Run the server to implement the example and perform the above method.

In order to make the above objects, features and advantages of the present invention clearer and easier to understand, the following detailed description will be made of the preferred embodiments together with the accompanying drawings.

To describe the technical solutions in the embodiments of the present invention more clearly, the following briefly introduces the drawings required in the embodiments. The drawings herein are incorporated in and constitute a part of this specification, and these drawings illustrate embodiments consistent with the present invention, and together with the description, the technical solution of the embodiments of the present invention. used to describe policies. The following drawings show only some embodiments of the present invention and should not be considered as limiting the scope, and those skilled in the art can follow these drawings to construct other relevant drawings without creative work. It should be understood that it is also possible to obtain
4 is an exemplary flowchart of a keypoint detection method according to an embodiment of the present invention; 1 is a schematic diagram of a keypoint map model according to an embodiment of the invention; FIG. 1 is a block diagram of a system of keypoint detection method according to an embodiment of the present invention; FIG. FIG. 4 is an application scenario diagram of a keypoint detection method according to an embodiment of the present invention; 1 is a schematic diagram of the architecture of a keypoint detection device according to an embodiment of the present invention; FIG. 4 is an exemplary structural diagram of an electronic device 400 according to an embodiment of the present invention; FIG.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the following will refer to the drawings of the embodiments of the present invention to clarify the technical solutions in the embodiments of the present invention, And fully described, apparently, the described embodiments are only some, but not all of the embodiments of the present invention. Generally, the components of the embodiments of the invention described and illustrated in the drawings herein can be arranged and designed in a variety of different configurations. Therefore, the detailed description of embodiments of the invention provided in the drawings, below, are not intended to limit the scope of the claimed invention, but rather to display selected embodiments of the invention. Not too much. All other embodiments obtained by persons skilled in the art without creative efforts based on the embodiments of the present invention shall all fall within the protection scope of the present invention.

Normally, deep neural network can be used to detect an image and determine the keypoint information of the target object contained in the image, where the keypoint detection method based on deep neural network is to detect the keypoint and clustering of keypoints.

Currently, keypoint detection and keypoint clustering are two independent steps: keypoint clustering is a post-processing operation that is usually not differentiable. However, when using such a scheme, the accuracy of the keypoint clustering process is low, i.e., clustering errors may be generated, and different people's keypoints are clustered as the same person's keypoints, As a result, an error occurs in the detection result.

To solve the above problems, embodiments of the present invention provide a keypoint detection method.

To facilitate understanding of the embodiments of the present invention, firstly, the keypoint detection methods disclosed in the embodiments of the present invention are introduced in detail.

The execution entity of the keypoint detection method according to the embodiments of the present invention may be a server, which may be a local server or a cloud server. Alternatively, the execution subject of the method may be a terminal device such as a mobile phone, tablet, or AR glasses.

Please refer to FIG. 1, which is an exemplary flow chart of a keypoint detection method according to an embodiment of the present invention, which includes S101-S104.

In S101, a detected image is obtained.

At S102, an image feature map and a plurality of keypoint heat maps are generated based on the detected image, and the image feature map is used to represent the relative positional relationship between each target object in the detected image. , each keypoint heatmap contains keypoints of one category of the detected image, and keypoints of different categories correspond to different parts of the target object.

At S103, generate a keypoint map model based on the image feature map and the plurality of keypoint heatmaps, the keypoint map model includes information of keypoints of different categories and information of connecting edges in the detected image; , where each connecting edge is an edge between two different categories of keypoints.

At S104, each keypoint belonging to the same target object is determined based on the keypoint map model.

In the above method, a keypoint map model corresponding to the detected image can be generated based on the generated image feature map and a plurality of keypoint heat maps, wherein the keypoint map model includes image features Based on the keypoint map model, different target To improve accuracy of keypoint clustering by correctly distinguishing target keypoints.

Regarding S101 and S102,
The detected image can be an image containing any one target object. In the implementation process, the detected images can be obtained from the connected storage device, and the detected images collected in real time from the connected imaging device can also be obtained.

In the implementation process, the acquired detected image is input to a trained keypoint detection neural network to generate an image feature map and multiple keypoint heatmaps, and the image feature map, multiple keypoint heatmaps Based on the map and a trained keypoint detection neural network, each keypoint for each target object can be determined.

Here, each keypoint heatmap contains keypoints of one category of the detected image, and keypoints of different categories correspond to different parts of the target object. For example, the keypoint categories could be head, neck, hands, etc., and a keypoint heatmap could be an image containing keypoints of the head, or a keypoint heatmap could contain keypoints of the neck. or the category of the keypoint may be a set first category, second category, etc. Here, the first category keypoint can be the thumb keypoint, the second category keypoint can be the forefinger keypoint, and the keypoint heatmap is an image containing the first category keypoint. Alternatively, the keypoint heatmap may be an image containing the second category keypoints, or the like. Here, the categories of key points and the number of categories can be set according to actual needs. And the number of keypoints corresponding to each target object can be set according to actual needs, for example, the number of keypoints corresponding to each target object can be 17, 105, and so on.

Here, the number of keypoint heatmaps matches the number of set keypoint categories. For example, when the number of set keypoint categories is 17, the key generated based on the detected image The number of point heat maps is also 17. Here, the number of keypoints in each category can be one or more.

The number of image feature maps may be one or plural. Here, when the number of image feature maps is one, the image feature map represents the relative positional relationship between the parts corresponding to the key points of each category of each target object in the detected image. can be done. If the number of image feature maps is multiple, the number of image feature maps and the number of keypoint heatmaps can be the same, i.e. each image feature map corresponds to one of each target object in the detected image. It can represent the relative positional relationship between parts corresponding to two categories of keypoints. Here, the size of the image feature map and the size of the keypoint heatmap match.

In the implementation process, an image feature map and multiple keypoint heatmaps can be obtained by setting different loss functions in the keypoint detection neural network.

Regarding S103,
Here, we extract the information of each keypoint from multiple keypoint heatmaps and image feature maps, let each keypoint containing information be a node, and the edges between keypoints of different categories as connecting edges, and extract the keypoint map Models can be constructed.

In an alternative embodiment, keypoint information may include location information, category information, and pixel feature information. where, based on the keypoint heat map, determine the location information of each keypoint; based on the location information of each keypoint, extract the pixel feature information of the keypoint from the image feature map, and determine where the keypoint belongs Based on the category labels of the keypoint heatmap, the information of each keypoint of the keypoint map model can be determined according to determining the category information corresponding to the keypoint.

In the implementation process, location information for each keypoint can be determined based on the pixel value of each pixel in the keypoint heatmap. Exemplarily, for each keypoint heatmap, a pixel with the maximum pixel value is selected and determined as one keypoint, and the position information of the selected pixel is used as the position information of the keypoint. can decide. Here, if the pixel value of a particular pixel in the keypoint heatmap is greater than the pixel values of the surrounding pixels, the pixel value of that pixel is considered to be the maximum value and that pixel is a keypoint.

After obtaining the position information of each pixel, the pixel value of the pixel corresponding to the position information can be extracted from the image feature map, and the extracted pixel value can be determined as the pixel feature information of the key point.

At the same time, the category information corresponding to the keypoint can also be determined according to the category label of the keypoint heatmap to which each keypoint belongs. For example, if the category label of keypoint heatmap A is head, the category information of each keypoint included in keypoint heatmap A is the keypoint of head, and the category label of keypoint heatmap B is If it is the neck, the category information of each keypoint included in the keypoint heatmap B is the neck keypoint.

In an alternative embodiment, each connecting edge in the keypoint map model can be generated according to the following two schemes.

In method 1, based on the category information corresponding to each keypoint, each keypoint is connected with other keypoints different from the category to which the keypoint belongs to form connecting edges in the keypoint map model.

In method 2, each keypoint is connected with the keypoint corresponding to the target category matching the category to which the keypoint belongs, based on the category information corresponding to each keypoint and the matching relationship between different categories of presets. , form the connecting edges in the keypoint map model.

In method 1, based on the category information corresponding to each keypoint, keypoints with the same category information are not connected, keypoints with different category information are connected, and connecting edges in the keypoint map model can be formed.

Here, based on the category information corresponding to each keypoint, by connecting each keypoint with other keypoints that are different from the category to which the keypoint belongs to form connecting edges in the keypoint map model. , to obtain the keypoint map model and provide data support for determining each keypoint for each subsequent target object.

For method 2, the matching relationship between different categories can be preset in advance based on the human body structure, for example, the preset matching relationship between different categories is to match the head category and the neck category, Matching points with left shoulder, right shoulder, and head categories, respectively, foot and knee categories, and so on. Here, the matching relationship between different categories can be set according to actual needs.

Further, for each keypoint, determine a target category that matches the keypoint's category, and connect the keypoint with the keypoint corresponding to the target category to form a connecting edge in the keypoint map model. be able to.

In the above embodiments, the matching relationship between different categories can be preset, for example, the head keypoints and neck keypoints are preset to match, and the foot keypoints and knee keypoints are preset to match. such that each keypoint is connected with the keypoint corresponding to the target category that matches the category to which the keypoint belongs, and each keypoint is not connected with keypoints that do not match, and the correlation It is possible to reduce the amount of computation for calculating properties and further improve the efficiency of keypoint detection.

Referring to the schematic diagram of the keypoint map model shown in FIG. is included. Here, the drawing includes first category keypoints 201, second category keypoints 202, and third category keypoints 203, and further includes connecting edges 204 between different keypoints. As can be seen from the drawing, the keypoint map model 21 contains the connecting edges between each keypoint of different categories, and the keypoint map model 22 contains the connecting edges between different keypoints with matching relations. As such, the matching relationship between different categories of presets is that the first category matches the second category, and the second category matches the third category.

Regarding S104,
Now, based on the keypoint map model, each keypoint included in the keypoint map model can be split to split multiple keypoints belonging to the same target object, thereby dividing the keypoints included in the detected image. Multiple keypoints can be obtained corresponding to each target object that is displayed. Here, the number of keypoints corresponding to each target object is the same.

In an alternative embodiment, determining each keypoint belonging to the same target object based on the keypoint map model may include the following steps.

At A1, based on the information of each keypoint in the keypoint map model and the information of the connecting edge, the correlation between two keypoints having a connection relationship in the keypoint map model is determined.

At A2, each keypoint belonging to the same target object is determined based on the determined correlation.

In the above embodiment, based on the information of each keypoint and the information of connecting edges, the correlation between two keypoints having a connection relationship in the keypoint map model can be determined, and the correlation is the corresponding can represent the probability that two keypoints belonging to the same target object belong to the same target object. can be realized to obtain

In step A1, for each connecting edge of the keypoint map model, the correlation between two keypoints corresponding to the connecting edge can be determined, i.e. obtaining the correlation corresponding to each connecting edge be able to.

As an alternative embodiment, determining the correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint and the information of the connecting edges of the keypoint map model is as follows: Including steps.

1. For each keypoint, determine the fusion features of the keypoint based on the information of the keypoint and the information of other keypoints that have a connection relationship with the keypoint in the keypoint map model.

2. Determine the correlation between two keypoints with connection relation in the keypoint map model based on the fusion features respectively corresponding to each keypoint.

When determining the fusion feature of each keypoint, determining the fusion feature of the keypoint based on the information of the keypoint and the information of other keypoints in the keypoint map model having a connection relationship with the keypoint. , where the other keypoints can be keypoints in the keypoint map model with connecting edges between them.

Here, for each keypoint in the keypoint map model, determine the corresponding fusion feature, and based on the fusion feature respectively corresponding to each keypoint, between the two keypoints corresponding to each connecting edge in the keypoint map model can be determined.

In the implementation process, if the fusion feature can be a feature vector corresponding to each keypoint, the similarity between the fusion features (feature vectors) of the two keypoints corresponding to each connecting edge is calculated and obtained by computing Correlation between the two keypoints can be determined through similarity.

In the above embodiment, for each keypoint, the fusion feature of the keypoint is determined based on the information of the keypoint and the information of other keypoints having a connection relationship with the keypoint; Then, the fusion feature of the keypoint can not only represent the feature of the keypoint, but also can represent the relationship between the keypoint and other keypoints. Based on the fused features, the correlation between two keypoints having a connection relationship can be determined, and each keypoint corresponding to each target object can be more accurately determined based on the correlation.

In step A2, after determining to obtain the correlation of two keypoints corresponding to each connecting edge of the keypoint map model, based on each determined correlation, each keypoint is divided to obtain the same target Each keypoint belonging to the object can be determined.

As an alternative embodiment, determining each keypoint belonging to the same target subject based on the determined correlation uses each keypoint whose corresponding correlation is greater than a set threshold as the keypoint of the same target subject. can include doing

Based on the set correlation threshold and the target number of keypoints in each category in the preset, the implementation process uses each keypoint whose corresponding correlation is greater than the set threshold as the same targeted keypoint can do. Here, each target keypoint includes multiple different categories of keypoints, and the number of keypoints in each category matches the preset target number.

In an alternative embodiment, after each keypoint belonging to the same target subject is determined based on the keypoint map model, the behavior type of the target subject is determined based on the information of each keypoint corresponding to each target subject. can further include:

Here, after obtaining the information of each keypoint of each target object, the information of each keypoint of each target object can be input into a behavior detection neural network to determine the behavior type of the target object, such as , the action type may be running, walking, raising arms, and so on.

In an alternative embodiment, after each keypoint belonging to the same target object is determined based on the keypoint map model, special effect information for the target object is generated based on the information of each keypoint corresponding to each target object. may further include:

Here, for the information of each key point of the target target, the position of the target site of the target target is determined, and based on the special effect information corresponding to the preset target site, the position of the target site of the target target is corresponded. Special effect information can be generated. Here, the target site may be an arm, head, hand, or the like. For example, for each keypoint information of the target target, determine the arm position of the target target, and generate corresponding special effect information at the arm position of the target target based on the preset arm special effect information. be able to.

Here, for the information of each key point corresponding to each target object, the motion type of the target object is determined, and based on the mapping relationship between the preset motion type and the special effect information, the corresponding target object is It is also possible to generate special effect information to be used. For example, if it is determined based on the information of each keypoint corresponding to the target object A that the motion type of the target object is a heart-making motion, a heart special effect information can be generated for the target object. can.

In the related art, keypoint detection for a target object is realized through a bottom-up multiple target object keypoint detection method. Such a method first detects all possible target object keypoints in a photograph, and at the same time extracts the information of each keypoint, such as the embedding feature of the keypoint, and then applies the optimization equation These keypoints are clustered and assigned to different target subjects through solving for . As can be seen, the bottom-up method has better execution speed and is more robust to nearby targets or occlusion situations.

There are two methods of fitting features for extracting keypoints.

In Method 1, the implementation is based on Keypoint Embedding (KE).

Method 2 is implemented based on the key point detection method of Spatial Instance Embedding (SIE).

Here, in the training process of Method 1, since KE mainly contains the apparent information of pixels near keypoints, we zoom in on the inset value of each keypoint of the same target object, and keypoints of different target objects By zooming out on the inset values of , keypoint inset features can be extracted.

Now, in the training process of Method 2, since the SIE contains the target object center location information, we can extract keypoint inset features by returning each pixel value to a vector of the target object center.

As can be seen, KE is mainly concerned with pixel feature information, where said pixel feature information may include pixel apparent information such as pixel values of pixels, is not sensitive to spatial location information, and is long-range However, relying only on KE can lead to problems of grouping together keypoints of distant and different targets because there is no spatial limitation.

It can be seen that in the related art, in the bottom-up keypoint detection method, keypoint detection and keypoint clustering are two independent steps. Keypoint clustering is a post-processing operation that cannot be differentiated. There are some problems below. 1) The keypoint clustering process is a fixed post-processing and cannot learn optimization with data. 2) Because there is no co-optimization, the clustering accuracy of keypoints is not high and may produce clustering errors, for example, clustering keypoints of different target subjects as keypoints of the same target subject. 3) Post-clustering processing requires additional hyperparameter settings.

To solve the above problems, an embodiment of the present invention provides a keypoint detection method, the system block diagram of which is shown in 2B.

The related art includes two steps: keypoint detection S21 and keypoint clustering S22. The keypoint clustering step is a post-processing operation after keypoint detection.

An embodiment of the present invention provides a keypoint detection method and includes two sub-modules: a keypoint candidate extraction module G1 (Keypoint Candidate Proposal) and a graph clustering module G2 (Graph Grouping).

The keypoint candidate extraction module G1 first extracts, for each frame image, keypoint heatmaps (Heatmaps) of target keypoints, and image feature maps or It is configured to directly output Image Feature Maps for spatial instances. The maximum value (argmax) can be obtained through the keypoint heatmap to obtain the pixel coordinate position X of the keypoint. At the same time, the keypoint feature information F can be obtained from the corresponding position information of the image feature map.

The graph clustering module G2 is configured to determine whether two keypoints belong to the same target object and cluster the keypoints of the same target object.

By way of example, FIG. 2C shows the process of inputting a graph convolutional network with a keypoint heatmap P1 and an image feature map P2. First, keypoint information is extracted and a keypoint map model G={V, E} is constructed. The keypoint map model G is divided into two parts: keypoints V and edges E, where keypoints V are the information of each keypoint, i.e., "keypoint category T, keypoint Coordinate X, key point feature information F”. Edge E represents the relationship between keypoints, ie whether they belong to the same target object or not. After building the keypoint map model, we compute the keypoint correlations. We build a graph convolutional neural network model using edge convolution (EdgeConv), convolve the keypoint map model, and keep updating the feature information of the keypoints. Then, one edge classifier is trained to discriminate each pair of keypoints and determine whether the pair of keypoints belong to the same target object. End-to-end training optimization can be performed via a keypoint candidate extraction module G1, a graph clustering module G2, and an edge classifier.

In one possible embodiment, in one aspect of a keypoint detection method according to an embodiment of the present invention, location information of a target subject's keypoints can be predicted, while determining a behavior type of said target subject. On the other hand, real-time special effect information can be added to different parts of the target object.

Using the above method, the multi-target object pose can be estimated and transformed into a graph clustering problem, the topological structure information of the target object can be preserved, and the feature information of the multi-target object pose diagram can be extracted, Through a single deep neural network model, the clustering results of multi-target object poses can be obtained, and post-processing operations such as clustering are performed on the key points in the related art to simplify the operation steps. It can integrate keypoint detection and keypoint clustering to improve the accuracy of keypoint clustering.

Those skilled in the art will know that in the above method of the embodiment, the writing order of the steps is a strict execution order, not a limitation of the implementation process, and the execution order of each step should be based on its functions and possibilities. can understand something.

Based on the same concept, an embodiment of the present invention further provides a keypoint detection device, see FIG. comprising a module 301, a first generation module 302, a second generation module 303, a determination module 304, an action type determination module 305, and a special effect information generation module 306, wherein:
The acquisition module 301 is configured to acquire the detected image,
A first generation module 302 is configured to generate an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map is for each target object of the detected image. each keypoint heatmap contains keypoints of one category of the detected image, keypoints of different categories correspond to different parts of the target object corresponds to
A second generation module 303 is configured to generate a keypoint map model based on the image feature map and a plurality of the keypoint heat maps, wherein the keypoint map model includes a contains information of keypoints of different categories and information of connecting edges, each connecting edge being an edge between keypoints of two different categories;
A determination module 304 is configured to determine each keypoint belonging to the same target object based on said keypoint map model.

In one possible embodiment, when the determination module 304 determines each keypoint belonging to the same target object based on the keypoint map model,
determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint in the keypoint map model and the information of the connecting edge;
It is configured to determine each keypoint belonging to the same target object based on the determined correlation.

In one possible embodiment, when the determining module 304 determines each keypoint belonging to the same target subject based on the determined correlations,
Each keypoint for which the corresponding correlation is greater than a set threshold is configured to be used as a keypoint of the same target object.

In one possible embodiment, the determination module 304 determines the relationship between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edge. When determining correlation,
determining, for each keypoint, a fusion feature of the keypoint based on information about the keypoint and information about other keypoints in the keypoint map model that have a connection relationship with the keypoint;
It is configured to determine a correlation between two keypoints having a connection relationship in the keypoint map model based on the fused features respectively corresponding to each keypoint.

In one possible embodiment, the keypoint information includes location information, category information, and pixel feature information,
The second generation module 303 is
determining location information for each keypoint based on the keypoint heatmap;
extracting the pixel feature information of the keypoint from the image feature map based on the location information of each of the keypoints, and determining the category corresponding to the keypoint based on the category label of the keypoint heatmap to which the keypoint belongs; It is configured to determine information for each keypoint of said keypoint map model according to determining information.

In one possible embodiment, the second generation module 303 comprises:
according to the category information corresponding to each keypoint, connecting each keypoint with other keypoints different from the category to which the keypoint belongs to form a connecting edge in the keypoint map model; , to generate each connected edge in the keypoint map model.

In one possible embodiment, the second generation module 303 comprises:
connecting each keypoint with a keypoint corresponding to a target category that matches the category to which the keypoint belongs, based on the category information corresponding to each keypoint and the matching relationship between different categories of the preset; Each connecting edge in the keypoint map model is generated according to forming connecting edges in the keypoint map model.

In one possible embodiment, after determining each keypoint belonging to the same target object based on said keypoint map model, further:
A behavior type determination module 305 is provided, configured to determine a behavior type of the target object based on the information of each keypoint corresponding to each target object.

In one possible embodiment, after determining each keypoint belonging to the same target object based on said keypoint map model, further:
A special effect information generation module 306 is provided, configured to generate special effect information for each target object based on the information of each keypoint corresponding to each target object.

In some embodiments, the functions or templates provided by the apparatus according to embodiments of the present invention can be used to perform the methods described in the above method embodiments, and the implementation thereof The description in the example can be referred to and will not be described again here for the sake of brevity.

Based on the same technical concept, embodiments of the present invention further provide an electronic device. Referring to FIG. 4, which is an exemplary structural diagram of an electronic device provided by an embodiment of the present invention, it includes a processor 401 , a memory 402 and a bus 403 . Here, memory 402 is configured to store execution instructions and includes storage 4021 and external memory 4022, where storage 4021 may also be referred to as internal memory, operational data in processor 401, and storage such as a hard disk. The processor 401 is configured to temporarily store data to exchange with the external memory 4022, the processor 401 exchanges data with the external memory 4022 via the storage 4021, and when the electronic device 400 is executed, the processor 401 and the memory 402 communicates via bus 403 to processor 401 to
Get the detected image,
generating an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map is used to represent the relative positional relationship between each target object in the detected image; , each of said keypoint heatmaps includes keypoints of one category of said detected image, wherein keypoints of different categories correspond to different parts of said target object;
generating a keypoint map model based on the image feature map and a plurality of the keypoint heatmaps, wherein the keypoint map model includes information of keypoints of different categories in the detected image and connection edge information; information is included, each connecting edge is an edge between two different categories of keypoints,
Cause the instructions to determine each keypoint belonging to the same target object based on the keypoint map model.

It should be noted that an embodiment of the present invention further provides a computer readable storage medium storing a computer program, which, when executed by a processor, detects keypoints according to the above method embodiments. Carry out the steps of the method.

A computer program product of a method for detecting keypoints according to an embodiment of the present invention includes a computer readable storage medium storing program code, instructions contained in the program code are instructions for detecting keypoints according to the above method embodiments. It can be configured to perform the steps of the method, and reference can be made to the above method embodiments, which will not be repeated here.

It can be understood by those skilled in the art that for convenience and brevity of explanation, the working process of the system and apparatus described above can refer to the corresponding process in the above method embodiments, and here again No explanation. It should be understood that in some embodiments according to this invention, the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are only exemplary, for example, the separation of the units is only the separation of logical functions, and there are other separation methods in actual implementation, such as multiple units Or components may be integrated or integrated into another system, or some features may be ignored or not performed. It should be noted that the mutual couplings or direct couplings or communication connections shown or discussed may be realized using a number of communication interfaces, and the indirect couplings or communication connections between devices or units may be electrical or mechanical. It may be in a typical form or in another form.

Units described as separate parts may or may not be physically separated, and parts denoted as units may or may not be physical units. That is, it may be located at one location or distributed over a plurality of network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the technical solution of this embodiment.

Furthermore, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be present. may be integrated into one unit.

When the functions are implemented in the form of software functional units and sold or used as stand-alone products, they can be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solution of the present invention may be embodied in the form of a software product, essentially or the part that contributes to the prior art or part of the said technical solution. The computer software product may be stored on a single storage medium and may be stored on a single computer device (which may be a personal computer, server, network device, etc.) to perform all or one of the methods described in each embodiment of the present invention. contains some instructions to cause the steps of the part to be carried out. The above-mentioned storage media include U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, optical disk, etc., which can store program code. media.

The above are only specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. All permutations should fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

The present invention generates a keypoint map model corresponding to the detected image based on the generated image feature map and a plurality of keypoint heat maps, wherein the keypoint map model includes an image feature map and a key point heat map. Based on the keypoint map model, because the information in the point heat map is included, and the image feature map can represent the relative positional relationship between different target objects in the image to be detected, the keypoints of different target objects can be correctly distinguished to improve the accuracy of keypoint clustering.

Claims (21)

A keypoint detection method comprising:
obtaining an image to be detected;
generating an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map represents the relative positional relationship between each target object in the detected image. wherein each said keypoint heatmap contains keypoints of one category of said detected image, with keypoints of different categories corresponding to different parts of said target object;
generating a keypoint map model based on the image feature map and a plurality of the keypoint heatmaps, wherein the keypoint map model contains information of different categories of keypoints in the detected image; and connecting edge information, each connecting edge being an edge between two different categories of keypoints;
determining each keypoint belonging to the same target object based on the keypoint map model. Determining each keypoint belonging to the same target object based on the keypoint map model includes:
determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edge;
determining each keypoint belonging to the same target subject based on the determined correlation;
The keypoint detection method according to claim 1 . Determining each keypoint belonging to the same target subject based on the determined correlations includes:
using each keypoint for which the corresponding correlation is greater than a set threshold as a keypoint of the same target;
The keypoint detection method according to claim 2. Determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edge,
determining, for each keypoint, a fusion feature for the keypoint based on information about the keypoint and information about other keypoints in the keypoint map model that have a connection relationship with the keypoint; ,
determining a correlation between two keypoints with a connection relationship in the keypoint map model based on the fusion features respectively corresponding to each keypoint;
4. The keypoint detection method according to claim 2 or 3. the keypoint information includes location information, category information, and pixel feature information;
determining location information for each keypoint based on the keypoint heatmap;
extracting the pixel feature information of the keypoint from the image feature map based on the location information of each of the keypoints, and determining the category corresponding to the keypoint based on the category label of the keypoint heatmap to which the keypoint belongs; determining information for each keypoint of the keypoint map model according to determining information;
A keypoint detection method according to any one of claims 1 to 4. according to the category information corresponding to each keypoint, connecting each keypoint with other keypoints different from the category to which the keypoint belongs to form a connecting edge in the keypoint map model; , to generate each connecting edge in the keypoint map model;
The keypoint detection method according to claim 5 . connecting each keypoint with a keypoint corresponding to a target category that matches the category to which the keypoint belongs, based on the category information corresponding to each keypoint and the matching relationship between different categories of the preset; generating each connecting edge in the keypoint map model according to forming connecting edges in the keypoint map model;
The keypoint detection method according to claim 5 . After determining each keypoint belonging to the same target object based on the keypoint map model,
further comprising determining a behavior type of the target subject based on the information of each keypoint corresponding to each target subject;
A keypoint detection method according to any one of claims 1 to 7. After determining each keypoint belonging to the same target object based on the keypoint map model,
further comprising generating special effect information for each target object based on the information of each keypoint corresponding to each target object;
A keypoint detection method according to any one of claims 1 to 7. A keypoint detection device comprising:
an acquisition module configured to acquire the detected image;
A first generating module configured to generate an image feature map and a plurality of keypoint heat maps based on the detected image, wherein the image feature map is for each target of the detected image. used to represent the relative positional relationship between objects, each said keypoint heatmap contains keypoints of one category of said detected image, and keypoints of different categories correspond to different positions of said target object; a first generation module corresponding to the part;
A second generation module configured to generate a keypoint map model based on the image feature map and a plurality of the keypoint heat maps, wherein the keypoint map model includes: a second generating module comprising information of keypoints of different categories and information of connecting edges of
a determination module configured to determine each keypoint belonging to the same target object based on the keypoint map model. When the determination module determines each keypoint belonging to the same target object based on the keypoint map model,
determining a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint in the keypoint map model and the information of the connecting edge;
configured to determine each keypoint belonging to the same target subject based on the determined correlation;
The keypoint detection apparatus according to claim 10. When the determining module determines each keypoint belonging to the same target object based on the determined correlation,
configured to use each keypoint for which the corresponding correlation is greater than a set threshold as the same target keypoint;
The keypoint detection device according to claim 11 . When the determination module determines a correlation between two keypoints having a connection relationship in the keypoint map model based on the information of each keypoint of the keypoint map model and the information of the connecting edge,
determining, for each keypoint, a fusion feature of the keypoint based on information about the keypoint and information about other keypoints in the keypoint map model that have a connection relationship with the keypoint;
configured to determine a correlation between two keypoints having a connection relationship in the keypoint map model based on the fusion features respectively corresponding to each keypoint;
A keypoint detection device according to claim 11 or 12. the keypoint information includes location information, category information, and pixel feature information;
The second generation module is
determining location information for each keypoint based on the keypoint heatmap;
extracting the pixel feature information of the keypoint from the image feature map based on the location information of each of the keypoints, and determining the category corresponding to the keypoint based on the category label of the keypoint heatmap to which the keypoint belongs; configured to determine information for each keypoint of the keypoint map model according to determining information;
14. A keypoint detection device according to any one of claims 10-13. The second generation module is
according to the category information corresponding to each keypoint, connecting each keypoint with other keypoints different from the category to which the keypoint belongs to form a connecting edge in the keypoint map model; , to generate each connecting edge in the keypoint map model;
15. The keypoint detection apparatus according to claim 14. The second generation module is
connecting each keypoint with a keypoint corresponding to a target category that matches the category to which the keypoint belongs, based on the category information corresponding to each keypoint and the matching relationship between different categories of the preset; generating each connecting edge in the keypoint map model according to forming connecting edges in the keypoint map model;
15. The keypoint detection apparatus according to claim 14. After determining each keypoint belonging to the same target object based on the keypoint map model, further:
a behavior type determination module configured to determine a behavior type of each target subject based on information of each keypoint corresponding to each target subject;
17. A keypoint detection device according to any one of claims 10-16. After determining each keypoint belonging to the same target object based on the keypoint map model, further:
a special effect information generation module configured to generate special effect information for each target object based on information of each keypoint corresponding to each target object;
17. A keypoint detection device according to any one of claims 10-16. An electronic device comprising a processor, a memory and a bus, wherein the memory stores executable machine-readable instructions of the processor, and when the electronic device is executed, the processor and the memory are connected via the bus. , wherein the machine-readable instructions, when executed by the processor, perform the steps of the keypoint detection method of any one of claims 1 to 9. A computer readable storage medium storing a computer program, said computer program, when executed by a processor, performing the steps of the keypoint detection method according to any one of claims 1 to 9. , said computer readable storage medium. 10. A computer program product comprising computer readable code, wherein when said computer readable code is executed in an electronic device, a processor in said electronic device executes the key point of any one of claims 1-9. Said computer program product for performing the steps for realizing the detection method of

Images