KR20220004009A - Key point detection method, apparatus, electronic device and storage medium - Google Patents

Abstract

The present invention provides a key point detection method, apparatus, electronic device and storage medium, the method comprising the steps of: acquiring an image to be detected; generating an image feature map and a plurality of key point heat maps based on the image to be detected, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected, each of the key points the point heat map includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object; generating a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information on key points of different categories among the images to be detected and information on connection edges , and each connecting edge is an edge between key points of two different categories; and determining each key point belonging to the same target object based on the key point image model.

Description

Key point detection method, apparatus, electronic device and storage medium

The present invention is filed on the basis of a Chinese patent application with an application number of 202010622132.3 and an application date of June 30, 2020, and claims the priority of the Chinese patent application, all contents of the Chinese patent application are incorporated herein by reference. are cited

The present invention relates to the field of image detection technology, and more particularly to a key point detection method, apparatus, electronic device and storage medium.

Human body key point detection is to provide support for higher layer information by analyzing human behavior in video, and is the basis for implementing video human motion recognition and human-machine interface. Recently, a human body key point detection method based on a deep neural network has become a topic of research because the input image is simple and easy to acquire, and the detection effect is efficient and accurate.

In general, in a human key point detection method based on a deep neural network, the key point detection and the key point clustering are two independent steps, that is, the key point clustering is generally an indifferentiable post-processing operation. However, in this way, the accuracy of the key point clustering process is relatively low, that is, clustering errors may occur, and different people's key points are clustered together with the same person's key points, resulting in an error in the detection result. .

In view of this, embodiments of the present invention provide at least a key point detection method, an apparatus, an electronic device, and a storage medium.

In a first aspect, an embodiment of the present invention provides a key point detection method,

acquiring an image to be detected;

generating an image feature map and a plurality of key point heat maps based on the image to be detected, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected, each of the key points the point heat map includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object;

generating a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information on key points of different categories among the images to be detected and information on connection edges , and each connecting edge is an edge between key points of two different categories; and

and determining, based on the key point image model, each key point belonging to the same target object.

Using the above method, based on the generated image feature map and the plurality of key point heat maps, a key point image model corresponding to the image to be detected may be generated, wherein the key point image model includes an image feature map and a key point hit map. Since it contains information in the map, and the image feature map can indicate the relative positional relationship between different target objects among the images to be detected, based on the key point image model, the key points of different target objects are distinguished relatively accurately, It can improve the accuracy of key point clustering.

In a possible embodiment, based on the key point image model, determining each key point belonging to the same target object comprises:

determining a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge; and

and determining each key point belonging to the same target object based on the determined correlation.

In a possible embodiment, the determining, based on the determined correlation, each key point belonging to the same target object comprises: assigning each key point having the corresponding correlation greater than the predetermined threshold value to the same target object. and taking as key points.

In a possible embodiment, the determining of a correlation between two key points having a connection relationship in the key point image model based on the information of each key point in the key point image model and the information of the connection edge includes: , for each key point, determining a fusion characteristic of the key point based on information on the key point and information on other key points having a connection relationship with the key point in the key point image model; and determining a correlation between two key points having a connection relationship in the key point image model, based on the fusion feature respectively corresponding to each key point.

In a possible embodiment, the information of the key point includes location information, category information, and pixel characteristic information; determining location information of each key point based on the key point heat map; and extracting pixel feature information of the key point from the image feature map based on the location information of each key point, and based on the category label of the key point heat map to which the key point belongs, corresponding to the key point According to the step of determining the category information, information of each key point in the key point image model is determined.

In a possible embodiment, based on the category information corresponding to each key point, each key point is associated with another key point different from the category to which the key point belongs, and a connection edge in the key point image model is formed. Create each connecting edge in the key point image model according to the steps.

In a possible embodiment, each key point is associated with a key point corresponding to a target category matching a category to which the key point belongs, based on a matching relationship between the category corresponding to each key point and a predetermined different category Thus, each connection edge in the key point image model is generated according to the step of forming a connection edge in the key point image model.

In a possible embodiment, after determining each key point belonging to the same target target based on the key point image model, the behavior type of the target target is determined based on information of each key point corresponding to each target target further comprising the step of

In a possible embodiment, after determining each key point belonging to the same target object based on the key point image model, based on the information of each key point corresponding to each target object, special effect information for the target object It further comprises the step of generating

The description of the effects of the device, electronic device, etc. below may refer to the description of the method, and will not be repeated any longer.

In a second aspect, an embodiment of the present invention provides a key point detection device,

an acquiring module configured to acquire an image to be detected;

a first generating module, configured to generate, based on the image to be detected, an image feature map and a plurality of key point heat maps, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected , each of the key point heatmaps includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object;

a second generating module, configured to generate a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information and connecting edges of key points of different categories among the to-be-detected images contains information from, each connecting edge being an edge between key points of two different categories; and

and a determining module, configured to determine, based on the key point image model, each key point belonging to the same target object.

In a third aspect, an embodiment of the present invention provides an electronic device, the electronic device comprising a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor; When a device is operated, communication between the processor and the memory is via a bus, and when the machine readable instructions are executed by the processor, the steps of the key point detection method according to the first aspect or any one of the embodiments run

In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is operated by a processor, the first aspect or Execute the steps of the key point detection method according to any one of the embodiments.

An embodiment of the present invention further provides a computer program product comprising computer readable code, wherein when the computer readable code is operated in an electronic device, a processor in the electronic device is configured to provide a server in the one or more embodiments to implement the method executed by

In order to make it easier to understand the above objects, features and advantages of the present invention, preferred embodiments will be described in detail below in conjunction with the accompanying drawings.

In order to more clearly explain the technical solutions of the embodiments of the present invention, the drawings to be used in the embodiments are briefly described below, wherein the drawings are a part of the specification and constitute the entire specification, and these drawings are consistent with the present invention. It shows an embodiment to be used, and is for explaining the technical solution of the present invention together with the specification. It is to be understood that the drawings below only show some embodiments of the present invention, and therefore should not be regarded as limiting in scope, and those skilled in the art may differ according to these drawings, even under the premise that creative labor is not imparted Associated drawings can be obtained.
1 is a flowchart illustrating a keypoint detection method provided in an embodiment of the present invention.
2A is an exemplary diagram of a key point image model provided in an embodiment of the present invention.
2B is a system block diagram of a key point detection method provided in an embodiment of the present invention.
Figure 2c is an application scenario diagram of the key point detection method provided in the embodiment of the present invention.
3 is an exemplary architecture diagram of a key point detection apparatus provided in an embodiment of the present invention.
4 is an exemplary structural diagram of an electronic device 400 provided in an embodiment of the present invention.

In order to make the objects, technical solutions and advantages of the embodiments of the present invention more clear, below, in combination with the drawings in the embodiments of the present invention, the technical solutions in the embodiments of the present invention are clearly and completely While describing, it is clear that the described embodiments are merely some, but not all, embodiments of the present invention. In general, the components of the embodiments of the invention described and illustrated in the drawings herein may be arranged and designed in a variety of different configurations. Accordingly, the detailed description of the embodiments of the present invention provided in the drawings below is not intended to limit the protection scope of the present invention, but to represent selected embodiments of the present invention. Based on the embodiment of the present invention, all other embodiments obtained by those skilled in the art without creative labor should fall within the protection scope of the present invention.

In general, by using a deep neural network to detect an image, it is possible to determine key point information of a target object included in the image, wherein the key point detection method based on the deep neural network includes the detection of the key point and the Includes clustering.

Currently, the detection of key points and the clustering of key points are two independent steps, that is, the clustering of key points is generally a non-differentiable post-processing operation. However, when this method is used, the accuracy of the key point clustering process is relatively low, that is, clustering errors may occur. will do

In order to solve the above problem, an embodiment of the present invention provides a key point detection method.

In order to easily understand the embodiment of the present invention, first, a key point detection method disclosed in the embodiment of the present invention will be described in detail.

The execution subject of the key point detection method provided in the embodiment of the present invention may be a server, and the server may be a local server or a cloud server; Alternatively, the execution subject of the method may be a terminal device, for example, the terminal device may be a mobile phone, a tablet PC, AR glasses, or the like.

1 is a process exemplification diagram of a key point detection method provided in an embodiment of the present invention, and the method includes steps S101 to S104.

In step S101, an image to be detected is acquired.

In step S102, based on the image to be detected, an image feature map and a plurality of key point heat maps are generated; The image feature map is for indicating the relative positional relationship between each target object in the image to be detected; Each key point heat map includes key points of one category of an image to be detected, and key points of different categories correspond to different parts of the target object.

In step S103, based on the image feature map and the plurality of key point heat maps, a key point image model is generated; The key point image model includes information on key points of different categories and information on connecting edges among images to be detected, and each connecting edge is an edge between key points of two different categories.

In step S104, each key point belonging to the same target object is determined based on the key point image model.

In the method, based on the generated image feature map and the plurality of key point heat maps, a key point image model corresponding to the image to be detected may be generated, and the key point image model is an image feature map and a key point heat map. Since the image feature map can indicate the relative positional relationship between different target objects in the image to be detected, based on the key point image model, key points of different target objects can be distinguished relatively accurately, so that the key The accuracy of point clustering can be improved.

For steps S101 and S102,

The image to be detected may be any image including the target object. In the implementation process, an image to be detected may be acquired from a connected storage device, and an image to be detected may be acquired in real time from a connected photographing device.

In the implementation process, input the obtained image to be detected into a trained key point detection neural network to generate an image feature map and a plurality of key point heat maps; Each key point of each target may be determined based on the image feature map, the plurality of key point heat maps, and the trained key point detection neural network.

Here, each key point heat map includes key points of one category of an image to be detected, and key points of different categories correspond to different parts of the target object. For example, a category of key points may be head, neck, hand, etc., a key point heatmap may be an image containing a head key point, or a key point heat map may be an image containing a neck key point. there is; Alternatively, the category of the key point may be a set first category, a second category, etc., where the first category key point may be a key point of a thumb, and the second category key point may be a key point of an index finger, etc. , the key point heat map may include an image of the first category key point, or the key point heat map may include an image of the second category key point. Here, the category of the key point and the number of categories may be set according to actual demand. The number of key points corresponding to each target may be set according to actual demand, for example, the number of key points corresponding to each target may be 17, 105, or the like.

Here, the number of key point heatmaps coincides with the set number of key point categories. For example, if the set number of key point categories is 17, the number of key point heatmaps generated based on the image to be detected is also 17 pieces. Here, the number of key points in each category may be one or plural.

The number of image feature maps may be one or plural. Here, when the number of image feature maps is one, the image feature map may indicate a relative positional relationship between each target object and a portion corresponding to a key point of each category among images to be detected. When the number of image feature maps is plural, the number of image feature maps and the number of key point heatmaps may be the same, and each image feature map corresponds to a key point of one category of each target object among images to be detected. It can represent the relative positional relationship between the parts. Here, the size of the image feature map and the size of the key point heat map match.

In the implementation process, by setting different loss functions in the key point detection neural network, an image feature map and a plurality of key point heat maps may be obtained.

For step S103,

Here, information of each key point can be extracted and obtained from a plurality of key point heatmaps and image feature maps, and each key point containing information is taken as a node, and edges between key points of different categories are used as connection edges. , to construct the key point image model.

In a selectable embodiment, the information of the key point may include location information, category information, and pixel characteristic information. Here, based on the key point heat map, determining the location information of each key point; and extracting pixel feature information of a key point from an image feature map based on the location information of each key point, and determining category information corresponding to the key point based on the category label of the key point heat map to which the key point belongs According to the steps, it is possible to determine the information of each key point in the key point image model.

In the implementation process, location information of each key point may be determined based on the pixel value of each pixel point in the key point heat map. For example, for each key point heatmap, a pixel point having a maximum pixel value may be selected and determined as one key point, and location information of the selected pixel point may be determined as location information of the key point. Here, when the pixel value of a specific pixel point in the key point heatmap is greater than the pixel value of a neighboring pixel point, the pixel value of the pixel point is regarded as the maximum value, and the pixel point is regarded as the key point.

After obtaining the location information of each pixel point, the pixel value of the pixel point corresponding to the location information may be extracted from the image feature map, and the extracted pixel value may be determined as the pixel feature information of the key point.

Also, based on the category label of the key point heat map to which each key point belongs, category information corresponding to the key point may be determined. For example, if the category label of the key point heat map A is head, the category information of each key point included in the key point heat map A is the head key point; When the category label of the key point heat map B is a neck, the category information of each key point included in the key point heat map B is a mock key point.

In a selectable embodiment, each connecting edge in the key point image model may be generated according to the following two methods.

In method 1, based on category information corresponding to each key point, each key point is linked with another key point different from the category to which the key point belongs to form a connection edge in the key point image model.

In method 2, based on a matching relationship between a category corresponding to each key point and a predetermined different category, each key point is linked with a key point corresponding to a target category matched with a category to which the key point belongs, It forms the connecting edge in the point image model.

In method 1, based on the category information corresponding to each key point, key points having the same category information are not connected, but key points having different category information are connected to form a connection edge in the key point image model can do.

Here, based on the category information corresponding to each key point, each key point and other key points different from the category to which the key point belongs are connected to form a connection edge in the key point image model, thereby obtaining a key point image model Thus, it provides data assistance in subsequently determining each key point of each target object.

In method 2, a matching relationship between different categories may be preset based on the human body structure, for example, the matching relationship between the different preset categories may be a head category and a neck category matching, and a neck category point are respectively matched with the left shoulder category, the right shoulder category, and the head category, respectively. Here, the matching relationship between different categories may be established according to actual demand.

Furthermore, for each key point, a target category matching the category of the key point may be determined, and the key point may be connected with a key point corresponding to the target category to form a connection edge in the key point image model.

In the above implementation manner, matching relationships between different categories may be preset, such as presetting that the head key point matches the neck key point, the foot key point matches the knee key point, etc. Since it connects with key points corresponding to the target category matching the category to which it belongs, it is possible to prevent each key point from being connected to an unmatched key point, and it is possible to reduce the amount of computation of correlation calculation, thereby improving the efficiency of key point detection. improve

As an exemplary view of the key point image model shown in FIG. 2A , the image includes a key point image model 21 generated based on method 1 and a key point image model 22 generated based on method 2 . Here, the image includes key points 201 of a first category, key points 202 of a second category, and key points 203 of a third category, further adding connecting edges 204 between different key points. include As shown in the figure, the key point image model 21 includes a connecting edge between each key point of a different category; The key point image model 22 includes a connecting edge between different key points having a matching relationship, and as shown in the figure, the preset matching relationship between different categories is the matching of the first category and the second category, the second It is category 2 and category 3 matching.

For step S104,

Here, based on the key point image model, each key point included in the key point image model may be divided, and a plurality of key points belonging to the same target target may be divided together, so that each target target included in the image to be detected may be divided. A plurality of key points corresponding to can be obtained. Here, the number of key points corresponding to each target object is the same.

In a selectable embodiment, determining, based on the key point image model, each key point belonging to the same target object comprises A1 and A2.

In step A1, a correlation between two key points having a connection relationship in the key point image model is determined based on information on each key point in the key point image model and information on a connection edge.

In step A2, each key point belonging to the same target object is determined based on the determined correlation.

Under the above implementation method, a correlation between two key points having a connection relationship in the key point image model may be determined based on the information of each key point and the information of the connection edge, and the correlation is Since the probability of belonging to the same target between key points may be indicated, each key point belonging to the same target may be clustered together through the determined correlation to obtain each key point corresponding to each target.

In step A1, for each connecting edge in the key point image model, a correlation between two key points corresponding to the connecting edge may be determined, that is, a correlation corresponding to each connecting edge may be obtained. .

As an optional embodiment, the determining of a correlation between two key points having a connection relationship in the key point image model based on information of each key point in the key point image model and information on a connection edge includes:

First, for each key point, the fusion characteristic of the key point is determined based on the key point information and information on other key points that have a connection relationship with the key point in the key point image model.

Second, a correlation between two key points having a connection relationship in the key point image model is determined based on a fusion feature corresponding to each key point.

When determining the fusion characteristic of each key point, the fusion characteristic of the key point can be determined based on the information on the key point and information on other key points that have a connection relationship with the key point in the key point image model, Here, the other key point may be a key point having a connecting edge between the key point and the key point in the key point image model.

Here, it is possible to determine a corresponding fusion feature for each key point in the key point image model, and based on the fusion feature respectively corresponding to each key point, two key points corresponding to each connection edge in the key point image model determine the correlation between

In the implementation process, the fusion feature may be a feature vector corresponding to each key point, and by calculating the similarity between the fusion feature (feature vector) of two key points corresponding to each connection edge, the degree of similarity obtained by calculating It is determined by the correlation between these two key points.

Under the above implementation manner, for each key point, the fusion of the key point is determined by determining the fusion characteristic of the key point based on the information of the key point and the information of other key points having a connection relationship with the key point. Since the feature may represent not only the characteristic of the key point, but also the related relationship between the key point and another key point, based on the fusion feature respectively corresponding to each key point, two keys having a connection relationship It is possible to determine a correlation between the points, and based on the correlation, each key point corresponding to each target object can be determined relatively accurately.

In step A2, after it is determined that the correlation between two key points corresponding to each connection edge in the key point image model is obtained, each key point is divided based on each determined correlation and belongs to the same target object Each key point can be determined.

As a selectable embodiment, based on the determined correlation, determining each key point belonging to the same target object comprises: taking each key point having a corresponding correlation greater than a set threshold as a key point of the same target object may include steps.

In the implementation process, based on the setting of the threshold value of correlation, and the target number of key points of each predetermined category, each key point having a corresponding correlation greater than the set threshold is set as a key point of the same target target get drunk Here, the key points of each target object include a plurality of key points of different categories, and the number of key points in each category coincides with the preset number of targets.

In a selectable embodiment, after determining each key point belonging to the same target target based on the key point image model, the behavior type of the target target is determined based on information on each key point corresponding to each target target It may further include the step of

Here, after obtaining information on each key point of each target target, information on each key point of each target target may be input to a behavior detection neural network to determine a behavior type of the target target, for example, the The action type may be running, walking, raising both arms, and the like.

In a selectable embodiment, after determining each key point belonging to the same target object based on the key point image model, special effect information for the target object is generated based on the information of each key point corresponding to each target object It may further include the step of generating.

Here, with respect to the information of each key point of the target target, the location of the target part of the target is determined, and based on the special effect information corresponding to the predetermined target part, special effect information corresponding to the location of the target part of the target target can create Here, the target region may be an arm, a head, a hand, or the like. For example, with respect to the information of each key point of the target target, the arm position of the target target can be determined, and based on the preset arm special effect information, corresponding special effect information can be generated from the arm position of the target target. have.

Here, with respect to the information of each key point corresponding to each target object, the operation type of the target object is determined, and based on the mapping relationship between the preset operation type and the special effect information, a special corresponding to the target object You can create effect information. For example, if it is determined that the operation type of the target object is a heart operation based on information on each key point corresponding to the target object A, heart-shaped special effect information may be generated for the target object.

In the related art, detection of a key point for a target is implemented through a bottom-up method of detecting a plurality of target key points. This method first detects all possible target target key points in the image, and simultaneously extracts information of each key point, such as an embedding feature of the key point; Then, by solving the optimization equation, these key points are clustered and assigned to a plurality of different target objects. As can be seen from this, the bottom-up method is faster to operate and more robust to adjacent targets or blocking situations.

A method of extracting an embedding feature of a key point includes method 1 and method 2 .

In method 1, it is implemented based on the key point embedding feature (Keypoint Embedding, KE).

In method 2, it is implemented based on a key point detection method of spatial instance embedding features (SIE).

Here, in the training process of method 1, since KE mainly includes the apparent information of pixels near the key point, the embedding value of each key point of the same target object is narrowed, and the embedding value of the key point of the different target object is widened. Therefore, it is possible to implement extraction of the embedding feature of the key point.

Here, in the training process of Method 2, since the SIE includes target target center position information, extraction of the embedding feature of the key point may be implemented through a vector in which each pixel value returns to the target target center.

As can be seen from this, KE mainly relates to pixel characteristic information, where the pixel characteristic information may include apparent information of a pixel, such as a pixel value of a pixel, is not sensitive to spatial location information, and is Although it is possible to model the relationship between key points, there may be a problem in that the key points of different target objects at a long distance gather together if only relying on KE because of the lack of space constraints.

As can be seen from this, in the related art, in the bottom-up key point detection method, the key point detection and the key point clustering are two independent steps. Clustering of key points is an indivisible post-processing operation. There are several problems as follows, first, the clustering process of key points is a fixed post-processing, and cannot be optimized by learning from data; Second, since joint optimization is not performed, the clustering precision of key points is not high, and clustering errors such as clustering key points of different target objects as key points of the same target object may occur; Third, the clustering post-processing process requires additional hyperparameter settings.

In order to solve the above problem, an embodiment of the present invention provides a key point detection method, the system block diagram of the method is as shown in Fig. 2b,

In the related art, it includes a key point detection step S21 and a key point clustering step S22. The clustering step of key points is a post-processing operation after key point detection.

An embodiment of the present invention provides a key point detection method, and includes a key point candidate extraction module G1 (Keypoint Candidate Proposal) and an image clustering module G2 (Graph Grouping).

Key point candidate extraction module G1, for the image of each frame, first through a multi-task bottom-up target target key point model, key point heatmaps of target target key points, and image feature maps or spatial instance images This is to directly output feature maps. Since the maximum value (argmax) is obtained through the key point heatmap, the pixel coordinate position X of the key point can be obtained. At the same time, it is possible to obtain the characteristic information F of the key point from the corresponding position information of the image characteristic map.

The image clustering module G2 is configured to determine whether every two key points belong to the same target object, and cluster the key points of the same target object.

For example, FIG. 2C shows a process of inputting an image convolution network into a key point heat map P1 and an image feature map P2. First, the key point information is extracted, and the key point image model G={V, E} is constructed. The key point image model G is divided into two parts, the key point V and the edge E, where the key point V is the information of each key point, that is, "the category T of the key point, the coordinate X of the key point, and the characteristic of the key point. information F”. Edge V indicates the relationship between key points, i.e. whether they belong to the same target object. After building the key point image model, the calculation of key point correlation is performed. Edge Convolution EdgeConv Construction Map Using the convolutional neural network model, convolution is performed on the key point image model, and the feature information of the key point is continuously updated. Next, the edge classification device is trained, and each pair of key points is determined to determine whether the pair of key points belong to the same target object. End-to-end training optimization can be performed through the key point candidate extraction module G1, the image clustering module G2, and the edge classification device.

In a possible embodiment, in one aspect of the key point detection method provided in the embodiments of the present invention, prediction may be performed on the location information of the target key point; In another aspect, it is possible to determine the behavioral type of the target object; In another aspect, real-time special effect information can be increased in different parts of the target object.

Using the above method, it is possible to transform the multi-target object pose estimation into an image clustering problem; by maintaining the topological structure information of the target target to perform feature information extraction for multiple target target pose images; Through one deep neural network model, it is possible to obtain clustering results of multiple target object poses; In the related art, it is possible to prevent post-processing operations such as clustering for key points, so that the operation steps can be simplified; By integrating the detection of key points and clustering of key points, it is possible to improve the clustering precision of key points.

A person skilled in the art will know that in the above method of the embodiment, the recording order of each step does not imply a strict execution order but constitutes any limitation on the implementation process, and the execution order of each step is determined by its function and possible internal logic. can be understood to be determined.

Based on the same concept, an embodiment of the present invention provides a key point detection device, and Fig. 3 is an architectural illustration of the key point detection device provided in an embodiment of the present invention, the acquisition module 301, the first a generating module 302 , a second generating module 303 , a determining module 304 , a behavior type determining module 305 , and a special effect information generating module 306 , wherein:

an acquiring module 301, configured to acquire an image to be detected;

a first generating module 302, configured to generate, based on the image to be detected, an image feature map and a plurality of key point heat maps, wherein the image feature map indicates a relative positional relationship between each target object in the image to be detected wherein each said key point heatmap includes key points of one category of said to-be-detected image, wherein key points of different categories correspond to different parts of said target object;

a second generating module 303, configured to generate a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information of key points of different categories among the to-be-detected images and information of connecting edges, each connecting edge being an edge between key points of two different categories; and

and a determining module 304, configured to determine, based on the key point image model, each key point belonging to the same target object.

In a possible embodiment, when determining, based on the key point image model, each key point belonging to the same target object, the determining module 304 comprises:

determine a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge;

and determine each key point belonging to the same target object based on the determined correlation.

In a possible embodiment, when determining, based on the determined correlation, each key point belonging to the same target object, the determining module 304 is configured to:

and make each key point having the corresponding correlation greater than the preset threshold as a key point of the same target object.

In a possible embodiment, when determining a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge, The determination module 304,

for each key point, determine a fusion characteristic of the key point, based on the information of the key point and information on other key points having a connection relationship with the key point in the key point image model;

and determine a correlation between two key points having a connection relationship in the key point image model, based on the fusion feature respectively corresponding to each key point.

In a possible embodiment, the information of the key point includes location information, category information, and pixel characteristic information;

The second generation module 303 is

determining location information of each key point based on the key point heat map; and

Based on the location information of each key point, pixel characteristic information of the key point is extracted from the image feature map, and a category corresponding to the key point is extracted based on the category label of the key point heat map to which the key point belongs. and determine information of each key point in the key point image model according to the determining information.

In a possible embodiment, the second generating module 303 comprises:

Based on the category information corresponding to each key point, each key point is linked with another key point different from the category to which the key point belongs to form a connection edge in the key point image model; It is configured to create each connecting edge in the image model.

In a possible embodiment, the second generating module 303 comprises:

Based on the category corresponding to each key point and a matching relationship between different preset categories, each key point is linked with a key point corresponding to a target category matching the category to which the key point belongs, and the key point image and generate each connecting edge in the key point image model according to the step of forming a connecting edge in the model.

In a possible embodiment, after determining, based on the key point image model, each key point belonging to the same target object,

and a behavior type determination module 305, configured to determine a behavior type of the target target based on the information of each key point corresponding to each target target.

In a possible embodiment, after determining, based on the key point image model, each key point belonging to the same target object,

and a special effect information generating module 306, configured to generate special effect information for the target object based on the information of each key point corresponding to each target object.

In some embodiments, the functions possessed by the apparatus provided in the embodiments of the present invention or templates including the functions may be used to execute the methods described in the above-described method embodiments, and for implementation, refer to the description of the above-described method embodiments. and, for the sake of brevity, are not further repeated here.

Based on the same technical idea, an embodiment of the present invention further provides an electronic device. 4 is an exemplary structural diagram of an electronic device provided in an embodiment of the present invention, and includes a processor 401 , a memory 402 and a bus 403 . Here, the memory 402 is configured to store an execution instruction, and includes an internal memory 4021 and an external memory 4022 ; Here, the internal memory 4021 is also referred to as an internal storage, and is configured to temporarily store data exchanged with the external memory 4022 such as arithmetic data in the processor 401 and a hard disk, and the processor 401 includes the internal memory ( 4021) and the external memory 4022 perform data exchange, and when the electronic device 400 operates, the processor 401 and the memory 402 communicate through the bus 403, and the processor 401 to execute the steps below. In other words

acquiring an image to be detected;

generating an image feature map and a plurality of key point heat maps based on the image to be detected, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected, each of the key points the point heat map includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object;

generating a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information on key points of different categories among the images to be detected and information on connection edges , and each connecting edge is an edge between key points of two different categories; and

Based on the key point image model, determining each key point belonging to the same target object is performed.

In addition, an embodiment of the present invention further provides a computer readable storage medium, wherein the computer program is stored in the computer readable storage medium, and when the computer program is operated by a processor, detecting the keypoint in the embodiment of the method Execute the steps of the method.

The computer program product of the key point detection method provided in the embodiment of the present invention includes a computer readable storage medium including a program code, and the instructions included in the program code are the key point detection method according to the embodiment of the method. may be used to perform the steps of , and reference may be made to the above method embodiments, which will not be repeated herein any longer.

A person skilled in the art may understand that, for convenience and brevity of description, the operation process of the above-described system and apparatus may refer to a corresponding process among the above-described method embodiments, which will be described repeatedly I never do that. In the several embodiments provided herein, it should be understood that the disclosed systems, apparatuses, and methods may be implemented in other ways. The above-described device embodiment is merely exemplary, for example, the division for the unit is merely logical function division, and when actually implemented, there may be other division methods, and for example, a plurality of units or components may be It may be combined or integrated into other systems, or some features may be ignored or not implemented. Further, any coupling or direct coupling or communication connection between each other shown or discussed may be implemented through some communication interface, and the indirect coupling or communication connection through a device or unit may be electrical, mechanical, or other form.

The unit described as the separation member may or may not be physically separated, and the member shown as a unit may or may not be a physical unit, that is, it may be located in one place or may be distributed in a plurality of network units. According to actual needs, some or all of the units may be selected to implement the purpose of the present embodiment method.

In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, each unit may be an independent physical entity, and two or two or more units may be integrated into one unit. .

When the above functions are implemented in the form of a software function unit and sold or used as a standalone product, one processor may be stored in an executable nonvolatile computer readable storage medium. Based on this understanding, the technical solution of the present invention, that is, a part contributing to the prior art or a part of the technical solution may be implemented in the form of a software product, the computer software product being stored in one storage medium, one It includes a plurality of instructions used to cause a computer device (which may be a personal computer, a server or a network device, etc.) of The above-mentioned storage medium includes a variety of media capable of storing a program code, such as a U disk, a mobile disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk. include

The above description is only specific embodiments of the present invention, and the protection scope of the present invention is not limited thereto. Those of ordinary skill in the art to which this application pertains will readily appreciate that all changes or replacements within the technical scope disclosed in the present invention must fall within the protection scope of the present invention. Accordingly, the protection scope of the present invention should be based on the protection scope of the claims.

The present invention can generate a key point image model corresponding to an image to be detected based on the generated image feature map and a plurality of key point heat maps, and the key point image model includes an image feature map and a key point heat map. information, and the image feature map can indicate the relative positional relationship between different target objects in the image to be detected, and thus, based on the key point image model, can distinguish relatively accurately for key points of different target objects, so that the key The accuracy of point clustering can be improved.

Claims (21)

A key point detection method comprising:
acquiring an image to be detected;
generating an image feature map and a plurality of key point heat maps based on the image to be detected, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected, each of the key points the point heat map includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object;
generating a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information on key points of different categories among the images to be detected and information on connection edges , and each connecting edge is an edge between key points of two different categories; and
and determining each key point belonging to the same target object based on the key point image model. According to claim 1,
Determining each key point belonging to the same target object based on the key point image model comprises:
determining a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge; and
and determining each key point belonging to the same target object based on the determined correlation. 3. The method of claim 2,
Based on the determined correlation, determining each key point belonging to the same target includes:
and taking each key point having the corresponding correlation greater than the preset threshold as a key point of the same target object. 4. The method of claim 2 or 3,
Determining a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge includes:
determining, for each key point, a fusion characteristic of the key point based on information on the key point and information on other key points having a connection relationship with the key point in the key point image model; and
and determining a correlation between two key points having a connection relationship in the key point image model based on a fusion feature corresponding to each key point, respectively. 5. The method according to any one of claims 1 to 4,
the key point information includes location information, category information, and pixel characteristic information;
determining location information of each key point based on the key point heat map; and
Based on the location information of each key point, pixel characteristic information of the key point is extracted from the image feature map, and a category corresponding to the key point is extracted based on the category label of the key point heat map to which the key point belongs. Determining information; according to the key point detection device, characterized in that the information of each key point in the key point image model is determined. 6. The method of claim 5,
Based on the category information corresponding to each key point, each key point is linked with another key point different from the category to which the key point belongs, and the key point according to the step of forming a connection edge in the key point image model A method for detecting key points, comprising generating each connecting edge in the image model. 6. The method of claim 5,
Based on the category corresponding to each key point and a matching relationship between different preset categories, each key point is linked with a key point corresponding to a target category matching the category to which the key point belongs, and the key point image A key point detection method, characterized in that generating each connecting edge in the key point image model according to the step of forming a connecting edge in the model. 8. The method according to any one of claims 1 to 7,
After determining each key point belonging to the same target object based on the key point image model,
The method of detecting a key point further comprising the step of determining a behavior type of the target target based on information on each key point corresponding to each target target. 8. The method according to any one of claims 1 to 7,
After determining each key point belonging to the same target object based on the key point image model,
The method of detecting a key point further comprising the step of generating special effect information on the target target based on the information of each key point corresponding to each target target. A key point detection device comprising:
an acquiring module configured to acquire an image to be detected;
a first generating module, configured to generate, based on the image to be detected, an image feature map and a plurality of key point heat maps, wherein the image feature map is for indicating a relative positional relationship between each target object in the image to be detected , each of the key point heatmaps includes key points of one category of the image to be detected, and key points of different categories correspond to different parts of the target object;
a second generating module, configured to generate a key point image model based on the image feature map and the plurality of key point heat maps, wherein the key point image model includes information and connecting edges of key points of different categories among the to-be-detected images contains information from, each connecting edge being an edge between key points of two different categories; and
and a determining module, configured to determine, based on the key point image model, each key point belonging to the same target object. 11. The method of claim 10,
When determining each key point belonging to the same target object based on the key point image model, the determination module is configured to:
determine a correlation between two key points having a connection relationship in the key point image model based on information on each key point in the key point image model and information on the connection edge;
and determine each key point belonging to the same target object based on the determined correlation. 12. The method of claim 11,
When determining each key point belonging to the same target object based on the determined correlation, the determination module is configured to:
and each key point having the corresponding correlation greater than the preset threshold is configured to be a key point of the same target object. 13. The method of claim 11 or 12,
When determining a correlation between two key points having a connection relationship in the key point image model based on the information on each key point in the key point image model and the information on the connection edge, the determination module is configured to:
for each key point, determine a fusion characteristic of the key point, based on the information of the key point and information on other key points having a connection relationship with the key point in the key point image model;
and determine a correlation between two key points having a connection relationship in the key point image model, based on a fusion feature respectively corresponding to each key point. 14. The method according to any one of claims 10 to 13,
the key point information includes location information, category information, and pixel characteristic information;
The second generation module,
determine location information of each key point based on the key point heat map;
Based on the location information of each key point, pixel characteristic information of the key point is extracted from the image feature map, and a category corresponding to the key point is extracted based on the category label of the key point heat map to which the key point belongs. and determine information of each key point in the key point image model according to the step of determining the information. 15. The method of claim 14,
The second generation module,
Based on the category information corresponding to each key point, each key point is linked with another key point different from the category to which the key point belongs, and the key point according to the step of forming a connection edge in the key point image model A key point detection device configured to generate each connecting edge in the image model. 15. The method of claim 14,
The second generation module,
Based on the category corresponding to each key point and a matching relationship between different preset categories, each key point is linked with a key point corresponding to a target category matching the category to which the key point belongs, and the key point image and generate each connecting edge in the key point image model according to the step of forming a connecting edge in the model. 17. The method according to any one of claims 10 to 16,
After determining each key point belonging to the same target object based on the key point image model,
The device for detecting a key point according to claim 1, further comprising: a behavior type determination module configured to determine a behavior type of the target target based on information on each key point corresponding to each target target. 17. The method according to any one of claims 10 to 16,
After determining each key point belonging to the same target object based on the key point image model,
The key point detection apparatus according to claim 1, further comprising: a special effect information generating module configured to generate special effect information on the target object based on the information of each key point corresponding to each target object. As an electronic device,
a processor, a memory and a bus, wherein the memory stores executable machine readable instructions, and when an electronic device is operated, the processor and the memory communicate via a bus, and the machine readable instructions 10. Computer device, characterized in that when executed by a processor, it executes the steps of the method for detecting a key point according to any one of claims 1 to 9. A computer-readable storage medium having a computer program stored therein, comprising:
A computer-readable storage medium, characterized in that when the computer program is executed by a processor, the steps of the method for detecting a key point according to any one of claims 1 to 9 are executed. A computer program product comprising computer readable code, comprising:
10. A computer program product, characterized in that when the computer readable code is run in an electronic device, a processor in the electronic device implements the step of the key point detection method according to any one of claims 1 to 9.

Images