CN110532984A - Critical point detection method, gesture identification method, apparatus and system - Google Patents

Abstract

The invention provides a key point detection method, a gesture recognition method, a device and a system, which relate to the technical field of artificial intelligence and include: acquiring an image to be detected, and the image to be detected contains a target object; performing target detection on the image to be detected to obtain each position frames of each target object; merge the position frames based on the position frames of each target object to obtain a target merged frame; the image corresponding to the target merged frame includes at least two target objects in contact; the image to be detected based on the target merged frame Perform key point detection to obtain multiple heat maps for each target object; wherein, different heat maps are used to represent key points located at different positions on the target object; based on the heat maps, the key points of each target object in contact are determined. The present invention can effectively improve the detection accuracy of key points.

Description

Key point detection method, gesture recognition method, device and system

technical field

The invention relates to the technical field of artificial intelligence, and in particular, to a key point detection method, a gesture recognition method, a device and a system.

Background technique

The detection technology of target objects such as gestures and faces in images is an important application of artificial intelligence. The key point detection of the target object is a key part of the target detection, and the key point information of the target object helps to determine the pose of the target object more accurately. In recent years, with the development of deep learning methods in the problem of target object pose estimation, the method of target object detection combined with target object key point estimation is the most common and computationally efficient key point detection method. Individual target objects or multiple target objects that are completely separated are processed. Once the target objects interact with each other, such as crossing hands, writing with one hand on the palm of the other hand, etc., it is difficult to accurately detect The keypoint of the target object.

SUMMARY OF THE INVENTION

In view of this, the purpose of the present invention is to provide a key point detection method, gesture recognition method, device and system, which can effectively improve the detection accuracy of key points.

In order to achieve the above purpose, the technical solutions adopted in the embodiments of the present invention are as follows:

In a first aspect, an embodiment of the present invention provides a method for detecting key points, the method includes: acquiring an image to be detected, the image to be detected contains a target object; performing target detection on the image to be detected to obtain each a position frame of the target object; perform position frame merging based on the position frame of each target object to obtain a target merge frame; wherein, the image corresponding to the target merge frame includes at least two target objects in contact; Perform key point detection on the to-be-detected image based on the target merging frame to obtain multiple heatmaps for each target object; wherein the different heatmaps are used to represent different positions on the target object The key points of each of the contacting target objects are determined based on the heat map.

Further, the step of performing position frame merging based on the position frame of each of the target objects to obtain the target merged frame includes: based on the position frame of each of the target objects, repeatedly performing a preset merging operation on the position frame, Until the overlapping degree between any two position boxes is not greater than the preset overlapping degree threshold, the target merged frame is obtained.

Further, the step of performing position frame merging based on the position frame of each of the target objects to obtain the target merged frame includes: based on the position frame of each of the target objects, repeatedly performing a preset merging operation on the position frame, Until the degree of overlap between any two position frames is not greater than the preset overlap degree threshold, a candidate merging frame is obtained; according to the number of target objects contained in the candidate merging frame, a target is obtained from the candidate merging frame by screening Merge box.

Further, the merging operation includes: determining a position frame pair to be merged from a plurality of position frames; wherein, the position frame pair includes two position frames; calculating the degree of overlap between the position frames in the position frame pair; If the calculated overlap degree is greater than the preset overlap degree threshold, merge the location boxes in the pair of location boxes into a new location box; wherein, the boundary of the new location box is based on two of the location boxes in the pair of location boxes. The bounds of the location box are determined.

Further, the step of determining the pair of position frames to be merged from a plurality of position frames includes: obtaining the confidence of the position frames; sorting the position frames according to the confidence of each position frame, and obtaining a ranking result of the position frames; A pair of location boxes to be merged is determined from the plurality of location boxes according to the result of the location box sorting.

Further, the step of obtaining the confidence level of the position box includes: acquiring the confidence level of two position boxes in the position box pair, and obtaining the confidence level of the new position box according to the confidence level of the two position boxes .

Further, the step of performing key point detection on the to-be-detected image based on the target merging frame to obtain multiple heat maps of each target object includes: extracting data from the to-be-detected image based on the target merging frame Extract a partial image from the middle; the partial image contains the contacting target objects; adjust the size of the partial image, and perform key point detection on the partial image after the size adjustment, so as to obtain the multi-dimensional image of each target object. a heatmap.

Further, the step of performing key point detection on the resized local image to obtain multiple heat maps of each of the target objects includes: performing key point detection on the resized local image through a trained detection model , to obtain multiple heatmaps for each of the target objects.

Further, the method further includes: inputting a plurality of training images marked with the positions of key points of the target object to the detection model to be trained, wherein the training images include at least two target objects in contact, and any two The degree of overlap between the target objects reaches a preset threshold of overlap degree; the training image is detected by the detection model to be trained, and the heat map of each of the target objects in the training image is output; based on the The heat map of each target object in the training image obtains the key point position in each target object in the training image; the key point position obtained based on the detection model to be trained and the marked key point position pair The parameters of the detection model to be trained are optimized, until the matching degree between the key point positions obtained by the detection model to be trained and the marked key point positions reaches the preset matching degree, it is determined that the training is over, and the post-training is obtained. detection model.

Further, the step of determining the key points of each of the contacted target objects based on the heat map includes: obtaining a brightness value of each pixel in the heat map; wherein the brightness value is used to characterize the The confidence of the corresponding key points in the heat map; filter the heat map according to the preset key point brightness threshold and the obtained maximum brightness value; determine the key points of the contacting target objects according to the filtered heat map point.

In a second aspect, an embodiment of the present invention further provides a gesture recognition method, the method comprising: performing key point detection on a hand image to be detected by using the key point detection method according to any one of the above first aspects, and obtaining each The key points of the hand; the gesture category is recognized according to the key points of each said hand.

In a third aspect, an embodiment of the present invention further provides a key point detection device, the device includes: an image acquisition module for acquiring an image to be detected, the to-be-detected image contains a target object; a target detection module for Performing target detection on the to-be-detected image to obtain a position frame of each of the target objects; a position frame merging module for merging position frames based on the position frames of each of the target objects to obtain a target merge frame; wherein, The image corresponding to the target merging frame includes at least two target objects in contact; a key point detection module is configured to perform key point detection on the to-be-detected image based on the target merging frame to obtain each target object A plurality of heatmaps of the target object; wherein, the different heatmaps are used to represent key points located at different positions on the target object; the keypoint determination module is used to determine each of the targets in contact based on the heatmaps keypoints of the object.

In a fourth aspect, an embodiment of the present invention further provides a gesture recognition device, the device includes: a hand key point detection module, configured to use the key point detection method according to any one of the first aspect to detect the hand to be detected The image is detected by key points to obtain the key points of each hand; the gesture recognition module is used for recognizing gesture categories according to the key points of each hand.

In a fifth aspect, an embodiment of the present invention provides a key point detection system, the system includes: an image acquisition device, a processor, and a storage device; the image acquisition device is used to acquire an image to be detected; A computer program is stored that, when executed by the processor, executes the keypoint detection method according to any one of the first aspects and the gesture recognition method according to the second aspect.

In a sixth aspect, an embodiment of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can run on the processor, and the processor implements the first aspect when executing the computer program The steps of any one of the key point detection method and the steps of the gesture recognition method according to the second aspect.

In a seventh aspect, an embodiment of the present invention provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, any one of the above-mentioned first aspects is executed. The steps of the key point detection method and the steps of the gesture recognition method according to the second aspect.

The embodiments of the present invention provide a key point detection method, a gesture recognition method, a device and a system, which can obtain the position frame of each target object by first performing target detection on the image to be detected, and then perform the target detection based on the position frame of each target object. The position frames are merged to obtain a target merged frame, and the image corresponding to the target merged frame includes at least two contacting target objects; then, based on the target merged frame, the key points of the image to be detected are detected, and multiple thermal powers of each target object are obtained. Figure; wherein, different heat maps are used to represent key points located at different positions on the target object; finally, the key points of each target object in contact are determined based on the heat map. This embodiment can detect key points based on target merging frames corresponding to at least two contacting target objects, and obtain multiple heatmaps that can represent keypoints at different positions on the contacting target objects, so as to determine based on the heatmaps The key points of the target object, this method can effectively improve the problem of inaccurate key point detection caused by the contact interaction between the target objects, and improve the detection accuracy of the key points.

Additional features and advantages of the present invention will be set forth in the description which follows, or some may be inferred or unambiguously determined from the description, or may be learned by practice of the above-described techniques of the present disclosure.

In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to illustrate the specific embodiments of the present invention or the technical solutions in the prior art more clearly, the following briefly introduces the accompanying drawings that need to be used in the description of the specific embodiments or the prior art. Obviously, the accompanying drawings in the following description The drawings are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings without creative efforts.

FIG. 1 shows a schematic structural diagram of an electronic device provided by an embodiment of the present invention;

FIG. 2 shows a flowchart of a key point detection method provided by an embodiment of the present invention;

FIG. 3 shows a schematic diagram of two hands with different degrees of contact provided by an embodiment of the present invention;

FIG. 4 shows a flowchart of a method for performing a merge operation multiple times according to an embodiment of the present invention;

FIG. 5 shows a schematic diagram of the combination of a location frame provided by an embodiment of the present invention;

6 shows a flowchart of another method for performing a merge operation multiple times provided by an embodiment of the present invention;

FIG. 7 shows a schematic diagram of a heat map of a hand provided by an embodiment of the present invention;

FIG. 8 shows a structural block diagram of a key point detection apparatus provided by an embodiment of the present invention.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of them. example. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

Considering the existing detection methods for object keypoints, it is difficult to accurately detect keypoints on contact-interactive objects. Based on this, in order to improve the above problems, a key point detection method, gesture recognition method, device and system provided by the embodiments of the present invention can be applied to various applications such as human-computer interaction and gesture recognition that need to use key point detection. In order to facilitate understanding, the following describes the embodiments of the present invention in detail.

Example 1:

First, an example electronic device 100 for implementing the keypoint detection method, gesture recognition method, apparatus, and system according to an embodiment of the present invention is described with reference to FIG. 1 .

As shown in FIG. 1 is a schematic structural diagram of an electronic device, the electronic device 100 includes one or more processors 102, one or more storage devices 104, an input device 106, an output device 108, and an image acquisition device 110. These components are The bus system 112 and/or other form of connection mechanism (not shown) are interconnected. It should be noted that the components and structures of the electronic device 100 shown in FIG. 1 are only exemplary and not restrictive, and the electronic device may also have other components and structures as required.

The processor 102 may be a central processing unit (CPU) or other form of processing unit having data processing capabilities and/or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

The storage device 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and/or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and/or cache memory, or the like. The non-volatile memory may include, for example, read only memory (ROM), hard disk, flash memory, and the like. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below. and/or other desired functionality. Various application programs and various data, such as various data used and/or generated by the application program, etc. may also be stored in the computer-readable storage medium.

The input device 106 may be a device used by a user to input instructions, and may include one or more of a keyboard, mouse, microphone, touch screen, and the like.

The output device 108 may output various information (eg, images or sounds) to the outside (eg, a user), and may include one or more of a display, a speaker, and the like.

The image capture device 110 may capture images (eg, photos, videos, etc.) desired by the user, and store the captured images in the storage device 104 for use by other components.

Exemplarily, exemplary electronic devices for implementing a key point detection method, gesture recognition method, apparatus, and system according to embodiments of the present invention may be implemented as smart phones, tablet computers, computers, VR devices, cameras, etc. on the terminal.

Embodiment 2:

Referring to the flowchart of a key point detection method shown in FIG. 2, the method specifically includes the following steps:

In step S202, an image to be detected is acquired, and the image to be detected contains a target object. The image to be detected may be an original image captured by an image acquisition device, or an image downloaded from the network, stored locally or manually uploaded. The to-be-detected image may include at least two contacting target objects, such as a person, a human face, a hand, a vehicle, etc. The at least two contacting target objects may refer to the three contact degrees shown in FIG. 3 . Two different hands: The left picture in Figure 3 shows two hands next to each other, the middle picture shows two hands that are partially crossed together, and the right picture shows two hands that are almost completely overlapped. Of course, other separate target objects may also be included in addition to the contacting target objects, which is not limited here.

In step S204, target detection is performed on the image to be detected, and a position frame of each target object is obtained.

In some optional embodiments, target detection may be performed on the image to be detected based on a neural network model such as a CNN network model (Convolutional Neural Networks, convolutional neural network), an R-CNN (Region-CNN) network model, or a Segnet network model, so as to obtain The location box of each target object in the image to be detected.

Step S206 , merging the position frames based on the position frames of each target object to obtain a target merging frame; wherein, the image corresponding to the target merging frame includes at least two contacting target objects.

At least two target objects that are in contact with each other will partially overlap, and the position frames of each target object will also overlap each other. The position frames of the at least two target objects that overlap each other are merged to obtain a target merged frame. It can be understood that for the target merging frame obtained based on the overlapping position frames, the number of target objects included in the corresponding image will not be more than the number of contacting target objects in the image to be detected. Small target objects may not be within the range of the corresponding image of the target merge box.

Step S208 , perform key point detection on the image to be detected based on the target merging frame, and obtain multiple heat maps for each target object; wherein, different heat maps are used to represent key points located at different positions on the target object.

In this embodiment, performing key point detection on the image to be detected based on the target merging frame can be understood as: the image at the target merging frame in the image to be detected uses a bottom-up manner to detect the key points of each target object, and obtains the target merging frame. Heatmap for each target object in the box. A heatmap is an illustration that displays key points in a special form such as highlighting or coloring. Among them, in the process of obtaining the heat map of each target object based on the target merging frame, a corresponding heat map can be generated for each key point on the current target object, that is, each heat map only reflects one key point. For example, if the target object is a hand, there are 21 key points conventionally used to locate the hand, and the obtained heatmap corresponding to each hand is 21, and different heatmaps correspond to the key points at different positions of the hand.

Step S210, determining the key points of the contacting target objects based on the heat map. The position coordinates of the key points represented by each heat map can be obtained first; then according to the preset mapping relationship between the heat map and the target object, the original position coordinates of the position coordinates of the key points in the heat map in the target object are determined; Finally, the key points of the contacting target objects are determined according to the original position coordinates of each key point.

A key point detection method provided by an embodiment of the present invention can obtain the position frame of each target object by first performing target detection on the image to be detected, and then combine the position frames based on the position frame of each target object to obtain the target merged frame , and the image corresponding to the target merging frame includes at least two target objects that are in contact; then, based on the target merging frame, the key points of the image to be detected are detected, and multiple heatmaps of each target object are obtained; among them, different heatmaps It is used to characterize the key points located at different positions on the target object; finally, the key points of each target object in contact are determined based on the heat map. This embodiment can detect key points based on target merging frames corresponding to at least two contacting target objects, and obtain multiple heatmaps that can represent keypoints at different positions on the contacting target objects, so as to determine based on the heatmaps The key points of the target object, this method can effectively improve the problem of inaccurate key point detection caused by the contact interaction between the target objects, and improve the detection accuracy of the key points.

When performing the above step S206, the following two position frame merging methods can be used to obtain the target merged frame according to the actual detection scene:

Merging method 1: Based on the position frame of each target object, the preset merge operation is repeatedly performed on the position frame until the overlap between any two position frames is not greater than the preset overlap threshold, and the target merge frame is obtained. This merging method 1 is suitable for relatively simple detection scenarios such as only one target position frame is obtained, or more than two target position frames are obtained and the number of target objects contained in each target position frame is the same.

For a more complex detection scene, for example, there are multiple groups of contacting target objects in an image to be detected, including not only two contacting target objects A and B, but also three contacting target objects C, D and E. , and C, D, E and A, B are spaced apart. In this scenario, the following merging method 2 can be used to obtain the target merged frame.

Merging method 2: First, based on the position frame of each target object, the preset merge operation is repeatedly performed on the position frame, until the overlap between any two position frames is not greater than the preset overlap threshold, and a candidate merge frame is obtained; The number of candidate merging frames is at least two, and the number of target objects included in the candidate merging frames is not the same. Then, according to the number of target objects contained in the candidate merging frame, the target merging frame is obtained by filtering from the candidate merging frame. In this embodiment, the candidate merging frames can be classified according to the number of included target objects, the number of target objects contained in the same type of candidate merging frames is the same, and the same type of candidate merging frames can be determined as one type of target merging frames; According to the actual number of detected target objects (for example, two target objects are specified), the candidate merged frame containing the required number of target objects is determined as the target merged frame.

By adopting the above-mentioned merging method 2, when step S208 is executed, it is convenient to select a key point detection method that matches the target merging frame. Different keypoint detection models are selected according to the different numbers of target objects included, so that the selected keypoint detection model has a high degree of matching with the keypoint detection task to be performed, thereby improving the accuracy of the detection results.

In order to facilitate the understanding of the above two merging methods, this embodiment describes possible implementations of the merging operation in the above merging methods. Referring to the flowchart of a method for performing a merging operation multiple times as shown in FIG. 4 , the above merging operation includes the following Steps S402 to S406:

Step S402: Determine the position frame pair to be merged from multiple position frames; wherein, the position frame pair includes two position frames, and when the merge operation is performed in the first round, the position frame pair includes two target objects. location box. Each two position frames in the multiple position frames are determined as a group of position frame pairs to be merged. Pairs include {ab}, {ac}, {ad}, {bc}, {bd}, and {cd}.

Step S404: Calculate the degree of overlap between the position frames centered on the position frames. For each of the above-mentioned position frame pairs, the ratio between the area of the overlapping portion between the two position frames in the position frame pair and the total area of the two position frames is calculated to obtain the degree of overlap of the two position frames.

Step S406, if the calculated overlapping degree is greater than the preset overlapping degree threshold, merge the position frame in the position frame pair into a new position frame; wherein, the boundary of the new position frame is based on the two position frames in the position frame pair. Boundaries are determined. For example, if the overlapping degree of the position frame a and the position frame b is greater than a preset overlapping degree threshold (eg 70%), the position frame a and the position frame b are merged to generate a new position frame. The new position frame can refer to the position frame merging schematic diagram shown in FIG. 5 . The boundary of the new position frame is obtained by extending and intersecting the borders of the non-overlapping parts of the position frame a and the position frame b respectively.

It can be understood that, after obtaining a new position frame by performing the above step S406, return to the above step S402, and continue to determine the pair of position frames to be merged from the new position frame and other position frames, that is, perform the above step S402 again on the position frame. Go to step S406, until the degree of overlap between any two position frames is not greater than the preset overlap degree threshold, to obtain a candidate merged frame or directly obtain a target merged frame.

In practical applications, some position boxes may not be positioned accurately, causing the position boxes to deviate from the target object. Merging these position boxes not only affects the merging efficiency but also affects the accuracy of the position box merging results. Based on this, when performing the above step S402 to determine the position frame pair to be merged from multiple position frames, the confidence of the position frame can be further considered to improve the above problem, and the specific reference is as follows:

First, get the confidence of the location box. The confidence of the position frame of each target object can be generated when the target detection is performed on the image to be detected by the neural network model.

Then, the position boxes are sorted according to the confidence of each position box to obtain the ranking result of the position boxes. When sorting the position boxes according to the confidence level from high to low, the confidence of the position boxes at the back of the ranking is lower, indicating that these position boxes may deviate from the target, and their reliability is poor. Based on this, some position boxes can be filtered out. During specific implementation, some position frames with lower confidence may be filtered out based on the preset position frame confidence threshold; alternatively, the position frames after the specified ranking may also be filtered out from the original sorting result of the position frames; thus, according to the filtered The location box determines the location box sorting results. It is assumed that the ranking result of the position boxes is the position frame a, the position frame b, the position frame c and the position frame d arranged from high to low confidence. This embodiment sorts the location boxes based on the confidence of the location boxes, which is beneficial to reduce the merging of the location boxes with lower confidence when the location boxes are subsequently merged, thereby effectively improving the efficiency and accuracy of the location box merging.

Finally, the position box pair to be merged is determined from the plurality of position boxes according to the position box sorting result; wherein, the position box pair includes two position boxes. The above position frame sorting results can reflect the accuracy and reliability of each position frame. Therefore, according to the position frame sorting results, on the one hand, the position frame pairs suitable for priority merging can be determined. Pairs of position boxes to be merged to avoid missing position boxes. Referring to the above position box sorting results as position box a, position box b, position box c, and position box d, the position box pairs to be merged in this embodiment may include {ab}, {ac}, {ad}, {bc} , {bd}, and {cd}.

On the basis of the above-mentioned determination of the position frame pair based on the confidence, this embodiment can also provide another possible implementation manner of the merging operation. Referring to the flowchart of another method for performing the merging operation multiple times as shown in FIG. 6 , The above-mentioned merging operation includes the following steps S602 to S610:

In step S602, the confidence level of the position frame is obtained.

Step S604: Rank the position boxes according to the confidence of each position box, and obtain a ranking result of the position boxes.

Step S606: Determine the pair of position frames to be merged from the plurality of position frames according to the position frame sorting result.

Step S608: Calculate the degree of overlap between the position frames centered on the position frames.

Step S610, if the calculated overlap degree is greater than the preset overlap degree threshold, merge the position frames in the position frame pair into a new position frame.

It can be understood that, after obtaining a new position frame by executing the above step S610, return to the above step S602, that is, the above steps S602 to S610 are re-executed for the position frame until the degree of overlap between any two position frames is not greater than Preset the overlap threshold to obtain candidate merged frames or directly obtain target merged frames.

Wherein, when a new round of step S602 is performed, the confidence of the new position frame can be obtained in the following manner: the confidence of the two position frames in the position frame is obtained, and the confidence of the two position frames can be obtained according to the confidence of the two position frames. Get the confidence of the new location box.

Considering that the ranking result of the position boxes based on the confidence can be used to determine the pair of position boxes suitable for preferential merging, therefore, in this embodiment, the confidence of the two position boxes can be determined as the high confidence of the new position box. . Of course, there may also be other ways of determining the confidence of the new position box, for example, determining the confidence of the two position boxes with a lower confidence level as the confidence of the new position box, or, in two positions One of the confidence levels of the box is randomly selected as the confidence level of the new position box, or the average value of the confidence levels of the two position boxes is determined as the confidence level of the new position box.

In order to reduce the key point detection pressure of the image to be detected, in this embodiment, when performing key point detection on the image to be detected based on the target merging frame in step S208, the image to be detected may be processed first, that is, refer to the following content: from the image to be detected based on the target merging frame The partial image is extracted from the middle; the partial image contains the contacting target objects. Specifically, the position parameters of the target merging frame in the image to be detected are obtained, and the position parameters may include the coordinates (x, y) of the upper left vertex of the target position frame and the height and width of the target merging frame; using tools such as OpenCv (Open Source Computer Vision Library) , the CvRect function in the open source computer vision library) operates on the position parameters of the target merging frame, and outputs an image determined by the position parameters, which is a partial image extracted from the image to be detected.

Usually the size of the extracted local images is different. In order to adapt to various key point detection scenarios, the size of the local images can be adjusted, and the key points of the resized local images can be detected to obtain multiple images of each target object. Heatmap. In practical applications, the size (including the height and width) of the partial image can be directly reset to the target size (including the target height and the target width), and the size adjustment method is simple and efficient. Alternatively, in order to ensure the original size ratio of the partial image, the height of the partial image can be adjusted to the target height h, and when the width of the partial image is insufficient, the width of the partial image can be adjusted to the target width by filling with 0, so that The adjusted target height h and target width still satisfy the original size ratio.

Through the above steps of extracting the local image and adjusting the length and width of the local image by resetting or filling, the model for detecting key points only needs to put the learning ability in the fixed-size frame when learning key point detection. The target object (such as the hand) can be placed on the local image, thereby effectively reducing the learning pressure of the model to detect key points. Based on this, this embodiment provides an example of performing keypoint detection on the resized local image to obtain multiple heatmaps for each target object: performing keypoint detection on the resized local image through the trained detection model , to get multiple heatmaps for each target object.

The detection model can be a key point detection model based on a convolutional neural network, and its input is a local image of size (h, w), and it is assumed that the number of contacting target objects contained in the local image is two ; The output of the detection model is 2*n heatmaps of size (h/s, w/s); each heatmap represents the confidence of a key point, and the position with the highest confidence in the heatmap is the corresponding key point location. Among them, s is the downsampling rate of the heat map relative to the local image, and the downsampling rate is coupled to the network structure design of the detection model; n is the number of key points of a single target object, and the number of key points of each target object is the same; 2*n The first heat maps represent the key points of the two target objects (such as the left and right hands), the first n heat maps represent the key points of the first target object (such as the left hand), and the last n heat maps represent the key points of the first target object (such as the left hand). The key points of the second target object (such as the right hand); and the key represented by the n heatmaps of each target object is fixed. Since the number of heat maps output by the detection model provided in this embodiment is twice the number of key points in a single target object, the detection model can be called a dual-channel detection model. Of course, it can be understood that when the number of contacting target objects contained in the local image is M (M=3, 4, 5...), the number of heat maps output by the corresponding trained detection model is The number of key points in a single target object is M times, and the detection model at this time can be called an M-channel detection model.

For ease of understanding, the hand can be used as the target object to describe the key points of the heat map and its characterization; referring to the schematic diagram of the hand heat map shown in Figure 7, the first heat map of one hand (right hand) can be represents the position information of the key point of the thumb fingertip, then the position information of the key point of the thumb fingertip of the other hand (left hand) corresponding to the n+1th heat map is identified. Correspondingly, the second heat map represents the position information of the key points of the index finger of the right hand, and the n+2th heat map represents the position information of the key points of the index finger of the left hand.

The step of determining the key points of the contacting target objects based on the heat map may include the following steps 1) to 3):

Step 1): Calculate the brightness value of each pixel in the heat map; wherein, the brightness value is used to represent the confidence of the corresponding key point in the heat map. The essence of the heatmap is a two-dimensional matrix of h*w. The larger the value of the corresponding position of the matrix, the higher the brightness value of the pixel in the corresponding heatmap. Use the following formula (1) to find the position coordinates of the pixel with the largest brightness value in the two-dimensional matrix:

(x,y)=argmax(hx,hy) (1)

Among them, argmax (hx, hy) means to traverse the heat map from the reference coordinates (hx, hy) of the heat map, so as to determine the position coordinates (x, y) of the pixel with the maximum brightness value.

Step 2), filter the heat map according to the preset key point brightness threshold and the obtained maximum brightness value with reference to the following formula (2); that is, if the position coordinates (x, y) of the pixel with the maximum brightness value are not If formula (2) is satisfied, the heat map to which the pixel belongs is filtered out.

heatmap[x,y]>conf_thresh (2)

Among them, heatmap[x, y] represents the obtained maximum brightness value, that is, the brightness value of the position coordinate (x, y), and conf_thresh is the preset key point brightness threshold. If the brightness value of the position coordinate (x, y) is lower than the preset brightness threshold, it indicates that the brighter pixels in the heatmap are caused by interference factors such as noise, and are not the key points of the target object.

Step 3): Determine the key points of the contacting target objects according to the filtered heat map.

For each filtered heatmap, you can first calculate the coordinates of the key points in the heatmap by using algorithms such as the soft-argmax function, and then according to the preset mapping relationship between the heatmap and the target object, convert the calculated heatmap. The coordinates of the target object are converted into the original position coordinates of the target object; finally, the key points of each target object are determined according to the original position coordinates of all the converted key points.

After the key points of each target object are determined, posture restoration is performed on all the key points of the target object, so as to identify the posture, expression and other contents of the target object according to the restoration result.

In addition, the training process of the above detection model can refer to the following four steps:

First, input multiple training images marked with key point positions of target objects to the detection model to be trained, wherein the training images include at least two target objects that are in contact, and the degree of overlap between any two target objects is reach the preset overlap threshold.

Second, the training image is detected by the detection model to be trained, and the heat map of each target object in the training image is output.

Third, based on the heat map of each target object in the training image, the position of the key point in each target object in the training image is obtained.

Fourth, optimize the parameters of the detection model to be trained based on the positions of the key points obtained by the detection model to be trained and the positions of the marked key points, until the position of the key points obtained by the detection model to be trained and the positions of the marked key points are between When the matching degree reaches the preset matching degree, it is determined that the training is over, and the trained detection model is obtained.

To sum up, the above-mentioned embodiments can detect key points based on the target merging frames corresponding to at least two contacting target objects, and obtain multiple heat maps that can represent key points at different positions on the contacting target objects, so that based on the The heatmap determines the key points of the target object, which can effectively improve the problem of inaccurate key point detection caused by the contact interaction between the target objects, and improve the detection accuracy of key points.

Embodiment three:

For the key point detection method provided in the second embodiment, an embodiment of the present invention provides a gesture recognition method, including the following steps 1 and 2:

Step 1. Use the key point detection method provided in the second embodiment to perform key point detection on the hand image to be detected, and obtain the key points of each hand. For a brief description, for the step of obtaining the key points of the hand, reference may be made to the corresponding content in the foregoing method embodiment 2.

Step 2: Identify the gesture category according to the key points of each hand. In a specific implementation manner, for each hand, gesture restoration can be performed on all key points of the hand, so as to identify gesture categories such as shaking hands, clenching fists, grasping, etc. of the hand according to the restoration results.

The gesture recognition method provided in this embodiment can first obtain the key points of each hand by using the key point detection method provided in the second embodiment, and then recognize the gesture category according to the key points of each hand. This embodiment can improve the detection accuracy of key points based on the key point detection method, thereby effectively improving the accuracy of recognizing gesture categories.

Embodiment 4:

For the key point detection method provided in the second embodiment, an embodiment of the present invention provides a key point detection device. Referring to the structural block diagram of a key point detection device shown in FIG. 8 , the device includes the following modules:

The image acquisition module 802 is configured to acquire an image to be detected, and the to-be-detected image contains a target object.

The target detection module 804 is configured to perform target detection on the image to be detected to obtain the position frame of each target object.

The position frame merging module 806 is configured to merge the position frames based on the position frames of each target object to obtain the target merged frame; wherein, the image corresponding to the target merged frame includes at least two contacting target objects.

The keypoint detection module 808 is used to perform keypoint detection on the image to be detected based on the target merged frame, and obtain multiple heatmaps of each target object; wherein, different heatmaps are used to represent keypoints located at different positions on the target object .

The key point determination module 810 is configured to determine the key points of the contacting target objects based on the heat map.

A key point detection device provided by an embodiment of the present invention can obtain a position frame of each target object by first performing target detection on an image to be detected, and then combine the position frames based on the position frame of each target object to obtain a target merged frame , and the image corresponding to the target merging frame includes at least two target objects that are in contact; then, based on the target merging frame, the key points of the image to be detected are detected, and multiple heatmaps of each target object are obtained; among them, different heatmaps It is used to characterize the key points located at different positions on the target object; finally, the key points of each target object in contact are determined based on the heat map. This embodiment can detect key points based on target merging frames corresponding to at least two contacting target objects, and obtain multiple heatmaps that can represent keypoints at different positions on the contacting target objects, so as to determine based on the heatmaps The key points of the target object, this method can effectively improve the problem of inaccurate key point detection caused by the contact interaction between the target objects, and improve the detection accuracy of the key points.

In some embodiments, the above-mentioned position frame merging module 806 is further configured to: based on the position frame of each target object, repeatedly perform a preset merging operation on the position frames, until the degree of overlap between any two position frames is not greater than Preset the overlap threshold to obtain the target merged frame.

In some embodiments, the above-mentioned position frame merging module 806 is further configured to: based on the position frame of each target object, repeatedly perform a preset merging operation on the position frames, until the degree of overlap between any two position frames is not greater than A threshold of the overlap degree is preset to obtain a candidate merging frame; according to the number of target objects contained in the candidate merging frame, a target merging frame is obtained by screening the candidate merging frame.

In some embodiments, the above-mentioned merging operation includes a sub-operation of determining a position frame pair and a sub-operation of merging a new position frame, wherein the sub-operation of determining a position frame pair includes: determining a position frame pair to be merged from a plurality of position frames; The frame pair includes two position frames; the merge new position frame sub-operation includes: calculating the degree of overlap between the position frames in the position frame pair; if the calculated overlap degree is greater than the preset overlap degree threshold, merge the position frames in the position frame pair into a A new location box; wherein, the boundary of the new location box is determined according to the boundaries of the two location boxes in the pair of location boxes.

In some embodiments, the above-mentioned sub-operation of determining the position frame pair further includes: obtaining the confidence of the position frames; sorting the position frames according to the confidence of each position frame to obtain a ranking result of the position frames; The location box determines the pair of location boxes to be merged.

In some implementation manners, the above sub-operation of determining the position frame pair further includes: obtaining the confidence levels of two position frames in the position frame pair, and obtaining the confidence level of the new position frame according to the confidence levels of the two position frames.

In some embodiments, the above-mentioned key point detection module 808 is further configured to: extract a partial image from the image to be detected based on the target merging frame; the partial image contains the contacting target objects; adjust the size of the partial image, and Keypoint detection is performed on the resized local image to obtain multiple heatmaps for each target object.

In some embodiments, the above-mentioned key point detection module 808 is further configured to: perform key point detection on the local image after resizing by using the trained detection model to obtain multiple heat maps of each target object.

In some embodiments, the above-mentioned key point detection apparatus further includes a model training module (not shown in the figure), the model training module is used for: inputting a plurality of key point positions marked with the target object into the detection model to be trained for training image, wherein the training image includes at least two target objects in contact, and the overlap between any two target objects reaches a preset overlap threshold; the training image is detected by the detection model to be trained, and the training output is output. The heat map of each target object in the image; the key point position of each target object in the training image is obtained based on the heat map of each target object in the training image; the key point position and the marked key point position obtained based on the detection model to be trained The parameters of the detection model to be trained are optimized until the matching degree between the key point positions obtained by the detection model to be trained and the marked key point positions reaches the preset matching degree, and the training is determined to be completed, and the trained detection model is obtained.

In some embodiments, the above-mentioned key point determination module 810 is further configured to: obtain the brightness value of each pixel point in the heat map; wherein, the brightness value is used to represent the confidence of the corresponding key point in the heat map; according to the preset key point The heat map is filtered by the brightness threshold and the obtained maximum brightness value; the key points of each contacting target object are determined according to the filtered heat map.

The implementation principle and technical effect of the device provided in this embodiment are the same as those in the second embodiment. For brief description, for the parts not mentioned in this embodiment, reference may be made to the corresponding content in the second method embodiment.

Embodiment 5:

For the gesture recognition method provided in the third embodiment, an embodiment of the present invention provides a gesture recognition device, and the device includes:

The hand key point detection module is used to perform key point detection on the hand image to be detected by using the key point detection method provided in the second embodiment to obtain the key points of each hand.

The gesture recognition module is used to recognize gesture categories according to the key points of each hand.

Embodiment 6:

Based on the foregoing embodiments, this embodiment provides a key point detection system, which includes: an image acquisition device, a processor, and a storage device; wherein, the image acquisition device is used to acquire images to be detected; the storage device stores a computer The program, when the computer program is run by the processor, executes any one of the key point detection methods provided in the second embodiment.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here.

Further, this embodiment also provides an electronic device, including a memory and a processor, wherein a computer program that can be run on the processor is stored in the memory, and when the processor executes the computer program, any one of the keys provided in the second embodiment above is implemented The steps of the point detection method or the steps of the gesture recognition method provided in the third embodiment.

Further, this embodiment also provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by the processing device, the steps of any one of the key point detection methods provided in the second embodiment above are executed. Or the steps of the gesture recognition method provided in the third embodiment.

A computer program product of a key point detection method, a gesture recognition method, an apparatus, and a system provided by the embodiments of the present invention includes a computer-readable storage medium storing program codes, and the instructions included in the program codes can be used to execute the foregoing methods. For the specific implementation of the method described in the embodiment, reference may be made to the method embodiment, which will not be repeated here.

The functions, if implemented in the form of software functional units and sold or used as independent products, may be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention can be embodied in the form of a software product in essence, or the part that contributes to the prior art or the part of the technical solution. The computer software product is stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present invention, and are used to illustrate the technical solutions of the present invention, but not to limit them. The protection scope of the present invention is not limited thereto, although referring to the foregoing The embodiment has been described in detail the present invention, those of ordinary skill in the art should understand: any person skilled in the art who is familiar with the technical field within the technical scope disclosed by the present invention can still modify the technical solutions described in the foregoing embodiments. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be covered in the present invention. within the scope of protection. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims (16)

1. a kind of critical point detection method, which is characterized in that the described method includes:

Image to be detected is obtained, includes target object in described image to be detected；

Target detection is carried out to described image to be detected, obtains the position frame of each target object；

Position frame based on each target object carries out the merging of position frame, obtains target and merges frame；Wherein, the target is closed It and include at least two target objects being in contact in the corresponding image of frame；

Merge frame based on the target and critical point detection is carried out to described image to be detected, obtains the more of each target object A thermodynamic chart；Wherein, the different thermodynamic charts is used to characterize the key point being located on the target object at different location；

The key point for each target object being in contact is determined based on the thermodynamic chart.

2. the method according to claim 1, wherein the position frame based on each target object carries out Position frame merges, and obtains the step of target merges frame, comprising:

Based on the position frame of each target object, preset union operation is repeated to position frame, until any two Degree of overlapping between the frame of position obtains target and merges frame no more than default degree of overlapping threshold value.

3. the method according to claim 1, wherein the position frame based on each target object carries out Position frame merges, and obtains the step of target merges frame, comprising:

Based on the position frame of each target object, preset union operation is repeated to position frame, until any two Degree of overlapping between the frame of position obtains candidate merging frame no more than default degree of overlapping threshold value；

According to the candidate quantity for merging target object included in frame, screening obtains target from the candidate merging frame Merge frame.

4. according to the method in claim 2 or 3, which is characterized in that the union operation includes:

Position frame pair to be combined is determined from multiple position frames；Wherein, the position frame is to including two position frames；

Calculate the degree of overlapping between the position frame center position frame；

If the degree of overlapping calculated is greater than default degree of overlapping threshold value, the position frame of the position frame centering is merged into new position Frame；Wherein, the boundary of the new position frame is to be determined according to the boundary of two position frames of the position frame centering.

5. according to the method described in claim 4, it is characterized in that, described determine position frame to be combined from multiple position frames Pair step, comprising:

Obtain the confidence level of position frame；

The position frame is ranked up according to the confidence level of each position frame, obtains position frame ranking results；

Position frame pair to be combined is determined from multiple position frames according to the position frame ranking results.

6. according to the method described in claim 5, it is characterized in that, it is described obtain position frame confidence level the step of, comprising:

The confidence level for obtaining two position frames of the position frame centering obtains described new according to the confidence level of described two position frames Position frame confidence level.

7. the method according to claim 1, wherein described merge frame to the mapping to be checked based on the target As the step of carrying out critical point detection, obtaining multiple thermodynamic charts of each target object, comprising:

Merge frame based on the target and takes topography from described image to be detected；It include to connect in the topography The target object of touching；

Size adjusting is carried out to the topography, and critical point detection is carried out to the topography after size adjusting, is obtained Multiple thermodynamic charts of each target object.

8. the method according to the description of claim 7 is characterized in that the topography to after size adjusting carries out key The step of point detects, and obtains multiple thermodynamic charts of each target object, comprising:

Critical point detection is carried out to the topography after size adjusting by the detection model after training, obtains each mesh Mark multiple thermodynamic charts of object.

9. according to the method described in claim 8, it is characterized in that, the method also includes:

The training image for the key point position that multiple are labeled with target object is inputted to detection model to be trained, wherein described It include at least two target objects being in contact in training image, and the degree of overlapping between target object described in any two reaches To default degree of overlapping threshold value；

The training image is detected by the detection model to be trained, exports each mesh in the training image Mark the thermodynamic chart of object；

Thermodynamic chart based on the target object each in the training image obtains in the training image in each target object Key point position；

The key point position obtained based on the detection model to be trained and the key point position marked to it is described to Trained detection model carries out parameter optimization, until the obtained key point position of detection model to be trained and having marked Matching degree between key point position reaches preset matching when spending, and determines that training terminates, the detection model after being trained.

10. the method according to claim 1, wherein described determine each institute being in contact based on the thermodynamic chart The step of stating the key point of target object, comprising:

Obtain the brightness value of each pixel in the thermodynamic chart；Wherein, the brightness value is right in the thermodynamic chart for characterizing Answer the confidence level of key point；

The thermodynamic chart is filtered according to preset key point luminance threshold and the maximum brightness value got；

The key point for each target object being in contact is determined according to filtered thermodynamic chart.

11. a kind of gesture identification method, which is characterized in that the described method includes:

Key point is carried out to hand images to be detected using critical point detection method as described in any one of claim 1 to 10 Detection, obtains the key point of each hand；

Gesture classification is identified according to the key point of each hand.

12. a kind of critical point detection device, which is characterized in that described device includes:

Image collection module includes target object in described image to be detected for obtaining image to be detected；

Module of target detection obtains the position of each target object for carrying out target detection to described image to be detected Frame；

Position frame merging module carries out the merging of position frame for the position frame based on each target object, obtains target conjunction And frame；Wherein, it includes at least two target objects being in contact that the target, which merges in the corresponding image of frame,；

Critical point detection module carries out critical point detection to described image to be detected for merging frame based on the target, obtains Multiple thermodynamic charts of each target object；Wherein, the different thermodynamic charts is located at the target object for characterizing Key point at different location；

Key point determining module, for determining the key point for each target object being in contact based on the thermodynamic chart.

13. a kind of gesture identifying device, which is characterized in that described device includes:

Hand critical point detection module, for being treated using critical point detection method as described in any one of claim 1 to 10 The hand images of detection carry out critical point detection, obtain the key point of each hand；

Gesture recognition module, for identifying gesture classification according to the key point of each hand.

14. a kind of critical point detection system, which is characterized in that the system comprises: image collecting device, processor and storage dress It sets；

Described image acquisition device, for acquiring image to be detected；

Computer program is stored on the storage device, the computer program is executed when being run by the processor as weighed Benefit requires 1 to 10 described in any item critical point detection methods or gesture identification method as claimed in claim 11.

15. a kind of electronic equipment, including memory, processor, it is stored with and can runs on the processor in the memory Computer program, which is characterized in that processor execute computer program when realize 1 to 10 any one of the claims described in Critical point detection method the step of or gesture identification method as claimed in claim 11.

16. a kind of computer readable storage medium, computer program, feature are stored on the computer readable storage medium It is, the described in any item critical point detections of the claims 1 to 10 is executed when the computer program is run by processor The step of method or gesture identification method as claimed in claim 11.