TW202303451A - Nail recognation methods, apparatuses, devices and storage media - Google Patents

Abstract

Nail recognition methods, apparatuses, devices and storage media are disclosed. The method comprises: obtaining a detection result of at least one nail in a first image, where the detection result comprises a first nail detection frame and a classification result of the nail, where the classification result indicates a finger type to which the nail belongs; obtaining an image area corresponding to the nail in the first image according to the first nail detection frame; obtaining a plurality of first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs.

Description

Nail recognition method, device, equipment and storage medium

The present application relates to the technical field of image recognition, and in particular to a nail recognition method, device, equipment and storage medium.

Nail recognition has important application prospects in mobile entertainment, virtual fitting, virtual reality VR, augmented reality AR and other fields. In related technologies, the segmentation model (segmentation) is usually used to obtain the nail region in the image or video. However, this method cannot obtain the semantic information of the nail outline, which limits the use of the recognition results in various application scenarios.

An embodiment of the present disclosure provides a nail recognition solution.

According to an aspect of the present disclosure, a nail recognition method is provided, the method comprising: acquiring a detection result of at least one nail in a first image, the detection result including a first nail detection frame and a classification result of the nail, the The classification result indicates the finger type to which the nail belongs; the image area corresponding to the nail in the first image is obtained according to the first nail detection frame; the image area corresponding to the nail is obtained according to the finger type to which the nail belongs Multiple first keypoints for the nails described in .

By obtaining the classification results of nails and obtaining the first key point corresponding to the corresponding nails according to the type of finger to which the nails belong, the category information of the nails and the complete semantic features of the nail outline can be obtained, which facilitates the application of the nail recognition results in various scenarios.

In combination with any of the implementations provided in the present disclosure, the obtaining multiple first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs includes: from the first image Cut out the image area corresponding to the nail; input the cropped image area into the first key point detection network corresponding to the finger type to which the nail belongs to obtain multiple first key points of the nail.

By cutting out the image area corresponding to the nail, and performing nail key point detection according to the clipped image, the efficiency and accuracy of nail key point detection can be improved.

In combination with any embodiment provided in the present disclosure, the method further includes: acquiring a binary classification result of each pixel in the image region corresponding to the nail, the binary classification result indicating that the pixel is a foreground pixel or a background pixel; A pixel indicated as a background pixel in the above binary classification result is set as the first pixel value.

By performing a filtering operation on background pixels in the image area corresponding to the nail or in the cropped image, only the area corresponding to the real nail is retained, which can reduce the probability of false detection of nail key points.

In combination with any of the implementations provided in the present disclosure, the method further includes: according to the position information of at least two first key points of the plurality of first key points of the nail in the image area, determining the direction.

By obtaining the direction of the nail, it is convenient to apply the nail recognition result in various scenarios. For example, in the scene of adding nail art effects to the nails in the image, it is more convenient to add nail art effects to the nails when the direction of the nails is known. .

In combination with any implementation manner provided by the present disclosure, the method further includes: acquiring a sample image; wherein, the sample image has annotation information, and the annotation information indicates a first key point corresponding to the finger type to which the sample image belongs; inputting the sample image into the first key point detection network to obtain a key point detection result; according to the difference between the key point detection result and the annotation information, the first key point detection network Adjust the network parameters.

By annotating the nail information according to the finger type of the nail in the sample image, and using the sample image to train the first key point detection network, the first key point detection network can realize the recognition of the nail of the specified finger type .

In combination with any implementation manner provided by the present disclosure, the first image is a frame in an image sequence, and the method further includes: for a second image following the first image, according to a previous frame of the second image A plurality of first key points of the nail in the frame, determine a second nail detection frame in the second image; obtain, in the image area corresponding to the second nail detection frame in the second image, the nail's Multiple second keys.

By obtaining the key point detection result in the current frame according to the key point detection result of the previous frame in the image sequence, the amount of data processing can be reduced, and the speed and efficiency of nail key point detection can be improved.

In combination with any implementation manner provided by the present disclosure, the determining the second nail detection frame in the second image according to the multiple first key points in the previous frame of the second image includes: according to the previous frame A plurality of first key points of the nail in the frame to obtain a circumscribing rectangle of the nail; according to the position information of the circumscribing rectangle in the previous frame, map the circumscribing rectangle to the In the second image, as the second nail detection frame in the second image.

Compared with the first nail detection frame detected by the nail, the second nail detection frame in the second image obtained by the above method is closer to the real area of the nail and contains less parts other than the nail, which is beneficial to improve the key Accuracy of point detection.

In combination with any implementation manner provided by the present disclosure, the obtaining multiple second key points of the nail in the image area corresponding to the second nail detection frame in the second image includes: cutting out the The image area corresponding to the second nail detection frame in the second image; input the cropped image area to the second key point detection network to obtain the second key point of the nail.

The second key point detection network can detect nail key points based on regression key points. Compared with the first key point detection network based on heat map for nail key point detection, the network structure is simpler, the number of layers is smaller, and the processing The speed is faster and the time-consuming for nail key point detection is reduced.

In combination with any of the implementations provided by the present disclosure, before inputting the cropped image region into the second key point detection network, according to the direction of the nail in the previous frame, the cropped image is Perform rotation.

The nail key point detection on the rotated image can improve the efficiency of detection on the one hand and the accuracy of detection on the other hand.

In combination with any implementation manner provided by the present disclosure, the method further includes: when the second key point of the nail is not detected or the second key point of the nail does not meet the set requirements, acquiring the second key point in the second image A detection result of at least one nail, the detection result comprising a first nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; the second nail detection frame is obtained according to the first nail detection frame An image area corresponding to the nail in the image; according to the finger type to which the nail belongs, a plurality of first key points of the nail in the image area corresponding to the nail are obtained.

By judging the second key point of the nail tracked, if it is not detected or does not meet the set requirements, the nail recognition will be performed again. On the one hand, it ensures the consistency of the nail recognition results, and on the other hand, it also ensures Accuracy of nail recognition results.

According to an aspect of the present disclosure, there is provided a nail recognition device, including: a first acquiring unit, configured to acquire a detection result of at least one nail in a first image, the detection result including a first nail detection frame and a classification of the nail As a result, the classification result indicates the finger type to which the nail belongs; the second acquisition unit is configured to obtain the image area corresponding to the nail in the first image according to the first nail detection frame; The finger type to which the nail belongs, and a plurality of first key points of the nail in the image area corresponding to the nail are obtained.

In combination with any of the implementations provided in the present disclosure, the identification unit is specifically configured to: crop out the image area corresponding to the nail from the first image; input the cropped image area into the In the first key point detection network, multiple first key points of the nail are obtained.

In combination with any embodiment provided in the present disclosure, the device further includes a filtering unit, configured to: obtain a binary classification result of each pixel in the image region corresponding to the nail, the binary classification result indicating that the pixel is a foreground pixel or Background pixels: setting the pixels indicated as background pixels in the binary classification result as the first pixel value.

In combination with any implementation manner provided by the present disclosure, the device further includes an orientation unit configured to: according to the position information of at least two first key points among the plurality of first key points of the nail in the image area , to determine the orientation of the nail.

In combination with any of the implementations provided in the present disclosure, the device further includes a training unit configured to: acquire a sample image; wherein, the sample image has annotation information, and the annotation information indicates the finger type corresponding to the sample image. The first key point; input the sample image to the first key point detection network to obtain a key point detection result; according to the difference between the key point detection result and the annotation information, the first key point detection The network parameters of the keypoint detection network are tuned.

In combination with any implementation manner provided in the present disclosure, the first image is a frame in an image sequence, and the device further includes a tracking unit configured to: for a second image after the first image, according to the first image A plurality of first key points of the nail in the previous frame of the second image, determine a second nail detection frame in the second image; obtain the image area corresponding to the second nail detection frame in the second image , a plurality of second key points of the nail.

In combination with any implementation manner provided by the present disclosure, when the tracking unit is used to determine the second nail detection frame in the second image based on the multiple first key points in the previous frame of the second image, specifically It is used to: obtain the circumscribed rectangular frame of the nail according to the multiple first key points of the nail in the previous frame; obtain the circumscribed rectangular frame according to the position information of the circumscribed rectangular frame in the previous frame. The circumscribed rectangular frame is mapped to the second image as a second nail detection frame in the second image.

In combination with any implementation manner provided by the present disclosure, when the tracking unit is used to obtain multiple second key points of the nail in the image area corresponding to the second nail detection frame in the second image, It is specifically used for: cutting out the image area corresponding to the second nail detection frame in the second image; inputting the cropped image area into the second key point detection network to obtain the second key point of the nail.

In combination with any of the implementations provided in the present disclosure, the device further includes a rotation unit, configured to, before inputting the cropped image region into the second key point detection network, according to the nail in the previous frame The direction of the cropped image is rotated.

In combination with any implementation manner provided by the present disclosure, the device further includes a judging unit configured to: if the second key point of the nail is not detected or the second key point of the nail does not meet the set requirements, Acquiring a detection result of at least one nail in the second image, the detection result including a first nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; according to the first nail detection frame Obtain an image area corresponding to the nail in the second image; and obtain a plurality of first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs.

According to an aspect of the present disclosure, there is provided an electronic device, the device includes a memory and a processor, the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions Implement the nail recognition method described in any implementation manner provided by the present disclosure.

According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the nail recognition method described in any implementation manner provided by the present disclosure is implemented.

According to an aspect of the present disclosure, a computer program product is provided, including a computer program, and when the program is executed by a processor, the nail recognition method described in any implementation manner provided in the present disclosure is implemented.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the specification.

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with this specification. Rather, they are merely examples of apparatus and methods consistent with aspects of the present specification, as detailed in the appended claims.

The terms used in this specification are for the purpose of describing particular embodiments only, and are not intended to limit the specification. As used in this specification and the appended claims, the singular forms "a", "the" and "the" are also intended to include the plural forms unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

It should be understood that although the terms first, second, third, etc. may be used in this specification to describe various pieces of information, these pieces of information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first information may also be called second information without departing from the scope of this specification, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

In related technologies, the recognition of the nail in the image still stops at the recognition of the nail area. The most commonly used method is to segment the model, that is, to detect each pixel in the image and combine the pixels belonging to the nail as Test results for nails. Although this nail detection method can identify the area belonging to the nail in the image, which finger, which hand the nail belongs to, and the orientation of the nail cannot be determined by the above-mentioned nail recognition method, which greatly Limits the use of recognition results in various application scenarios. For example, in scenarios such as mobile entertainment, virtual makeup trial, virtual reality VR, and augmented reality AR, users want to add directional special effects to nails, or for To add different special effects to different nails, it is necessary to be able to identify the type of nails or the orientation of the nails when identifying the nails.

In view of the above problems, at least one embodiment of the present disclosure provides a nail recognition method, which can be executed by electronic devices such as terminal devices or servers, and the terminal devices can be fixed terminals or mobile terminals, such as mobile phones, tablet computers, game machine, desktop machine, advertising machine, all-in-one machine, vehicle-mounted terminal, etc., the server includes a local server or a cloud server, etc., and the method can also use the method of calling the computer-readable instructions stored in the memory by the processor to fulfill.

FIG. 1 shows a flowchart of a nail recognition method according to at least one embodiment of the present disclosure. As shown in FIG. 1 , the method includes steps 101 to 104 .

In step 101, a detection result of at least one fingernail in a first image is acquired.

Wherein, the first image may be a still image or a video image shot in real time, or may be a still image or a video image acquired from a storage or other media. Moreover, the nails in the embodiments of the present disclosure may be the nails of the hand or the nails of the feet, which is not limited in the present disclosure. Taking the nail as a hand nail as an example, the first image may be a single hand image, or a human body image including the hand area; it may also be a partial hand including one or more nails image.

In an embodiment of the present disclosure, a nail detection network may be used to perform nail detection on the first image to obtain a detection result of at least one nail in the first image. Wherein, the nail detection network is a deep learning network, such as RCNN, Fast RCNN, Faster RCNN and the like. The detection result may include the first nail detection frame, the location information of the first nail detection frame, the classification result of the nail, etc., wherein the classification result indicates the finger type to which the nail belongs. The finger type to which the nail belongs indicates which finger the nail belongs to, or which finger of which hand the nail belongs to. For example, if the classification result indicates the index finger, it indicates that the nail belongs to the index finger; for another example, if the classification result indicates the left thumb, it indicates that the nail belongs to the left thumb.

In the case of using the nail detection network to perform multi-target detection on the first image, a classification result of the nails can be obtained while detecting the nails.

The nail detection network can be trained by using the following sample images: the sample images mark the bounding boxes of at least one nail contained in each, and mark the type of the nail in the bounding boxes.

In step 102, an image area corresponding to the nail in the first image is obtained according to the first nail detection frame. Wherein, the image area surrounded by the detection frame of the nail is the image area corresponding to the nail.

In step 103, according to the finger type to which the nail belongs, a plurality of first key points of the nail in the image area corresponding to the nail are obtained.

Wherein, multiple first key points of the nail are used to represent different position points of the nail outline, wherein each first key point is used to represent a specific position point of the nail. The outline of the nail has certain characteristics, and the outline of the nail can be outlined by specific position points. Therefore, when multiple first key points of the nail are detected, the outline of the nail can be determined to obtain edge information of the nail.

In general, one nail can correspond to any number of first key points within the range of 12 to 32. Taking the schematic diagram of the first key point shown in FIG. 2 as an example, one fingernail corresponds to 16 first key points. As shown in Figure 2, the first key point P1 is used to characterize the leftmost point at the bottom of the nail outline, the first key point P5 is used to characterize the rightmost point at the bottom of the nail outline, and the first key point P11 is represented by to characterize the top-most middle point of the nail outline. Those skilled in the art should understand that the nails shown in FIG. 2 corresponding to the 16 first key points are only for example, and the present disclosure is not limited thereto.

The number of first key points corresponding to nails of different fingers and the position of each first key point on the nail outline may be the same or different. Generally, the nail area of the thumb is larger than that of the little finger, and so is the outline length. Therefore, the nail of the thumb can correspond to a larger number of first key points, such as 32; relatively, the nail of the little finger can correspond to A smaller number of first key points, eg 12. Those skilled in the art should understand that the above-mentioned first key number is only an example, which is not limited by the embodiments of the present disclosure.

In the embodiment of the present disclosure, a corresponding first key point detection network may be constructed for the type of finger to which each nail belongs, for performing nail key point detection on nails belonging to the finger type. For example, when the classification result of the nail indicates that the nail belongs to the thumb of the left hand, the first key point detection network of the thumb of the left hand is called to detect key points of the nail in the image area corresponding to the nail, and the left hand thumb is obtained. Multiple first keys on the nail of the thumb.

In an embodiment of the present disclosure, a detection result of at least one nail in the first image is acquired, the detection result includes a first nail detection frame and a classification result of the nail, the classification result indicates the finger type to which the nail belongs; then Obtain the image area corresponding to the nail in the first image according to the first nail detection frame; and obtain a plurality of first images of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs. key point. By obtaining the classification results of nails and obtaining the first key point corresponding to the corresponding nails according to the type of finger to which the nails belong, the category information of the nails and the complete semantic features of the nail outline can be obtained, which facilitates the application of the nail recognition results in various scenarios.

In some implementations, the image area corresponding to the nail can be cropped first, and then the cropped image area can be input into the first key point detection network corresponding to the finger type to which the nail belongs, to obtain multiple the first key point.

By cutting out the image area corresponding to the nail, and performing nail key point detection according to the clipped image, the efficiency and accuracy of nail key point detection can be improved.

In some embodiments, the binary classification result of each pixel in the image area corresponding to the nail is obtained, and the binary classification result indicates that the pixel is a foreground pixel or a background pixel, the foreground pixel is the pixel corresponding to the nail area, and the background pixel is That is, pixels corresponding to areas other than the nail area. Next, the pixels indicated as background pixels in the binary classification result can be set as the first pixel value, wherein the first pixel value can be 0, or 255, or other values, and the first pixel value The value of the value is determined according to the setting of the background color. By setting the pixels indicated as background pixels as the first pixel value, the image area in the image area corresponding to the nail that is judged to be the background can be filtered out, so that only the real nail corresponding to the nail remains in the image area corresponding to the nail. Area.

In an example, for the cropped image, according to the binary classification result of each pixel in the image, the pixels indicated as background pixels may be set as the first pixel value.

By performing a filtering operation on background pixels in the image area corresponding to the nail or in the cropped image, only the area corresponding to the real nail is retained, which can reduce the probability of false detection of nail key points.

In some implementations, the direction of the nail can be determined according to position information of at least two first key points of the plurality of first key points of the nail in the image area.

Since each first key point of the nail represents a specific position point of the nail, the direction of the nail can be represented according to at least two first key points.

Generally, the direction indicated by the normal line of the nail, that is, the growth direction of the nail, can be determined as the direction of the nail. Generally, among the plurality of first key points of the nail, a line between the first key point located in the middle of the bottommost end of the nail outline and the first key point located in the middle of the topmost end forms the first key point of the nail. The normal, therefore, the direction of the nail can be determined by the direction indicated by the normal in the first image, or in the coordinate system of the image.

As shown in FIG. 2 , the line connecting the first key point P3 and P11 may be used as the normal line of the nail. In FIG. 2 , the normal line formed by P3 and P11 indicates the vertical direction of the first image, so it can be determined that the direction of the nail in FIG. 2 is the vertical direction of the first image.

The direction of the nail may also be determined according to other key points of the multiple key points of the nail, which is not limited in the present disclosure.

By obtaining the direction of the nail, it is convenient to apply the nail recognition result in various scenarios. For example, in the scene of adding nail art effects to the nails in the image, it is more convenient to add nail art effects to the nails when the direction of the nails is known. .

In some implementation manners, the first key point detection network can be trained by the following method.

Acquiring a sample image; wherein, the sample image has annotation information, and the annotation information indicates a first key point corresponding to the finger type to which the sample image belongs. The number of sample key points marked in the sample image can be any number within the range of 12 to 32. For example, the sample image includes the nail of the index finger, and 16 sample key points are marked on the edge of the nail of the index finger. Among them, each sample key point has a serial number, as shown in Figure 2, the point at the bottom and left of the nail contour is the key point of sample No. 1, denoted as P1, and the point at the bottom and right is the key point of sample No. 5 point, denoted as P5, and so on.

The sample image is input to the first key point detection network to obtain a key point detection result. The number of predicted first key points in the key point detection result is the same as the number of labeled sample key points, and the predicted first key points also have serial numbers.

According to the difference between the key point detection result and the annotation information, the network parameters of the first key point detection network are adjusted. That is, the network parameters of the first key point detection network are adjusted according to the difference between each sample key point and the corresponding predicted first key point. When the difference is less than the set threshold, or the iteration reaches the set number of times, the training is stopped, and the first key point detection network that has completed the training is obtained.

By labeling the nails according to the finger type in the sample image, and using the sample image to train the first key point detection network, the first key point detection network can recognize the nail of the specified finger type.

In an embodiment of the present disclosure, the first key point detection network may perform nail key point detection based on a heat map.

First, a heat map of each first key point in the first image is generated. The heat map of the first key point is a probability distribution map of possible locations of the first key point in the first image.

Furthermore, according to the heat map of the first key point, the coordinates of the first key point in the first image can be determined.

The position of the first key point of each nail in the first image can be accurately determined according to the key point heat map.

However, because the first key point detection network for nail key point detection based on the heat map has a complex structure, a large number of layers, and takes a long time, an embodiment of the present disclosure proposes a nail key point tracking method. When the first image is a video image, that is, the first image is a frame in an image sequence for the same scene, for any frame of image (second image) after the first image, The following methods can be used for nail key point tracking.

Firstly, a second nail detection frame in the second image is determined according to a plurality of first key points of the nail in the previous frame of the second image.

In the image sequence of the same scene, the position of the nail in two consecutive frames of images usually changes little, therefore, according to the multiple first key points of a nail in the previous frame, the nail in the current frame (second image) can be determined The second nail detection frame of .

In some implementation manners, the circumscribed rectangular frame of the nail can be obtained according to a plurality of first key points of the nail in the previous frame; according to the position of the circumscribed rectangular frame in the previous frame information, mapping the circumscribing rectangle to the second image, that is, placing the circumscribing rectangle at the same position in the second image as in the previous frame, as the first frame in the second image 2. Nail detection frame.

Compared with the first nail detection frame detected by the nail, the second nail detection frame in the second image obtained by the above method is closer to the real area of the nail and contains less parts other than the nail, which is beneficial to improve the key Accuracy of point detection.

In some embodiments, the image area corresponding to the second nail detection frame in the second image can be cropped; the cropped image area can be input to the second key point detection network to obtain the second nail detection frame. key point.

In the embodiment of the present disclosure, the function of the second key point detection network is the same as that of the first key point detection network, both of which can be used to detect nail key points from the input image; and the second key point The training method of the point detection network may also be the same as that of the first key point detection network. However, in the case that the second detection frame contains less parts other than nails, the coordinates of each second key point in the input image can be obtained based on key point regression.

Compared with the first key point detection network based on heat map for nail key point detection, the second key point detection network based on key point regression for nail key point detection has a simpler network structure, fewer layers, The processing speed is faster and the time-consuming for nail keypoint detection is reduced.

In some embodiments, before the cropped image is input to the second key point detection network, the cropped image may be rotated according to the direction of the nail in the previous frame.

For example, when the direction of the nail is an angle of 85 degrees between the normal line and the horizontal direction in the image coordinate system, the cropped image can be rotated counterclockwise by 5 degrees according to the direction, so that the The direction of the nail is that the normal points to the vertical direction.

The nail key point detection on the rotated image can improve the efficiency of detection on the one hand and the accuracy of detection on the other hand.

In the embodiment of the present disclosure, by obtaining the key point detection result in the current frame according to the key point detection result of the previous frame in the image sequence, the amount of data processing can be reduced, and the speed and efficiency of nail key point detection can be improved.

However, when the second key point of the nail is not detected or the second key point of the nail does not meet the set requirements, for example, if the detected second key point exceeds the scope of the cropped image area, then If it is determined that the tracking fails, the method of detecting the nail key points on the first image is still adopted, and the nail key point detection is performed on the second image, which specifically includes: obtaining a detection result of at least one nail in the second image, and the The detection result includes a first nail detection frame and a classification result of the nail, the classification result indicates the finger type to which the nail belongs; an image area corresponding to the nail in the second image is obtained according to the first nail detection frame Obtaining multiple first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs.

By judging the second key point of the nail tracked, if it is not detected or does not meet the set requirements, the nail recognition will be performed again. On the one hand, it ensures the consistency of the nail recognition results, and on the other hand, it also ensures Accuracy of nail recognition results.

In some implementations, nail recognition can be performed on video images containing hands in the following manner. As shown in FIG. 3 , the method may include steps 301 to 309 .

In step 301, for any frame of the first image in the image sequence contained in the video image, the nail detection network may be used to perform nail detection on the first image to obtain at least one nail in the first image The first nail detection frame, and the classification result of the nail. Wherein, the classification result indicates the finger type to which the nail belongs.

Usually, the first image is the first frame of image in a scene.

In step 302, the image area corresponding to the first nail detection frame is cut out to obtain a first nail area image.

In step 303, the binary classification result of each pixel in the nail region image is obtained, and the binary classification result indicates that the pixel is a foreground pixel or a background pixel; the pixel indicated as a background pixel in the binary classification result is set as The first pixel value.

In step 304, the nail area image processed in step 303 is input to the first key point detection network to obtain a plurality of first key points of the nail.

In step 305, for the second image following the first image, according to the plurality of first key points of the nail in the previous frame of the second image, a circumscribed rectangular frame of the nail is obtained; according to The location information of the circumscribed rectangular frame in the previous frame is used to map the circumscribed rectangular frame to the second image as a second nail detection frame in the second image.

In step 306, the image area corresponding to the second nail detection frame is cut out to obtain a second nail area image.

In step 307, the second nail region image is rotated according to the direction of the nail in the previous frame to obtain a rotated image. Wherein, the direction of the nail is determined according to the position information of at least two first key points among the plurality of first key points of the nail in the previous frame.

In step 308, the rotated image is input to the second key point detection network to obtain a plurality of second key points of the nail.

In step 309, the key point detection result obtained in step 308 is judged, and if the plurality of second key points of the nail meet the set requirements, it is judged that the tracking of the second image is successful, and the process returns to step 305 , continue to track the next frame of image; if the second key point is not detected, or if multiple second key points of the nail do not meet the set requirements, it is determined that the tracking is unsuccessful, and then return to step 301, Processing the second image as the first image.

Fig. 4 is a schematic structural diagram of a nail recognition device proposed by at least one embodiment of the present disclosure. As shown in Fig. 4, the device may include: a first acquisition unit 401, configured to acquire a detection result of at least one nail in the first image, so The detection result includes a first nail detection frame and a classification result of the nail, and the classification result indicates the finger type to which the nail belongs; the second acquiring unit 402 is configured to obtain the first nail detection frame according to the first nail detection frame. An image area corresponding to the nail in the image; an identification unit 403 configured to obtain a plurality of first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs.

In combination with any of the implementations provided in the present disclosure, the identification unit is specifically configured to: crop out the image area corresponding to the nail from the first image; input the cropped image area into the In the first key point detection network, multiple first key points of the nail are obtained.

In combination with any embodiment provided in the present disclosure, the device further includes a filtering unit, configured to: obtain a binary classification result of each pixel in the image region corresponding to the nail, the binary classification result indicating that the pixel is a foreground pixel or Background pixels: setting the pixels indicated as background pixels in the binary classification result as the first pixel value.

In combination with any implementation manner provided by the present disclosure, the device further includes an orientation unit configured to: according to the position information of at least two first key points among the plurality of first key points of the nail in the image area , to determine the orientation of the nail.

In combination with any of the implementations provided in the present disclosure, the device further includes a training unit configured to: acquire a sample image; wherein, the sample image has annotation information, and the annotation information indicates the finger type corresponding to the sample image. The first key point; input the sample image to the first key point detection network to obtain a key point detection result; according to the difference between the key point detection result and the annotation information, the first key point detection The network parameters of the keypoint detection network are tuned.

In combination with any implementation manner provided in the present disclosure, the first image is a frame in an image sequence, and the device further includes a tracking unit configured to: for a second image after the first image, according to the first image A plurality of first key points of the nail in the previous frame of the second image, determine a second nail detection frame in the second image; obtain the image area corresponding to the second nail detection frame in the second image , a plurality of second key points of the nail.

In combination with any implementation manner provided by the present disclosure, when the tracking unit is used to determine the second nail detection frame in the second image based on the multiple first key points in the previous frame of the second image, specifically It is used to: obtain the circumscribed rectangular frame of the nail according to the multiple first key points of the nail in the previous frame; obtain the circumscribed rectangular frame according to the position information of the circumscribed rectangular frame in the previous frame. The circumscribed rectangular frame is mapped to the second image as a second nail detection frame in the second image.

In combination with any implementation manner provided by the present disclosure, when the tracking unit is used to obtain multiple second key points of the nail in the image area corresponding to the second nail detection frame in the second image, It is specifically used for: cutting out the image area corresponding to the second nail detection frame in the second image; inputting the cropped image area into the second key point detection network to obtain the second key point of the nail.

In combination with any of the implementations provided in the present disclosure, the device further includes a rotation unit, configured to, before inputting the cropped image region into the second key point detection network, according to the nail in the previous frame The direction of the cropped image is rotated.

In combination with any implementation manner provided by the present disclosure, the device further includes a judging unit configured to: if the second key point of the nail is not detected or the second key point of the nail does not meet the set requirements, Acquiring a detection result of at least one nail in the second image, the detection result including a first nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; according to the first nail detection frame Obtain an image area corresponding to the nail in the second image; and obtain a plurality of first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs.

At least one embodiment of the present disclosure also provides an electronic device. As shown in FIG. 5 , the device includes a storage 501 and a processor 502. The computer instructions implement the image processing method described in any embodiment of the present disclosure.

At least one embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image processing method described in any embodiment of the present disclosure is implemented.

At least one embodiment of the present disclosure further provides a computer program product, including a computer program, which implements the image processing method described in any embodiment of the present disclosure when the program is executed by a processor.

Those skilled in the art should understand that one or more embodiments of this specification may be provided as a method, system or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of the present description may be implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein. in the form of computer program products.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the data processing device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment.

The foregoing describes specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that in the examples and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Multitasking and parallel processing are also possible or may be advantageous in certain embodiments.

Embodiments of the subject matter and functional operations described in this specification can be implemented in digital electronic circuitry, tangibly embodied computer software or firmware, computer hardware including the structures disclosed in this specification and their structural equivalents, or A combination of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, that is, one or more of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or to control the operation of data processing apparatus. Multiple mods. Alternatively or additionally, the program instructions may be encoded on an artificially generated propagated signal, such as a machine-generated electrical, optical or electromagnetic signal, which is generated to encode and transmit information to a suitable receiver device for transmission by the data The processing means executes. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform corresponding functions by operating on input data and generating output. The processes and logic flows can also be performed by, and devices can also be implemented as, special purpose logic circuits, such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits).

Computers suitable for the execution of computer programs include, for example, general and/or special purpose microprocessors, or any other type of central processing unit. Typically, a central processing unit will receive instructions and data from read only memory and/or random access memory. The basic components of a computer include a central processing unit for implementing or executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic, magneto-optical, or optical disks, to receive data therefrom or Send data to it, or both. However, a computer is not required to have such a device. Additionally, a computer may be embedded in another device such as a mobile phone, personal digital assistant (PDA), mobile audio or video player, game console, Global Positioning System (GPS) receiver, or USB) flash memory drives, to name a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and storage devices including, for example, semiconductor memory devices (such as EPROM, EEPROM and flash memory devices), magnetic disks (such as internal hard disk or removable disk), magneto-optical disk, and CD ROM and DVD-ROM. The processor and memory can be supplemented by, or incorporated in, special purpose logic circuitry.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any inventions or of what may be claimed, but rather as primarily describing features of particular embodiments of particular inventions. Certain features that are described in this specification in multiple embodiments can also be implemented in combination in a single embodiment. On the other hand, various features that are described in a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Furthermore, although features may function in certain combinations as described above and even be initially so claimed, one or more features from a claimed combination may in some cases be removed from that combination and the claimed A protected combination can point to a subcombination or a variant of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be construed as requiring that those operations be performed in the particular order shown, or sequentially, or that all illustrated operations be performed, to achieve desirable results . In some cases, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can often be integrated together in a single software product in, or packaged into multiple software products.

Thus, certain embodiments of the subject matter have been described. Other embodiments are within the scope of the appended claims. In some cases, the actions recited in the claims can be performed in a different order and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.

The above descriptions are only preferred embodiments of one or more embodiments of this specification, and are not intended to limit one or more embodiments of this specification. Within the spirit and principles of one or more embodiments of this specification, Any modification, equivalent replacement, improvement, etc. should be included in one or more embodiments of this specification.

101: Obtain the detection result of at least one nail in the first image 102: Obtain the image area corresponding to the nail in the first image according to the first nail detection frame 103: According to the type of finger to which the nail belongs, obtain a plurality of first key points of the nail in the image area corresponding to the nail P1~P16: The first key point 301: For any frame of the first image in the image sequence included in the video image, the nail detection network may be used to perform nail detection on the first image to obtain the first frame of at least one nail in the first image. The nail detection frame, and the classification result of the nail 302: Cut out the image area corresponding to the first nail detection frame to obtain the first nail area image 303: Obtain the binary classification result of each pixel in the nail region image, and set the pixel indicated as background pixel in the binary classification result as the first pixel value 304: Input the image of the nail area processed in step 303 to the first key point detection network to obtain a plurality of first key points of the nail 305: For the second image after the first image, according to the multiple first key points of the nail in the previous frame of the second image, obtain the circumscribed rectangular frame of the nail; according to the circumscribed The position information of the rectangular frame in the previous frame, mapping the circumscribed rectangular frame to the second image as the second nail detection frame in the second image 306: Cut out the image area corresponding to the second nail detection frame to obtain the second nail area image 307: Rotate the second nail region image according to the direction of the nail in the previous frame to obtain a rotated image 308: Input the rotated image to the second key point detection network to obtain multiple second key points of the nail 309: determine the key point detection result obtained in step 308 401: The first acquisition unit 402: The second acquisition unit 403: Identification unit 501: storage 502: Processor

Fig. 1 is a flow chart of a nail recognition method proposed by at least one embodiment of the present disclosure. Fig. 2 is a schematic diagram of a first key point of a nail in a nail recognition method proposed by at least one embodiment of the present disclosure. Fig. 3 is a flowchart of another nail recognition method proposed by at least one embodiment of the present disclosure. Fig. 4 is a schematic structural diagram of a nail recognition device proposed by at least one embodiment of the present disclosure. Fig. 5 is a schematic structural diagram of an electronic device proposed by at least one embodiment of the present disclosure.

101: Obtain the detection result of at least one nail in the first image

102: Obtain the image area corresponding to the nail in the first image according to the first nail detection frame

103: According to the type of finger to which the nail belongs, obtain a plurality of first key points of the nail in the image area corresponding to the nail

Claims (13)

A nail recognition method, characterized in that the method comprises: Acquiring a detection result of at least one nail in the first image, the detection result comprising a first nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; Obtaining an image area corresponding to the nail in the first image according to the first nail detection frame; According to the finger type to which the nail belongs, a plurality of first key points of the nail in the image area corresponding to the nail are obtained. The method according to claim 1, wherein, according to the finger type to which the nail belongs, obtaining a plurality of first key points of the nail in the image area corresponding to the nail includes: cutting out an image area corresponding to the nail from the first image; The cropped image area is input into the first key point detection network corresponding to the finger type to which the nail belongs, to obtain a plurality of first key points of the nail. The method according to claim 1 or 2, wherein the method further comprises: Obtaining a binary classification result of each pixel in the image region corresponding to the nail, the binary classification result indicating that the pixel is a foreground pixel or a background pixel; Set the pixels indicated as background pixels in the binary classification result as the first pixel value. The method according to claim 1 or 2, wherein the method further comprises: The direction of the nail is determined according to the position information of at least two first key points of the plurality of first key points of the nail in the image area. The method as described in claim 2, wherein the method further comprises: Acquiring a sample image; wherein, the sample image has annotation information, and the annotation information indicates a first key point corresponding to the type of finger to which the sample image belongs; inputting the sample image into the first key point detection network to obtain a key point detection result; According to the difference between the key point detection result and the annotation information, the network parameters of the first key point detection network are adjusted. The method according to any one of claims 1, 2, and 5, wherein the first image is a frame in an image sequence, and the method further includes: For a second image following the first image, determine a second nail detection frame in the second image according to a plurality of first key points of the nail in a previous frame of the second image; Obtaining multiple second key points of the nail in the image area corresponding to the second nail detection frame in the second image. The method according to claim 6, wherein the determining the second nail detection frame in the second image according to a plurality of first key points in the previous frame of the second image includes: Obtaining a circumscribed rectangular frame of the nail according to a plurality of first key points of the nail in the previous frame; According to the position information of the circumscribing rectangle in the previous frame, the circumscribing rectangle is mapped to the second image as a second nail detection frame in the second image. The method according to claim 6, wherein the obtaining multiple second key points of the nail in the image area corresponding to the second nail detection frame in the second image includes: cropping out the image area corresponding to the second nail detection frame in the second image; The clipped image area is input to the second key point detection network to obtain the second key point of the nail. The method according to claim 8, wherein, before inputting the cropped image area to the second key point detection network, according to the direction of the nail in the previous frame, the cropped The output image is rotated. The method according to any one of claim 6, wherein the method further comprises: In the case that the second key point of the nail is not detected or the second key point of the nail does not meet the set requirements, the detection result of at least one nail in the second image is obtained, and the detection result includes the first A nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; Obtain an image area corresponding to the nail in the second image according to the first nail detection frame; According to the finger type to which the nail belongs, a plurality of first key points of the nail in the image area corresponding to the nail are obtained. A nail recognition device, characterized in that the device comprises: A first acquiring unit, configured to acquire a detection result of at least one nail in the first image, the detection result including a first nail detection frame and a classification result of the nail, the classification result indicating the finger type to which the nail belongs; A second acquisition unit, configured to obtain an image area corresponding to the nail in the first image according to the first nail detection frame; The identification unit is configured to obtain a plurality of first key points of the nail in the image area corresponding to the nail according to the finger type to which the nail belongs. An electronic device, characterized in that the device includes a memory and a processor, the memory is used to store computer instructions that can be run on the processor, and the processor is used to realize claim 1 when executing the computer instructions to the method described in any one of 10. A computer-readable storage medium, on which a computer program is stored, is characterized in that, when the program is executed by a processor, the method described in any one of claims 1 to 10 is implemented.

Images