CN114596476A - Key point detection model training method, key point detection method and device - Google Patents

Abstract

The present disclosure provides a training method for a key point detection model, a key point detection method, and a device, which relate to the technical field of artificial intelligence, and in particular, to the technical fields of computer vision, deep learning, and augmented reality. The implementation scheme is: acquiring a sample image of a target object, the target object including multiple key points and at least one auxiliary point having a preset positional relationship with the multiple key points; inputting the sample image into the key point detection model to obtain multiple predicted key points output by the key point detection model; based on the multiple predicted key points and the preset positional relationship, determine at least one prediction auxiliary point; based on the multiple key points, all The at least one auxiliary point, the plurality of prediction key points, and the at least one prediction auxiliary point are used to determine a loss value of the key point detection model; and based on the loss value, parameters of the key point detection model are adjusted.

Description

Training method of key point detection model, key point detection method and device

technical field

The present disclosure relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, and augmented reality, and in particular to a training method and device for a key point detection model, a key point detection method and device, electronic equipment, and computer-readable storage media and computer program products.

Background technique

Artificial intelligence is the study of making computers to simulate certain thinking processes and intelligent behaviors of people (such as learning, reasoning, thinking, planning, etc.), both hardware-level technology and software-level technology. AI hardware technologies generally include technologies such as sensors, dedicated AI chips, cloud computing, distributed storage, and big data processing: AI software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, and machine learning/depth Learning, big data processing technology, knowledge graph technology and other major directions.

Keypoint detection is a computational task in the field of computer vision, which is used to detect keypoints of target objects in images, such as human joint points, contour points of obstacles, etc. Keypoint detection techniques are widely used in scenarios such as pose estimation, target tracking, and autonomous driving.

The approaches described in this section are not necessarily approaches that have been previously conceived or employed. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the issues raised in this section should not be considered to be recognized in any prior art.

SUMMARY OF THE INVENTION

The present disclosure provides a training method and apparatus for a key point detection model, a key point detection method and apparatus, an electronic device, a computer-readable storage medium, and a computer program product.

According to an aspect of the present disclosure, a method for training a keypoint detection model is provided, including: acquiring a sample image of a target object, where the target object includes a plurality of keypoints and has a preset positional relationship with the plurality of keypoints at least one auxiliary point; input the sample image into the keypoint detection model to obtain multiple predicted keypoints output by the keypoint detection model; based on the multiple predicted keypoints and the preset positional relationship , determine at least one prediction auxiliary point; based on the multiple key points, the at least one auxiliary point, the multiple prediction key points and the at least one prediction auxiliary point, determine the loss value of the key point detection model; and adjusting parameters of the keypoint detection model based on the loss value.

According to an aspect of the present disclosure, a key point detection method is provided, comprising: inputting an image to be detected of a target object into a key point detection model, where the key point detection model is obtained according to the above-mentioned training method of the key point detection model; and acquiring multiple key points of the target object output by the key point detection model.

According to an aspect of the present disclosure, there is provided a training device for a keypoint detection model, comprising: an acquisition module configured to acquire a sample image of a target object, the target object including a plurality of keypoints and a correlation with the plurality of keypoints The point has at least one auxiliary point with a preset positional relationship; the input and output module is configured to input the sample image into the key point detection model to obtain a plurality of predicted key points output by the key point detection model; the first a determination module, configured to determine at least one prediction auxiliary point based on the multiple prediction key points and the preset positional relationship; a second determination module, configured to determine at least one prediction auxiliary point based on the multiple key points, the at least one auxiliary point point, the plurality of predicted keypoints, and the at least one predictive auxiliary point, determining a loss value of the keypoint detection model; and an adjustment module configured to adjust the keypoint detection model based on the loss value parameter.

According to an aspect of the present disclosure, there is provided a keypoint detection apparatus, comprising: an input module configured to input an image to be detected of a target object into a keypoint detection model, where the keypoint detection model is based on the above-mentioned keypoint detection model and an acquisition module configured to acquire multiple key points of the target object output by the key point detection model.

According to an aspect of the present disclosure, there is provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor, the memory storing instructions executable by the at least one processor, the instructions Executed by the at least one processor to enable the at least one processor to perform the method of any of the above aspects.

According to one aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of any of the above-described aspects.

According to an aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the above-mentioned aspects.

According to one or more embodiments of the present disclosure, the accuracy of keypoint detection can be improved.

It should be understood that what is described in this section is not intended to identify key or critical features of embodiments of the disclosure, nor is it intended to limit the scope of the disclosure. Other features of the present disclosure will become readily understood from the following description.

Description of drawings

The accompanying drawings illustrate the embodiments by way of example and constitute a part of the specification, and together with the written description of the specification serve to explain exemplary implementations of the embodiments. The shown embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numbers refer to similar but not necessarily identical elements.

1 shows a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to embodiments of the present disclosure;

2 shows a flowchart of a training method for a keypoint detection model according to an embodiment of the present disclosure;

FIG. 3 shows a schematic diagram of a sample image according to an embodiment of the present disclosure;

4A-4D show schematic diagrams of key points and auxiliary points under four different preset positional relationships;

5 shows a flowchart of a key point detection method according to an embodiment of the present disclosure;

6 shows a structural block diagram of a training device for a keypoint detection model according to an embodiment of the present disclosure;

FIG. 7 shows a structural block diagram of a key point detection apparatus according to an embodiment of the present disclosure; and

FIG. 8 shows a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed ways

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to facilitate understanding and should be considered as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted from the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, timing relationship or importance relationship of these elements, and such terms are only used for Distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of the element, while in some cases they may refer to different instances based on the context of the description.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly dictates otherwise, if the number of an element is not expressly limited, the element may be one or more. Furthermore, as used in this disclosure, the term "and/or" covers any and all possible combinations of the listed items.

Keypoint detection techniques are widely used in a variety of scenarios. For example, in the augmented reality (AR) virtual shoe test scene, an image of the user's foot can be collected, key points of the image can be detected, and the key points of the foot can be identified from the image, and then based on the identified key points of the foot And determine the posture of the user's feet, and render the shoes that the user is interested in on the image of the user's feet, so as to show the wearing effect of the shoes to the user. For another example, in an autonomous driving scenario, key points of the images collected by the vehicle can be detected, and key points of target objects such as lane guidance arrows, parking spaces, obstacles, etc. can be identified, and then based on the identified key points, automatic Lane change, autonomous parking, obstacle avoidance and other functions.

In the related art, the key point detection technology based on deep learning is usually used to detect the key points of the target object in the image. That is, the image is input to the keypoint detection model, and the keypoint detection model outputs the keypoints of the target object in the image. However, it often happens that some keypoints of the target object are occluded by other objects in the image and are not visible (but the keypoints are still in the image), or some keypoints of the target object are truncated (that is, the keypoints are not in the image). For example, in a virtual shoe trial scenario, the key points of the ankle in the user's foot image are likely to be occluded by the user's current shoes, trousers or the user's limbs; the key points of the user's toes may be truncated, that is, not captured in the image. middle. For another example, in an autonomous driving scenario, some key points of a guide arrow in a road image collected by a vehicle may be occluded by other vehicles or floating objects (such as pieces of paper, leaves, etc.) on the road. In the above cases, the key points output by the model usually have large errors and are not accurate enough.

In view of the above problems, the present disclosure provides a training method for a key point detection model and a key point detection method, so as to improve the accuracy of key point detection. When some key points in the image are occluded, the positions of these occluded key points can still be accurately detected; when some key points are truncated, the key points in the image (that is, not truncated keypoints).

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

1 shows a schematic diagram of an exemplary system 100 in which the various methods and apparatuses described herein may be implemented in accordance with embodiments of the present disclosure. Referring to FIG. 1 , the system 100 includes one or more client devices 101 , 102 , 103 , 104 , 105 and 106 , a server 120 , and one or more communication networks coupling the one or more client devices to the server 120 110. Client devices 101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

In an embodiment of the present disclosure, the server 120 may run one or more services or software applications that enable the training method of the keypoint detection model and/or the keypoint detection method to be performed.

In some embodiments, server 120 may also provide other services or software applications that may include non-virtual and virtual environments. In some embodiments, these services may be provided as web-based services or cloud services, eg, provided to users of client devices 101, 102, 103, 104, 105, and/or 106 under a software-as-a-service (SaaS) model .

In the configuration shown in FIG. 1 , server 120 may include one or more components that implement the functions performed by server 120 . These components may include software components executable by one or more processors, hardware components, or a combination thereof. Users operating client devices 101, 102, 103, 104, 105, and/or 106 may in turn utilize one or more client applications to interact with server 120 to utilize the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100 . Accordingly, FIG. 1 is one example of a system for implementing the various methods described herein, and is not intended to be limiting.

A user may use client devices 101 , 102 , 103 , 104 , 105 and/or 106 to navigate. A client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although FIG. 1 depicts only six types of client devices, those skilled in the art will appreciate that the present disclosure may support any number of client devices.

Client devices 101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general-purpose computers (such as personal computers and laptop computers), workstation computers, wearable devices, Smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, etc. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux or Linux-like operating systems (such as GOOGLE Chrome OS); or include various mobile operating systems , such as MICROSOFT WindowsMobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular phones, smart phones, tablet computers, personal digital assistants (PDAs), and the like. Wearable devices may include head mounted displays (such as smart glasses) and other devices. Gaming systems may include various handheld gaming devices, Internet-enabled gaming devices, and the like. Client devices are capable of executing a variety of different applications, such as various Internet-related applications, communication applications (eg, e-mail applications), Short Message Service (SMS) applications, and may use various communication protocols.

Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, and the like. By way of example only, the one or more networks 110 may be a local area network (LAN), an Ethernet-based network, a token ring, a wide area network (WAN), the Internet, a virtual network, a virtual private network (VPN), an intranet, an extranet, Public Switched Telephone Network (PSTN), infrared networks, wireless networks (eg, Bluetooth, Wi-Fi), and/or any combination of these and/or other networks.

Server 120 may include one or more general purpose computers, special purpose server computers (eg, PC (personal computer) servers, UNIX servers, midrange servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination . Server 120 may include one or more virtual machines running virtual operating systems, or other computing architectures that involve virtualization (eg, may be virtualized to maintain one or more flexible pools of logical storage devices of the server's virtual storage devices). In various embodiments, server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. Server 120 may also run any of a variety of additional server applications and/or middle-tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

In some implementations, server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of client devices 101 , 102 , 103 , 104 , 105 , and 106 . Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of client devices 101 , 102 , 103 , 104 , 105 , and 106 .

In some embodiments, the server 120 may be a server of a distributed system, or a server combined with a blockchain. The server 120 may also be a cloud server, or an intelligent cloud computing server or an intelligent cloud host with artificial intelligence technology. Cloud server is a host product in the cloud computing service system to solve the defects of difficult management and weak business expansion in traditional physical host and virtual private server (VPS, Virtual Private Server) services.

System 100 may also include one or more databases 130 . In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as music files. Database 130 may reside in various locations. For example, the database used by server 120 may be local to server 120, or may be remote from server 120 and may communicate with server 120 via a network-based or dedicated connection. Database 130 may be of different types. In some embodiments, the database used by server 120 may be, for example, a relational database or a non-relational database. One or more of these databases can store, update, and retrieve data to and from the databases in response to commands.

In some embodiments, one or more of the databases 130 may also be used by applications to store application data. Databases used by applications can be different types of databases such as key-value stores, object stores, or regular stores backed by a file system.

The system 100 of FIG. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.

FIG. 2 shows a flowchart of a training method 200 of a keypoint detection model according to an embodiment of the present disclosure. Method 200 is typically performed at a server (eg, server 120 shown in FIG. 1 ), but may also be performed at client devices (eg, client devices 101 , 102 , 103 , 104 , 105 , and 106 shown in FIG. 1 ) . That is, the execution body of each step of the method 200 may be the server 120 shown in FIG. 1 , or may be the client devices 101 , 102 , 103 , 104 , 105 and 106 .

As shown in FIG. 2, method 200 includes steps 210-250.

In step 210, a sample image of a target object is obtained, where the target object includes multiple key points and at least one auxiliary point having a preset positional relationship with the multiple key points.

In step 220, the sample image is input into the keypoint detection model to obtain multiple predicted keypoints output by the keypoint detection model.

In step 230, at least one prediction auxiliary point is determined based on the plurality of prediction key points and the preset positional relationship.

In step 240, a loss value of the keypoint detection model is determined based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point.

In step 250, the parameters of the keypoint detection model are adjusted based on the loss value.

According to an embodiment of the present disclosure, in addition to the key points of the target object, auxiliary points having a certain positional relationship with the key points are also marked. Both key points and auxiliary points are involved in the calculation of the loss value. Since the auxiliary points have a certain positional relationship with the key points, the spatial position information of the key points can be expressed through the auxiliary points, which can strengthen the learning of the position of the key points by the model, thereby improving the accuracy of the key point detection.

According to an embodiment of the present disclosure, the multiple key points are points capable of expressing the characteristics of the target object, and at least one auxiliary point is determined based on the multiple key points and a preset positional relationship. There can be various preset positional relationships between auxiliary points and key points. For example, auxiliary points can be pixel points located at the preset positions of a straight line or line segment formed by connecting two key points, or a pixel that connects two key points. Any point on the formed line segment that differs from multiple keypoints, etc.

According to some embodiments, multiple key points and at least one auxiliary point marked in the sample image may be used as real values, and multiple predicted key points output by the key point detection model and at least one prediction determined based on the multiple predicted key points may be used as real values. The auxiliary point is used as the predicted value, and the loss value of the key point detection model can be calculated by comparing the predicted value with the real value. Based on the calculated loss value, parameters of the keypoint detection model can be adjusted using, for example, a back-propagation algorithm, thereby obtaining a trained keypoint detection model.

A target object, a sample image, a plurality of key points, a preset positional relationship, and at least one auxiliary point in the embodiments of the present disclosure will be described in detail below.

In the embodiment of the present disclosure, the target object refers to an object to be subjected to key point detection. For example, in a virtual shoe fitting scene, the target object may be the user's foot. In autonomous driving scenarios, the target objects can be guide arrows, parking spaces, obstacles, etc.

A sample image is an image that contains the target object. For example, in a virtual shoe fitting scenario, the sample image may be an image containing the user's foot. In an autonomous driving scenario, the sample image can be a road surface image containing guiding arrows.

FIG. 3 shows a schematic diagram of a sample image 300 according to an embodiment of the present disclosure. As shown in FIG. 3 , the sample image 300 is a road surface image that includes a guide arrow 310 (ie, a target object).

In the embodiment of the present disclosure, the key point is a point that can express the characteristics of the target object. Keypoints are usually located at specific parts of the target object. The number of key points and the corresponding parts of the target object can be set by those skilled in the art according to actual application scenarios. For example, in a virtual shoe trial scene, the target object may be the user's foot, and the key points may include the medial ankle joint point, lateral ankle joint point, heel point, toe point, connection point between the instep and the leg, etc.

In an embodiment of the present disclosure, the sample image is annotated with a plurality of key points of the target object.

According to some embodiments, the sample image is also marked with adjacent relationships between multiple key points. For example, in the application scenario of human pose detection, the key points of the human body (that is, the target object) include shoulder joint points, elbow joint points, wrist joint points, knee joint points, etc., wherein the shoulder joint point is adjacent to the elbow joint point, The elbow joint point is adjacent to the wrist joint point.

In an embodiment of the present disclosure, the sample image is further marked with at least one auxiliary point having a preset positional relationship with the plurality of key points. The number of auxiliary points can be determined by those skilled in the art with reference to actual application scenarios, which is not limited in the present disclosure.

In an embodiment of the present disclosure, at least one auxiliary point is determined based on a plurality of key points and a preset positional relationship. There are various preset positional relationships between auxiliary points and key points. The present disclosure does not limit the specific form of the preset positional relationship, as long as at least one auxiliary point can be determined according to the preset positional relationship and a plurality of given key points.

4A-4D show schematic diagrams of auxiliary points obtained according to four different preset positional relationships given the key points A, B, C, and D of the target object 410 . The manner of determining the auxiliary points in FIGS. 4A-4D will be described below with reference to the corresponding embodiments.

According to some embodiments, any one of the at least one auxiliary point lies on a line formed by corresponding two of the plurality of key points. Specifically, an auxiliary point may be located at any specified position on a line formed by two key points, for example, at a specified position (eg, the midpoint) on a line segment formed by two key points, or at a specified position on a line segment formed by two key points. The specified location on the extension of the line segment formed by the keypoint. Based on the above embodiments, the positioning and calculation of auxiliary points can be facilitated, that is, each auxiliary point can be quickly determined based on a plurality of given key points.

It should be noted that, in the embodiments of the present disclosure, the corresponding two key points for connecting to form a straight line (or line segment) may be any two key points among multiple key points. In the case where the adjacent relationship between multiple key points is marked in the sample image, the corresponding two key points used for connecting to form a straight line may also be two adjacent key points.

FIG. 4A shows a schematic diagram of key points and auxiliary points obtained by annotating a target object in a sample image according to the above embodiment. Among them, the key points are represented by solid rectangular points and uppercase letters, and the auxiliary points are represented by shaded rectangular points and lowercase letters. As shown in FIG. 4A , the guide arrow 410 (ie, the target object) includes four key points A, B, C, D and two auxiliary points a, b. The auxiliary point a is located on the extension line of the line segment BD formed by the key points B and D, and the distance from the key point B is 1/2 of the length of the line segment BD. The auxiliary point b is located at the midpoint of the line segment BC formed by the key points B and C.

According to some embodiments, on the basis of the above-mentioned embodiments, further, any auxiliary point in the at least one auxiliary point is located on a line segment formed by corresponding two key points among the plurality of key points, instead of being located on the line segment extension line. Therefore, all the auxiliary points are located on the line segment formed by connecting the key points, which facilitates the positioning and calculation of the auxiliary points, and avoids the auxiliary points overflowing the image (ie, being located outside the image).

According to some embodiments, on the basis that the auxiliary point is located on the line segment formed by the two key points, further, the auxiliary point may be located at a preset position of the line segment. That is, according to some embodiments, any one of the at least one auxiliary point is located at a preset position of a line segment formed by corresponding two of the plurality of key points. The preset position may be, for example, the midpoint of the line segment, or a distance of 1/3 the length of the line segment, 1/4 of the length of the line segment from the specified key point among the two key points, and so on. In this way, each pair of corresponding key points in the multiple key points can generate an auxiliary point, and each auxiliary point is evenly distributed among the multiple key points, so the auxiliary points can reasonably and effectively express the space of the key points position information, so as to improve the learning effect of the model on the position of key points and improve the accuracy of key point detection. Also, based on this embodiment, the number of auxiliary points of the target object in different sample images is fixed (same as the number of key point pairs) and small, so that the computational efficiency of key point detection can be improved.

FIG. 4B shows a schematic diagram of key points and auxiliary points obtained by annotating a target object in a sample image according to the above embodiment. As shown in FIG. 4B , the guide arrow 410 (ie, the target object) includes four key points A, B, C, and D, and the adjacent relationship between these four key points is marked in the sample image, that is, A and B are in phase adjacent, B is adjacent to C, and B is adjacent to D. By connecting adjacent key points, three line segments can be obtained, namely line segments AB, BC, and BD. As shown in FIG. 4B , the guide arrow 410 also includes three auxiliary points c, d, and e, which are located at the midpoints of the line segments AB, BC, and BD, respectively.

According to other embodiments, on the basis that the auxiliary point is located on a line segment formed by two key points, further, all pixel points on the line segment except the key point may be used as auxiliary points. That is, the at least one auxiliary point included in the target object is a pixel point on a plurality of line segments formed by connecting a plurality of key points two by two, and is different from the plurality of key points. Therefore, the number of auxiliary points of the target object in different sample images is not fixed (because the number of pixel points on each line segment in different sample images is not fixed) and is large, which can express the key points more comprehensively, abundantly and flexibly. The spatial position information of the points (compared to the embodiment described above with reference to FIG. 4B ), thereby improving the learning effect of the model on the positions of key points and improving the accuracy of key point detection.

FIG. 4C shows a schematic diagram of key points and auxiliary points obtained by annotating a target object in a sample image according to the above embodiment. As shown in FIG. 4C , the guide arrow 410 (ie, the target object) includes four key points A, B, C, D. Connect these four key points two by two to obtain six line segments, namely line segments AB, AC, AD, BC, BD, CD (since the three key points A, B, and C are collinear, in Figure 4C, the line segment AC is The line segments AB and BC are covered). All pixels except key points A, B, C, and D on these six line segments are auxiliary points. In order to make the drawing more concise and clear, each auxiliary point is not drawn separately in FIG. 4C , but only the line segment where the auxiliary point is located is drawn.

According to other embodiments, on the basis that the auxiliary point is located on the line segment formed by the two key points, and the sample image is marked with the adjacent relationship between the multiple key points, further, the adjacent key points may be formed by All pixels on the line segment except key points are used as auxiliary points. That is, in the case that the sample image is marked with the adjacent relationship between multiple key points, at least one auxiliary point included in the target object is a plurality of line segments formed by connecting adjacent key points among the multiple key points. on the pixel points that are different from the plurality of key points. As a result, the spatial location information of key points can be expressed comprehensively and pertinently, and the amount of calculation is reduced while improving the accuracy of key point detection (compared to the embodiment described above with reference to FIG. 4C ) , which improves the computational efficiency and achieves a balance between accuracy and computational efficiency, thereby enabling accurate and real-time detection of key points.

FIG. 4D shows a schematic diagram of key points and auxiliary points obtained by annotating a target object in a sample image according to the above embodiment. As shown in FIG. 4D , the guide arrow 410 (ie, the target object) includes four key points A, B, C, and D, and the adjacent relationship between these four key points is marked in the sample image, that is, A and B are in phase adjacent, B is adjacent to C, and B is adjacent to D. By connecting adjacent key points, three line segments can be obtained, namely line segments AB, BC, and BD. All pixels except key points A, B, C, and D on these three line segments are auxiliary points. In order to make the drawing more concise and clear, each auxiliary point is not drawn separately in FIG. 4D , but only the line segment where the auxiliary point is located is drawn. According to an embodiment of the present disclosure, a sample image is input into the keypoint detection model, and a plurality of predicted keypoints output by the keypoint detection model can be obtained.

It should be understood that the multiple predicted key points output by the key point detection model respectively correspond to the multiple key points marked in the sample image. That is, each predicted keypoint output by the keypoint detection model corresponds to a labeled keypoint. Taking the above Figures 4A-4D as an example, the marked key points are key points A, B, C, and D, then the key point detection model will output four predicted key points A', B', C', D'. The predicted key points A', B', C', and D' correspond to the marked key points A, B, C, and D, respectively.

In addition, it should be understood that multiple predicted key points output by the key point detection model are predicted values, and multiple key points marked in the sample image are real values.

The key point detection model can be any neural network model, and the present disclosure does not limit the specific structure of the key point detection model.

In some embodiments, the keypoint detection model may be, for example, a neural network model composed of a feature extraction module, a heatmap generation module, and a keypoint output module. Among them, the feature extraction module is used to extract the image features of the sample image. The heatmap generation module generates a keypoint heatmap corresponding to the sample image based on the image features, and the keypoint heatmap is used to represent the confidence that each pixel in the sample image is a keypoint of the target object. The key point output module determines and outputs multiple key points of the target object based on the key point heat map.

After the multiple prediction key points output by the key point detection model are obtained, at least one prediction auxiliary point may be determined based on the multiple prediction key points and the preset positional relationship.

Determining at least one auxiliary point for prediction based on the predicted key point and the preset positional relationship is similar to the above-described process for determining at least one auxiliary point based on the relationship between the key point and the preset position, and will not be repeated here.

After the plurality of predicted keypoints and the at least one predicted auxiliary point are obtained, a loss value of the keypoint detection model may be determined based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point.

According to some embodiments, determining the loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point includes: based on the plurality of keypoints and the at least one auxiliary point, generating A label image corresponding to the sample image; based on multiple prediction key points and at least one prediction auxiliary point, a prediction image corresponding to the sample image is generated; and based on the label image and the prediction image, the loss value of the key point detection model is determined. The loss value determination method in this embodiment can be applied to any preset positional relationship between the key point and the auxiliary point, that is, can be applied to any of the embodiments described above with reference to FIGS. 4A-4D .

In the above-mentioned embodiment, the size of the label image and the predicted image (ie, the number of pixels included in the horizontal and vertical directions) are the same as those of the sample image. Also, the label image is used to indicate the positions of the key points and auxiliary points (true values) in the sample image, and the predicted image is used to indicate the positions of the predicted key points and the predicted auxiliary points (predicted values) in the sample image.

According to some embodiments, in the label image, the pixel values of the multiple key points and the at least one auxiliary point are a first preset value (for example, 1), and the pixel values of the other pixel points except the multiple key points and the at least one auxiliary point have a first preset value (for example, 1). The pixel value is a second preset value (eg, 0). In the predicted image, the pixel values of the multiple prediction key points and the at least one prediction auxiliary point are a first preset value (for example, 1), and the pixels of other pixel points except the multiple prediction key points and the at least one prediction auxiliary point are the first preset value (for example, 1). The value is the second preset value (eg, 0).

After obtaining the label image and the predicted image, the loss value of the model can be calculated based on the label image and the predicted image. According to some embodiments, the loss value of the model may be, for example, Mean Square Error (MSE), Mean Absolute Error (MAE), etc. between the label image and the predicted image.

According to other embodiments, in the case where multiple key points correspond to multiple prediction key points, and at least one auxiliary point corresponds to at least one prediction auxiliary point, respectively, based on multiple key points, at least one auxiliary point, multiple prediction points The key point and at least one prediction auxiliary point, determining the loss value of the key point detection model may also include: based on the first distance between each key point and the corresponding prediction key point and the second distance between each auxiliary point and the corresponding prediction auxiliary point. Distance, which determines the loss value of the keypoint detection model. For example, the loss value of the model may be the average of each of the first distances and each of the second distances. The loss value determination method in this embodiment is only applicable to the case where the number of auxiliary points is fixed, that is, only applicable to the embodiment described above with reference to FIG. 4B .

Based on the determined loss value, the parameters of the keypoint detection model can be adjusted.

According to some embodiments, a back-propagation algorithm may be employed to adjust the parameters of the keypoint detection model.

It should be understood that each step of the method for training a keypoint detection model according to an embodiment of the present disclosure may be performed repeatedly until a preset termination condition is reached (for example, the loss value is less than a threshold value, the number of cycles reaches a preset maximum number of cycles, etc.) When , the training process of the model is ended, and the trained keypoint detection model is obtained.

According to an embodiment of the present disclosure, based on the key point detection model trained by the method 200, a key point detection method is also provided.

FIG. 5 shows a flowchart of a keypoint detection method 500 according to an embodiment of the present disclosure. Method 500 is typically performed at a server (eg, server 120 shown in FIG. 1 ), but may also be performed at a client device (eg, client devices 101 - 106 shown in FIG. 1 ). That is, the execution body of each step of the method 500 may be the server 120 shown in FIG. 1 or the client devices 101-106.

Specifically, according to some embodiments, the keypoint detection model trained based on the method 200 may be deployed at a server. The user can upload the image to be detected of the target object to the server through the client device, and the server executes the key point detection method 500 of the embodiment of the present disclosure based on the key point detection model to obtain multiple key points of the target object, and then returns the key points. to the client device.

According to other embodiments, the keypoint detection model trained based on the method 200 may also be deployed at the client device. Correspondingly, the user can capture (shoot) or specify the image to be detected of the target object through the client device. The client device executes the key point detection method 500 of the embodiment of the present disclosure based on the locally deployed key point detection model to obtain multiple key points of the target object.

As shown in FIG. 5 , method 500 includes step 510 and step 520 .

In step 510, the to-be-detected image of the target object is input into the keypoint detection model, where the keypoint detection model is obtained based on the training method of the keypoint detection model of the embodiment of the present disclosure.

In step 520, multiple key points of the target object output by the key point detection model are acquired.

According to the embodiments of the present disclosure, accurate detection of key points can be achieved.

According to an embodiment of the present disclosure, a training apparatus for a keypoint detection model is also provided. FIG. 6 shows a structural block diagram of an apparatus 600 for training a keypoint detection model according to an embodiment of the present disclosure. As shown in FIG. 6, the apparatus 600 includes:

an acquisition module 610, configured to acquire a sample image of a target object, where the target object includes a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points;

an input-output module 620, configured to input the sample image into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model;

a first determination module 630, configured to determine at least one prediction auxiliary point based on the plurality of prediction key points and the preset positional relationship;

The second determination module 640 is configured to determine a loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point ;as well as

The adjustment module 650 is configured to adjust the parameters of the keypoint detection model based on the loss value.

According to an embodiment of the present disclosure, in addition to the key points of the target object, auxiliary points having a certain positional relationship with the key points are also marked. Both key points and auxiliary points are involved in the calculation of the loss value. Since the auxiliary points have a certain positional relationship with the key points, the spatial position information of the key points can be expressed through the auxiliary points, which can strengthen the learning of the position of the key points by the model, thereby improving the accuracy of the key point detection.

According to some embodiments, any one of the at least one auxiliary point is located on a straight line formed by corresponding two key points of the plurality of key points.

According to some embodiments, any one of the at least one auxiliary point is located on a line segment formed by corresponding two of the plurality of key points.

According to some embodiments, any one of the at least one auxiliary point is located at a preset position of a line segment formed by corresponding two of the plurality of key points.

According to some embodiments, the at least one auxiliary point is a pixel point on a plurality of line segments formed by connecting the plurality of key points two by two, which is different from the plurality of key points.

According to some embodiments, the sample image is marked with an adjacent relationship between the plurality of key points, and wherein the at least one auxiliary point is used to connect adjacent key points among the plurality of key points. Pixel points on the formed multiple line segments that are different from the multiple key points.

According to some embodiments, the second determining module 640 includes: a first generating unit configured to generate a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point; a second generating unit, is configured to generate a predicted image corresponding to the sample image based on the plurality of predicted key points and the at least one predicted auxiliary point; and a determining unit configured to determine the predicted image based on the label image and the predicted image. The loss value of the keypoint detection model described above.

According to some embodiments, in the label image, the pixel values of the plurality of key points and the at least one auxiliary point are a first preset value, except that between the plurality of key points and the at least one auxiliary point The pixel values of other pixel points except for the second preset value; in the predicted image, the pixel values of the multiple prediction key points and the at least one prediction auxiliary point are the first preset value, except The pixel values of other pixel points other than the plurality of prediction key points and the at least one prediction auxiliary point are the second preset value.

According to some embodiments, the second determination module 640 is further configured to determine the keypoints based on the first distance of each keypoint from the corresponding predicted keypoint and the second distance of each auxiliary point from the corresponding predicted auxiliary point The loss value of the detection model.

According to an embodiment of the present disclosure, a key point detection apparatus is also provided. FIG. 7 shows a structural block diagram of a keypoint detection apparatus 700 according to an embodiment of the present disclosure. As shown in FIG. 7, apparatus 700 includes:

The input module 710 is configured to input the to-be-detected image of the target object into a keypoint detection model, wherein the keypoint detection model is obtained by a training device for a keypoint detection model according to an embodiment of the present disclosure; and

The obtaining module 720 is configured to obtain a plurality of key points of the target object output by the key point detection model.

According to the embodiments of the present disclosure, accurate detection of key points can be achieved.

It should be understood that each module or unit of the apparatus 600 shown in FIG. 6 may correspond to each step in the method 200 described with reference to FIG. 2 , and each module or unit of the apparatus 700 shown in FIG. The various steps in method 500 correspond to each other. Therefore, the operations, features and advantages described above for method 200 are also applicable to apparatus 600 and the modules and units it includes, and the operations, features and advantages described above for method 500 are also applicable to apparatus 700 and the modules and units it includes. . For the sake of brevity, certain operations, features and advantages are not repeated here.

Although specific functionality is discussed above with reference to specific modules, it should be noted that the functionality of the various modules discussed herein may be divided into multiple modules, and/or at least some of the functionality of multiple modules may be combined into a single module. For example, the first determination module 630 and the second determination module 640 described above may be combined into a single module in some embodiments.

It should also be understood that various techniques may be described herein in the general context of software hardware elements or program modules. The various modules described above with respect to Figures 6 and 7 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, these modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a non-transitory computer readable storage medium. Alternatively, these modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the modules 610-720 may be implemented together in a System on Chip (SoC). An SoC may include an integrated circuit chip (which includes a processor (eg, a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, a or more communication interfaces, and/or one or more components of other circuits), and may optionally execute the received program code and/or include embedded firmware to perform functions.

According to an embodiment of the present disclosure, there is also provided an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores data executable by the at least one processor The instructions are executed by the at least one processor, so that the at least one processor can execute the above-mentioned training method and/or keypoint detection method of the keypoint detection model.

According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to cause a computer to execute the above-mentioned training method for a keypoint detection model and/or keypoint detection method.

According to an embodiment of the present disclosure, a computer program product is also provided, including a computer program, wherein the computer program implements the above-mentioned training method and/or keypoint detection method of a keypoint detection model when executed by a processor.

Referring to FIG. 8 , a structural block diagram of an electronic device 800 that can serve as a server or client of the present disclosure will now be described, which is an example of a hardware device that can be applied to various aspects of the present disclosure. Electronic devices are intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Electronic devices may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are by way of example only, and are not intended to limit implementations of the disclosure described and/or claimed herein.

As shown in FIG. 8 , the electronic device 800 includes a computing unit 801 that can be programmed according to a computer program stored in a read only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803 Various appropriate actions and processes are performed. In the RAM 803, various programs and data necessary for the operation of the electronic device 800 can also be stored. The computing unit 801 , the ROM 802 , and the RAM 803 are connected to each other through a bus 804 . An input/output (I/O) interface 805 is also connected to bus 804 .

Various components in the electronic device 800 are connected to the I/O interface 805 , including: an input unit 806 , an output unit 807 , a storage unit 808 , and a communication unit 809 . The input unit 806 may be any type of device capable of inputting information to the electronic device 800, the input unit 806 may receive input numerical or character information, and generate key signal input related to user settings and/or function control of the electronic device, and This may include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and/or remote control. The output unit 807 may be any type of device capable of presenting information, and may include, but is not limited to, a display, speakers, video/audio output terminals, vibrators, and/or printers. The storage unit 808 may include, but is not limited to, magnetic disks and optical disks. Communication unit 809 allows electronic device 800 to exchange information/data with other devices through computer networks such as the Internet and/or various telecommunication networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and/or chips Groups such as Bluetooth ^™ devices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices and/or the like.

Computing unit 801 may be various general-purpose and/or special-purpose processing components with processing and computing capabilities. Some examples of computing units 801 include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processing processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 801 performs the various methods and processes described above, eg, method 200 and/or method 500 . For example, in some embodiments, method 200 and/or method 500 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808 . In some embodiments, part or all of the computer program may be loaded and/or installed on the electronic device 800 via the ROM 802 and/or the communication unit 809 . When a computer program is loaded into RAM 803 and executed by computing unit 801, one or more steps of method 200 and method 500 described above may be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform method 200 and/or method 500 in any other suitable manner (eg, by means of firmware).

Various implementations of the systems and techniques described herein above may be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips system (SOC), complex programmable logic device (CPLD), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include being implemented in one or more computer programs executable and/or interpretable on a programmable system including at least one programmable processor that The processor, which may be a special purpose or general-purpose programmable processor, may receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device an output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer or other programmable data processing apparatus, such that the program code, when executed by the processor or controller, performs the functions/functions specified in the flowcharts and/or block diagrams. Action is implemented. The program code may execute entirely on the machine, partly on the machine, partly on the machine and partly on a remote machine as a stand-alone software package or entirely on the remote machine or server.

In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in connection with the instruction execution system, apparatus or device. The machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media would include one or more wire-based electrical connections, portable computer disks, hard disks, random access memory (RAM), read only memory (ROM), erasable programmable read only memory (EPROM or flash memory), fiber optics, compact disk read only memory (CD-ROM), optical storage, magnetic storage, or any suitable combination of the foregoing.

To provide interaction with a user, the systems and techniques described herein may be implemented on a computer having a display device (eg, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user ); and a keyboard and pointing device (eg, a mouse or trackball) through which a user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (eg, visual feedback, auditory feedback, or tactile feedback); and can be in any form (including acoustic input, voice input, or tactile input) to receive input from the user.

The systems and techniques described herein may be implemented on a computing system that includes back-end components (eg, as a data server), or a computing system that includes middleware components (eg, an application server), or a computing system that includes front-end components (eg, a user's computer having a graphical user interface or web browser through which a user may interact with implementations of the systems and techniques described herein), or including such backend components, middleware components, Or any combination of front-end components in a computing system. The components of the system may be interconnected by any form or medium of digital data communication (eg, a communication network). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

A computer system can include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server arises by computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, a distributed system server, or a server combined with blockchain.

It should be understood that steps may be reordered, added or deleted using the various forms of flow shown above. For example, the steps described in the present disclosure can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved, which are not limited herein.

Although the embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it should be understood that the above-described methods, systems and devices are merely exemplary embodiments or examples, and the scope of the present invention is not limited by these embodiments or examples, but is limited only by the appended claims and their equivalents. Various elements of the embodiments or examples may be omitted or replaced by equivalents thereof. Furthermore, steps may be performed in an order different from that described in this disclosure. Further, various elements of the embodiments or examples may be combined in various ways. Importantly, as technology evolves, many of the elements described herein may be replaced by equivalent elements that appear later in this disclosure.

Claims (18)

1. A method for training a keypoint detection model comprises the following steps:

acquiring a sample image of a target object, wherein the target object comprises a plurality of key points and at least one auxiliary point having a preset position relationship with the key points;

inputting the sample image into the key point detection model to obtain a plurality of predicted key points output by the key point detection model;

determining at least one prediction auxiliary point based on the plurality of prediction key points and the preset position relation;

determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and

adjusting parameters of the keypoint detection model based on the loss value.

2. The method of claim 1, wherein any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints.

3. The method of claim 2, wherein any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints.

4. The method of claim 2, wherein the at least one auxiliary point is a pixel point different from the plurality of key points on a plurality of line segments formed by connecting the plurality of key points two by two.

5. The method of claim 2, wherein the sample image is labeled with neighboring relationships between the plurality of keypoints, and wherein the at least one auxiliary point is a pixel point on a plurality of line segments formed by connecting neighboring keypoints of the plurality of keypoints that is different from the plurality of keypoints.

6. The method of any of claims 1-5, wherein determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point comprises:

generating a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point;

generating a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and

determining a loss value of the keypoint detection model based on the tag image and the predicted image.

7. The method of claim 6, wherein, in the label image, the pixel values of the plurality of key points and the at least one auxiliary point are a first preset value, and the pixel values of other pixel points except the plurality of key points and the at least one auxiliary point are a second preset value;

in the predicted image, the pixel values of the plurality of prediction key points and the at least one prediction auxiliary point are the first preset value, and the pixel values of other pixel points except the plurality of prediction key points and the at least one prediction auxiliary point are the second preset value.

8. The method of claim 3, wherein determining a loss value for the keypoint detection model, based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point, comprises:

determining a loss value of the keypoint detection model based on a first distance of each keypoint from a corresponding predicted keypoint and a second distance of each auxiliary point from a corresponding predicted auxiliary point.

9. A keypoint detection method comprising:

inputting an image to be detected of a target object into a key point detection model, wherein the key point detection model is obtained according to the method of any one of claims 1-8; and

and acquiring a plurality of key points of the target object output by the key point detection model.

10. A training apparatus for a keypoint detection model, comprising:

an acquisition module configured to acquire a sample image of a target object, wherein the target object includes a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points;

an input-output module configured to input the sample image into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model;

a first determination module configured to determine at least one prediction assistance point based on the plurality of prediction key points and the preset positional relationship;

a second determination module configured to determine a loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and

an adjustment module configured to adjust parameters of the keypoint detection model based on the loss values.

11. The apparatus of claim 10, wherein any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints.

12. The apparatus of claim 11, wherein any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints.

13. The apparatus of claim 11, wherein the sample image is labeled with neighboring relationships between the plurality of keypoints, and wherein the at least one auxiliary point is a pixel point on a plurality of line segments formed by connecting neighboring keypoints of the plurality of keypoints that is different from the plurality of keypoints.

14. The apparatus of any of claims 10-13, wherein the second determining means comprises:

a first generating unit configured to generate a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point;

a second generating unit configured to generate a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and

a determining unit configured to determine a loss value of the key point detection model based on the tag image and the prediction image.

15. A keypoint detection apparatus comprising:

an input module configured to input an image to be detected of a target object into a keypoint detection model, wherein the keypoint detection model is obtained by the apparatus according to any one of claims 10-14; and

an obtaining module configured to obtain a plurality of key points of the target object output by the key point detection model.

16. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

17. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-9.

18. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-9 when executed by a processor.

Images