CN114596476A

CN114596476A - Key point detection model training method, key point detection method and device

Info

Publication number: CN114596476A
Application number: CN202210239623.9A
Authority: CN
Inventors: 宫延河
Original assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Current assignee: Beijing Baidu Netcom Science and Technology Co Ltd
Priority date: 2022-03-11
Filing date: 2022-03-11
Publication date: 2022-06-07

Abstract

The disclosure provides a training method of a key point detection model, a key point detection method and a key point detection device, relates to the technical field of artificial intelligence, and particularly relates to the technical field of computer vision, deep learning and augmented reality. The implementation scheme is as follows: obtaining a sample image of a target object, wherein the target object comprises a plurality of key points and at least one auxiliary point having a preset position relation with the key points; inputting the sample image into the key point detection model to obtain a plurality of predicted key points output by the key point detection model; determining at least one prediction auxiliary point based on the plurality of prediction key points and the preset position relation; determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and adjusting parameters of the keypoint detection model based on the loss value.

Description

Key point detection model training method, key point detection method and device

Technical Field

The present disclosure relates to the field of artificial intelligence technologies, and in particular, to the field of computer vision, deep learning, and augmented reality technologies, and in particular, to a method and an apparatus for training a keypoint detection model, a method and an apparatus for keypoint detection, an electronic device, a computer-readable storage medium, and a computer program product.

Background

Artificial intelligence is the subject of research that makes computers simulate some human mental processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.), both at the hardware level and at the software level. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing, and the like: the artificial intelligence software technology mainly comprises a computer vision technology, a voice recognition technology, a natural language processing technology, machine learning/deep learning, a big data processing technology, a knowledge map technology and the like.

The key point detection is a calculation task in the field of computer vision, and is used for detecting key points of a target object in an image, such as human body joint points, contour points of an obstacle, and the like. Keypoint detection techniques are widely used in scenarios such as pose estimation, target tracking, autonomous driving, and the like.

The approaches described in this section are not necessarily approaches that have been previously conceived or pursued. Unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, unless otherwise indicated, the problems mentioned in this section should not be considered as having been acknowledged in any prior art.

Disclosure of Invention

The disclosure provides a method and a device for training a key point detection model, a method and a device for detecting key points, electronic equipment, a computer readable storage medium and a computer program product.

According to an aspect of the present disclosure, there is provided a method for training a keypoint detection model, including: obtaining a sample image of a target object, wherein the target object comprises a plurality of key points and at least one auxiliary point having a preset position relation with the key points; inputting the sample image into the key point detection model to obtain a plurality of predicted key points output by the key point detection model; determining at least one prediction auxiliary point based on the plurality of prediction key points and the preset position relation; determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and adjusting parameters of the keypoint detection model based on the loss value.

According to an aspect of the present disclosure, there is provided a keypoint detection method, including: inputting an image to be detected of a target object into a key point detection model, wherein the key point detection model is obtained according to a training method of the key point detection model; and acquiring a plurality of key points of the target object output by the key point detection model.

According to an aspect of the present disclosure, there is provided a training apparatus for a keypoint detection model, including: an acquisition module configured to acquire a sample image of a target object, the target object including a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points; an input-output module configured to input the sample image into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model; a first determination module configured to determine at least one prediction assistance point based on the plurality of prediction key points and the preset positional relationship; a second determination module configured to determine a loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and an adjustment module configured to adjust parameters of the keypoint detection model based on the loss values.

According to an aspect of the present disclosure, there is provided a keypoint detection apparatus, comprising: the image processing device comprises an input module, a processing module and a processing module, wherein the input module is configured to input an image to be detected of a target object into a key point detection model, and the key point detection model is obtained according to a training device of the key point detection model; and an acquisition module configured to acquire a plurality of keypoints of the target object output by the keypoint detection model.

According to an aspect of the present disclosure, there is provided an electronic device including: at least one processor; and a memory communicatively coupled to the at least one processor, the memory storing instructions executable by the at least one processor to enable the at least one processor to perform the method of any of the above aspects.

According to an aspect of the present disclosure, there is provided a non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any of the above aspects.

According to an aspect of the disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements the method of any of the above aspects.

According to one or more embodiments of the present disclosure, the accuracy of keypoint detection can be improved.

It should be understood that the statements in this section do not necessarily identify key or critical features of the embodiments of the present disclosure, nor do they limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the embodiments and, together with the description, serve to explain the exemplary implementations of the embodiments. The illustrated embodiments are for purposes of illustration only and do not limit the scope of the claims. Throughout the drawings, identical reference numbers designate similar, but not necessarily identical, elements.

FIG. 1 illustrates a schematic diagram of an exemplary system in which various methods described herein may be implemented, according to an embodiment of the present disclosure;

FIG. 2 shows a flow diagram of a method of training a keypoint detection model according to an embodiment of the disclosure;

FIG. 3 shows a schematic diagram of a sample image according to an embodiment of the present disclosure;

FIGS. 4A-4D are schematic diagrams illustrating key points and auxiliary points in four different predetermined positional relationships;

FIG. 5 shows a flow diagram of a keypoint detection method according to an embodiment of the disclosure;

FIG. 6 shows a block diagram of a training apparatus for a keypoint detection model according to an embodiment of the present disclosure;

FIG. 7 shows a block diagram of a keypoint detection apparatus according to an embodiment of the present disclosure; and

FIG. 8 illustrates a block diagram of an exemplary electronic device that can be used to implement embodiments of the present disclosure.

Detailed Description

Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, in which various details of the embodiments of the disclosure are included to assist understanding, and which are to be considered as merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope of the present disclosure. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

In the present disclosure, unless otherwise specified, the use of the terms "first", "second", etc. to describe various elements is not intended to limit the positional relationship, the timing relationship, or the importance relationship of the elements, and such terms are used only to distinguish one element from another. In some examples, a first element and a second element may refer to the same instance of the element, and in some cases, based on the context, they may also refer to different instances.

The terminology used in the description of the various described examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context clearly indicates otherwise, if the number of elements is not specifically limited, the elements may be one or more. Furthermore, the term "and/or" as used in this disclosure is intended to encompass any and all possible combinations of the listed items.

The key point detection technology is widely applied to various scenes. For example, in an Augmented Reality (AR) virtual shoe fitting scene, an image of a user's foot may be captured, keypoint detection may be performed on the image, keypoints of the foot may be identified therefrom, and then a shoe of interest to the user may be rendered onto the image of the user's foot based on the identified keypoints of the foot and a determination of the posture of the user's foot to show the wearing effect of the shoe to the user. For another example, in an automatic driving scenario, the key point detection may be performed on an image captured by a vehicle, and then the key points of target objects such as lane guide arrows, parking spaces, and obstacles may be identified, and then based on the identified key points, functions such as automatic lane changing, autonomous parking, and obstacle avoidance may be implemented.

In the related art, a key point detection technique based on deep learning is generally employed to detect key points of a target object in an image. That is, the image is input to the keypoint detection model, which outputs keypoints of the target object in the image. However, it often happens that part of the keypoints of the target object are obscured from view by other objects in the image (but the keypoints are still in the image), or part of the keypoints of the target object are truncated (i.e. the keypoints are not in the image). For example, in a virtual shoe fitting scenario, ankle key points in the user's foot image are likely to be occluded by shoes, pants legs, or the user's limbs currently worn by the user; the user's toe keypoints may be truncated, i.e., not captured in the image. For another example, in an automatic driving scenario, some key points of the guiding arrows in the road surface image captured by the vehicle may be blocked by other vehicles or floating objects (e.g., paper pieces, leaves, etc.) on the road surface. Under the above conditions, the key points of the model output are usually large in error and not accurate enough.

In order to solve the above problems, the present disclosure provides a method for training a keypoint detection model and a method for detecting keypoints, so as to improve accuracy of keypoint detection. Under the condition that part of key points in the image are shielded, the positions of the shielded key points can be accurately detected; under the condition that part of the key points are cut off, the positions of the key points (namely, the key points which are not cut off) in the image can be accurately detected.

Embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings.

Fig. 1 illustrates a schematic diagram of an exemplary system 100 in which various methods and apparatus described herein may be implemented in accordance with embodiments of the present disclosure. Referring to fig. 1, the system 100 includes one or

more client devices

101, 102, 103, 104, 105, and 106, a server 120, and one or more communication networks 110 coupling the one or more client devices to the server 120.

Client devices

101, 102, 103, 104, 105, and 106 may be configured to execute one or more applications.

In embodiments of the present disclosure, the server 120 may run one or more services or software applications that enable the execution of the training method of the keypoint detection model and/or the keypoint detection method.

In some embodiments, the server 120 may also provide other services or software applications that may include non-virtual environments and virtual environments. In certain embodiments, these services may be provided as web-based services or cloud services, for example, provided to users of

client devices

101, 102, 103, 104, 105, and/or 106 under a software as a service (SaaS) model.

In the configuration shown in fig. 1, server 120 may include one or more components that implement the functions performed by server 120. These components may include software components, hardware components, or a combination thereof, which may be executed by one or more processors. A user operating a

client device

101, 102, 103, 104, 105, and/or 106 may, in turn, utilize one or more client applications to interact with the server 120 to take advantage of the services provided by these components. It should be understood that a variety of different system configurations are possible, which may differ from system 100. Accordingly, fig. 1 is one example of a system for implementing the various methods described herein and is not intended to be limiting.

The user may navigate using

client devices

101, 102, 103, 104, 105, and/or 106. The client device may provide an interface that enables a user of the client device to interact with the client device. The client device may also output information to the user via the interface. Although fig. 1 depicts only six client devices, those skilled in the art will appreciate that any number of client devices may be supported by the present disclosure.

Client devices

101, 102, 103, 104, 105, and/or 106 may include various types of computer devices, such as portable handheld devices, general purpose computers (such as personal computers and laptops), workstation computers, wearable devices, smart screen devices, self-service terminal devices, service robots, gaming systems, thin clients, various messaging devices, sensors or other sensing devices, and so forth. These computer devices may run various types and versions of software applications and operating systems, such as MICROSOFT Windows, APPLE iOS, UNIX-like operating systems, Linux, or Linux-like operating systems (e.g., GOOGLE Chrome OS); or include various Mobile operating systems, such as MICROSOFT Windows Mobile OS, iOS, Windows Phone, Android. Portable handheld devices may include cellular telephones, smart phones, tablets, Personal Digital Assistants (PDAs), and the like. Wearable devices may include head-mounted displays (such as smart glasses) and other devices. The gaming system may include a variety of handheld gaming devices, internet-enabled gaming devices, and the like. The client device is capable of executing a variety of different applications, such as various Internet-related applications, communication applications (e.g., email applications), Short Message Service (SMS) applications, and may use a variety of communication protocols.

Network 110 may be any type of network known to those skilled in the art that may support data communications using any of a variety of available protocols, including but not limited to TCP/IP, SNA, IPX, etc. By way of example only, one or more networks 110 may be a Local Area Network (LAN), an ethernet-based network, a token ring, a Wide Area Network (WAN), the internet, a virtual network, a Virtual Private Network (VPN), an intranet, an extranet, a Public Switched Telephone Network (PSTN), an infrared network, a wireless network (e.g., bluetooth, Wi-Fi), and/or any combination of these and/or other networks.

The server 120 may include one or more general purpose computers, special purpose server computers (e.g., PC (personal computer) servers, UNIX servers, mid-end servers), blade servers, mainframe computers, server clusters, or any other suitable arrangement and/or combination. The server 120 may include one or more virtual machines running a virtual operating system, or other computing architecture involving virtualization (e.g., one or more flexible pools of logical storage that may be virtualized to maintain virtual storage for the server). In various embodiments, the server 120 may run one or more services or software applications that provide the functionality described below.

The computing units in server 120 may run one or more operating systems including any of the operating systems described above, as well as any commercially available server operating systems. The server 120 may also run any of a variety of additional server applications and/or middle tier applications, including HTTP servers, FTP servers, CGI servers, JAVA servers, database servers, and the like.

In some implementations, the server 120 may include one or more applications to analyze and consolidate data feeds and/or event updates received from users of the

client devices

101, 102, 103, 104, 105, and 106. Server 120 may also include one or more applications to display data feeds and/or real-time events via one or more display devices of

client devices

101, 102, 103, 104, 105, and 106.

In some embodiments, the server 120 may be a server of a distributed system, or a server incorporating a blockchain. The server 120 may also be a cloud server, or a smart cloud computing server or a smart cloud host with artificial intelligence technology. The cloud Server is a host product in a cloud computing service system, and is used for solving the defects of high management difficulty and weak service expansibility in the traditional physical host and Virtual Private Server (VPS) service.

The system 100 may also include one or more databases 130. In some embodiments, these databases may be used to store data and other information. For example, one or more of the databases 130 may be used to store information such as music files. The database 130 may reside in various locations. For example, the database used by the server 120 may be local to the server 120, or may be remote from the server 120 and may communicate with the server 120 via a network-based or dedicated connection. The database 130 may be of different types. In certain embodiments, the database used by the server 120 may be, for example, a relational database or a non-relational database. One or more of these databases may store, update, and retrieve data to and from the database in response to the command.

In some embodiments, one or more of the databases 130 may also be used by applications to store application data. The databases used by the application may be different types of databases, such as key-value stores, object stores, or regular stores supported by a file system.

The system 100 of fig. 1 may be configured and operated in various ways to enable application of the various methods and apparatus described in accordance with the present disclosure.

FIG. 2 shows a flow diagram of a method 200 for training a keypoint detection model, according to an embodiment of the disclosure. Method 200 is typically performed at a server (e.g., server 120 shown in FIG. 1) and may also be performed at a client device (e.g.,

client devices

101, 102, 103, 104, 105, and 106 shown in FIG. 1). That is, the execution subject of each step of the method 200 may be the server 120 shown in fig. 1, or the

client devices

101, 102, 103, 104, 105, and 106.

As shown in FIG. 2, the method 200 includes steps 210-250.

In step 210, a sample image of a target object is acquired, the target object including a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points.

In step 220, the sample image is input into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model.

In step 230, at least one prediction assistance point is determined based on the plurality of prediction key points and a preset positional relationship.

In step 240, a loss value for the keypoint detection model is determined based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point.

In step 250, parameters of the keypoint detection model are adjusted based on the loss values.

According to the embodiment of the present disclosure, in addition to labeling the key points of the target object, auxiliary points having a certain positional relationship with the key points are labeled. Both the key points and the auxiliary points participate in the calculation of the loss values. Because the auxiliary points and the key points have a certain position relationship, the spatial position information of the key points can be expressed through the auxiliary points, the learning of the model on the key point positions is strengthened, and the accuracy of key point detection is improved.

According to an embodiment of the present disclosure, the plurality of key points are points that can express a feature of the target object, and the at least one auxiliary point is determined based on the plurality of key points and a preset positional relationship. The preset position relationship between the auxiliary point and the key point may be various, for example, the auxiliary point may be a pixel point located at a preset position of a straight line or a line segment formed by connecting two key points, or any point different from a plurality of key points on a line segment formed by connecting two key points, and the like.

According to some embodiments, a plurality of key points and at least one auxiliary point labeled in a sample image may be used as true values, a plurality of predicted key points output by a key point detection model and at least one predicted auxiliary point determined based on the plurality of predicted key points may be used as predicted values, and a loss value of the key point detection model may be calculated by comparing the predicted values with the true values. Based on the calculated loss values, parameters of the keypoint detection model may be adjusted using, for example, a back propagation algorithm, resulting in a trained keypoint detection model.

The target object, the sample image, the plurality of key points, the preset positional relationship, and the at least one auxiliary point of the embodiment of the present disclosure are explained in detail below.

In the embodiments of the present disclosure, the target object refers to an object to be subjected to keypoint detection. For example, in a virtual shoe fit scenario, the target object may be a user's foot. In an autonomous driving scenario, the target object may be a directional arrow, a parking space, an obstacle, or the like.

The sample image is an image containing the target object. For example, in a virtual shoe fitting scenario, the sample image may be an image containing the user's foot. In an autonomous driving scenario, the sample image may be a road surface image containing a guide arrow.

Fig. 3 shows a schematic diagram of a sample image 300 according to an embodiment of the disclosure. As shown in fig. 3, the sample image 300 is a road surface image that contains a guide arrow 310 (i.e., a target object).

In the embodiments of the present disclosure, the key points are points that can express the features of the target object. The keypoints are usually located at specific parts of the target object. The number of key points and the corresponding location of the target object may be set by a person skilled in the art according to the actual application scenario. For example, in a virtual shoe fitting scenario, the target object may be a user's foot, and the key points may include a medial ankle joint point, a lateral ankle joint point, a heel point, a toe point, a connection point of the instep to the leg, and so forth.

In an embodiment of the present disclosure, the sample image is labeled with a plurality of key points of the target object.

According to some embodiments, the sample image is also labeled with neighboring relationships between the plurality of keypoints. For example, in an application scenario of human body posture detection, key points of a human body (i.e., a target object) include a shoulder joint point, an elbow joint point, a wrist joint point, a knee joint point, and the like, wherein the shoulder joint point is adjacent to the elbow joint point, and the elbow joint point is adjacent to the wrist joint point.

In an embodiment of the present disclosure, the sample image is further marked with at least one auxiliary point having a preset positional relationship with the plurality of key points. The number of assist points may be determined by one skilled in the art with reference to the actual application scenario, and the present disclosure is not limited thereto.

In an embodiment of the present disclosure, the at least one auxiliary point is determined based on the plurality of key points and a preset positional relationship. The preset position relationship between the auxiliary point and the key point can be various. The present disclosure does not limit the specific form of the preset positional relationship as long as at least one auxiliary point can be determined from the preset positional relationship and the given plurality of key points.

Fig. 4A-4D show schematic views of auxiliary points derived from four different preset positional relationships given the keypoint A, B, C, D of the target object 410. The manner in which the assist point is determined in fig. 4A-4D will be described below in conjunction with corresponding embodiments.

According to some embodiments, any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints. Specifically, one auxiliary point may be located at an arbitrary specified position on a straight line formed by two key points, for example, at a specified position (e.g., a midpoint) on a line segment formed by two key points, or at a specified position on an extension line of a line segment formed by two key points. Based on the above embodiments, the positioning and calculation of the auxiliary points can be facilitated, i.e. based on a given plurality of keypoints, individual auxiliary points can be determined quickly.

It should be noted that, in the embodiment of the present disclosure, the two corresponding keypoints for connecting to form a straight line (or a line segment) may be any two keypoints of a plurality of keypoints. In the case where the adjacent relationship between the plurality of key points is marked in the sample image, the respective two key points for connecting to form a straight line may also be the adjacent two key points.

Fig. 4A is a schematic diagram illustrating key points and auxiliary points obtained by labeling a target object in a sample image according to the above embodiment. The key points are represented by solid rectangular points and capital letters, and the auxiliary points are represented by shaded rectangular points and lowercase letters. As shown in fig. 4A, the guide arrow 410 (i.e., the target object) includes four key points A, B, C, D and two auxiliary points a, b. The auxiliary point a is located on the extension of the line segment BD formed by the key point B, D, and the distance to the key point B is 1/2 of the length of the line segment BD. The auxiliary point b is located at the midpoint of the line segment BC formed by the key point B, C.

According to some embodiments, on the basis of the above embodiments, further, any one of the at least one auxiliary point is located on a line segment formed by the respective two keypoints of the plurality of keypoints, rather than on an extension of the line segment. Therefore, all the auxiliary points are located on the line segment formed by connecting the key points, so that the positioning and calculation of the auxiliary points are more convenient, and the auxiliary points are prevented from overflowing the image (namely being located outside the image).

According to some embodiments, the auxiliary point may be located at a preset position of the line segment, based on the auxiliary point being located on the line segment formed by the two key points. That is, according to some embodiments, any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints. The preset position may be, for example, the midpoint of the line segment, or 1/3 line segment lengths, 1/4 line segment lengths, etc. from a specified keypoint of the two keypoints. Therefore, each pair of corresponding key points in the plurality of key points can generate one auxiliary point, and the auxiliary points are uniformly distributed among the plurality of key points, so that the auxiliary points can reasonably and effectively express the spatial position information of the key points, the learning effect of the model on the key point positions is improved, and the accuracy of key point detection is improved. Also, based on this embodiment, the number of auxiliary points of the target object in different sample images is fixed (the same as the number of key point pairs) and less, so that the calculation efficiency of key point detection can be improved.

Fig. 4B is a schematic diagram illustrating key points and auxiliary points obtained by labeling the target object in the sample image according to the above embodiment. As shown in fig. 4B, the guide arrow 410 (i.e., the target object) includes four keypoints A, B, C, D, and the neighboring relationship between the four keypoints is labeled in the sample image, i.e., a is adjacent to B, B is adjacent to C, and B is adjacent to D. And connecting adjacent key points to obtain three line segments, namely line segments AB, BC and BD. As shown in fig. 4B, the guide arrow 410 further includes three auxiliary points c, d, e, respectively located at the midpoints of the line segments AB, BC, BD.

According to other embodiments, on the basis that the auxiliary point is located on the line segment formed by the two key points, further, all the pixel points on the line segment except the key points can be used as the auxiliary points. That is, at least one auxiliary point included in the target object is a pixel point different from the plurality of key points on a plurality of line segments formed by connecting the plurality of key points two by two. Therefore, the number of the auxiliary points of the target object in different sample images is not fixed (because the number of the pixel points on each line segment in different sample images is not fixed) and is large, and the spatial position information of the key points can be more comprehensively, abundantly and flexibly expressed (compared with the embodiment described above with reference to fig. 4B), so that the learning effect of the model on the key point positions is improved, and the accuracy of key point detection is improved.

Fig. 4C is a schematic diagram illustrating key points and auxiliary points obtained by labeling the target object in the sample image according to the above embodiment. As shown in fig. 4C, the guide arrow 410 (i.e., the target object) includes four keypoints A, B, C, D. The four keypoints are connected two by two to obtain six line segments, i.e., line segments AB, AC, AD, BC, BD, CD (since the keypoints A, B, C are collinear at three points, line segment AC is covered by line segments AB, BC in fig. 4C). All the pixel points on the six line segments except the key point A, B, C, D are auxiliary points. In order to make the drawings more concise and clearer, the auxiliary points are not respectively drawn in fig. 4C, but only line segments where the auxiliary points are located are drawn.

According to other embodiments, on the basis that the auxiliary points are located on the line segment formed by the two key points and the sample image is labeled with the neighboring relationship among the plurality of key points, further, all the pixel points on the line segment formed by the neighboring key points except the key points may be used as the auxiliary points. That is, in a case where the sample image is labeled with the adjacent relationship between the plurality of key points, the at least one auxiliary point included in the target object is a pixel point different from the plurality of key points on a plurality of line segments formed by connecting adjacent key points among the plurality of key points. Therefore, the spatial position information of the key points can be comprehensively and pertinently expressed, the accuracy of key point detection is improved, meanwhile, the calculation amount is reduced (compared with the embodiment described above with reference to fig. 4C), the calculation efficiency is improved, the balance between the accuracy and the calculation efficiency is realized, and the accurate and real-time detection of the key points can be realized.

Fig. 4D is a schematic diagram illustrating key points and auxiliary points obtained by labeling the target object in the sample image according to the above embodiment. As shown in fig. 4D, the guide arrow 410 (i.e., the target object) includes four keypoints A, B, C, D, and the neighboring relationship between the four keypoints is labeled in the sample image, i.e., a is adjacent to B, B is adjacent to C, and B is adjacent to D. And connecting adjacent key points to obtain three line segments, namely line segments AB, BC and BD. All the pixel points on the three line segments except the key point A, B, C, D are auxiliary points. In order to make the drawings more concise and clearer, each auxiliary point is not drawn in fig. 4D, and only a line segment where the auxiliary point is located is drawn. According to the embodiment of the disclosure, the sample image is input into the key point detection model, and a plurality of predicted key points output by the key point detection model can be obtained.

It should be understood that the plurality of predicted keypoints output by the keypoint detection model correspond to the plurality of keypoints labeled in the sample image, respectively. That is, each predicted keypoint output by the keypoint detection model corresponds to a labeled keypoint. 4A-4D above, the labeled keypoint is keypoint A, B, C, D, and the keypoint detection model outputs four predicted keypoints A ', B', C ', D'. The predicted keypoints A ', B', C ', D' correspond to the labeled keypoints A, B, C, D, respectively.

In addition, it should be understood that the plurality of predicted key points output by the key point detection model are predicted values, and the plurality of key points labeled in the sample image are real values.

The keypoint detection model may be any neural network model, and the specific structure of the keypoint detection model is not limited by the present disclosure.

In some embodiments, the keypoint detection model may be, for example, a neural network model composed of a feature extraction module, a thermodynamic diagram generation module, and a keypoint output module. The characteristic extraction module is used for extracting image characteristics of the sample image. The thermodynamic diagram generation module generates a key point thermodynamic diagram corresponding to the sample image based on the image features, wherein the key point thermodynamic diagram is used for representing the confidence degree that each pixel point in the sample image is the key point of the target object. The key point output module determines and outputs a plurality of key points of the target object based on the key point thermodynamic diagram.

After obtaining the plurality of predicted keypoints output by the keypoint detection model, at least one prediction auxiliary point may be determined based on the plurality of predicted keypoints and a preset positional relationship.

Determining at least one predicted auxiliary point based on the predicted key point and the preset position relationship is similar to the above-described process of determining at least one auxiliary point based on the key point and the preset position relationship, and is not repeated here.

After obtaining the plurality of predicted keypoints and the at least one predicted auxiliary point, a loss value for the keypoint detection model may be determined based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point.

According to some embodiments, determining the loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point comprises: generating a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point; generating a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and determining a loss value of the key point detection model based on the tag image and the predicted image. The loss value determination manner of this embodiment may be applied to any preset positional relationship of the key point and the auxiliary point, that is, may be applied to any of the embodiments described above with reference to fig. 4A to 4D.

In the above embodiment, the sizes of the label image and the prediction image (i.e., the number of pixels included in the horizontal direction and the vertical direction) are the same as those of the sample image. Also, the tag image is used to indicate the positions of the key points and the auxiliary points (true values) in the sample image, and the prediction image is used to indicate the positions of the prediction key points and the prediction auxiliary points (predicted values) in the sample image.

According to some embodiments, in the label image, the pixel values of the plurality of key points and the at least one auxiliary point are a first preset value (e.g., 1), and the pixel values of other pixel points except the plurality of key points and the at least one auxiliary point are a second preset value (e.g., 0). In the predicted image, the pixel values of the plurality of prediction key points and the at least one prediction auxiliary point are a first preset value (e.g., 1), and the pixel values of other pixel points than the plurality of prediction key points and the at least one prediction auxiliary point are a second preset value (e.g., 0).

After the tag image and the prediction image are obtained, a loss value of the model can be calculated based on the tag image and the prediction image. According to some embodiments, the loss value of the model may be, for example, a Mean Square Error (MSE), a Mean Absolute Error (MAE), or the like of the tag image and the predicted image.

According to further embodiments, in the case that the plurality of key points correspond to the plurality of predicted key points, respectively, and the at least one auxiliary point corresponds to the at least one predicted auxiliary point, respectively, determining the loss value of the key point detection model based on the plurality of key points, the at least one auxiliary point, the plurality of predicted key points, and the at least one predicted auxiliary point may also include: a loss value of the keypoint detection model is determined based on a first distance of each keypoint from the corresponding predicted keypoint and a second distance of each auxiliary point from the corresponding predicted auxiliary point. For example, the loss value of the model may be an average of the first distances and the second distances. The loss value determination manner of this embodiment is applicable only to the case where the number of auxiliary points is fixed, that is, only to the embodiment described above with reference to fig. 4B.

Based on the determined loss value, parameters of the keypoint detection model may be adjusted.

According to some embodiments, a back propagation algorithm may be employed to adjust the parameters of the keypoint detection model.

It should be understood that, the steps of the method for training the keypoint detection model according to the embodiment of the present disclosure may be executed repeatedly until a preset termination condition is reached (e.g., the loss value is less than the threshold value, the number of cycles reaches a preset maximum number of cycles, etc.), and the training process of the model is ended, so as to obtain a trained keypoint detection model.

According to the embodiment of the disclosure, a key point detection method is further provided based on the key point detection model obtained by the method 200 training.

Fig. 5 shows a flow diagram of a keypoint detection method 500 according to an embodiment of the disclosure. The method 500 is typically performed at a server (e.g., server 120 shown in FIG. 1) and may also be performed at a client device (e.g.,

client device

101 and 106 shown in FIG. 1). That is, the execution subject of each step of the method 500 may be the server 120 shown in fig. 1, or may be the

client device

101 and 106.

In particular, according to some embodiments, the keypoint detection model trained based on the method 200 may be deployed at a server. The user can upload the image to be detected of the target object to the server through the client device, and the server executes the key point detection method 500 of the embodiment of the disclosure based on the key point detection model to obtain a plurality of key points of the target object, and then returns the key points to the client device.

According to other embodiments, the keypoint detection model trained based on the method 200 may also be deployed at the client device. Accordingly, the user may capture (shoot) or designate the image to be detected of the target object through the client device. The client device executes the keypoint detection method 500 of the embodiment of the present disclosure based on the locally deployed keypoint detection model to obtain multiple keypoints of the target object.

As shown in fig. 5, method 500 includes step 510 and step 520.

In step 510, an image to be detected of the target object is input into a key point detection model, wherein the key point detection model is obtained based on the training method of the key point detection model of the embodiment of the present disclosure.

In step 520, a plurality of keypoints of the target object output by the keypoint detection model are obtained.

According to the embodiment of the disclosure, the accurate detection of the key point can be realized.

According to the embodiment of the disclosure, a training device of the key point detection model is also provided. Fig. 6 shows a block diagram of a training apparatus 600 for a keypoint detection model according to an embodiment of the disclosure. As shown in fig. 6, the apparatus 600 includes:

an obtaining module 610 configured to obtain a sample image of a target object, the target object including a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points;

an input/output module 620 configured to input the sample image into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model;

a first determining module 630 configured to determine at least one prediction auxiliary point based on the plurality of prediction key points and the preset positional relationship;

a second determining module 640 configured to determine a loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and

an adjusting module 650 configured to adjust parameters of the keypoint detection model based on the loss values.

According to some embodiments, any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints.

According to some embodiments, any one of the at least one auxiliary point is located on a line segment formed by respective two keypoints of the plurality of keypoints.

According to some embodiments, any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints.

According to some embodiments, the at least one auxiliary point is a pixel point different from the plurality of key points on a plurality of line segments formed by connecting the plurality of key points two by two.

According to some embodiments, the sample image is labeled with neighboring relationships between the plurality of keypoints, and wherein the at least one auxiliary point is a pixel point different from the plurality of keypoints on a plurality of line segments formed by connecting neighboring keypoints of the plurality of keypoints.

According to some embodiments, the second determining module 640 comprises: a first generating unit configured to generate a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point; a second generating unit configured to generate a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and a determination unit configured to determine a loss value of the key point detection model based on the tag image and the prediction image.

According to some embodiments, in the label image, the pixel values of the plurality of key points and the at least one auxiliary point are a first preset value, and the pixel values of other pixel points except the plurality of key points and the at least one auxiliary point are a second preset value; in the predicted image, the pixel values of the plurality of prediction key points and the at least one prediction auxiliary point are the first preset value, and the pixel values of other pixel points except the plurality of prediction key points and the at least one prediction auxiliary point are the second preset value.

According to some embodiments, the second determination module 640 is further configured to determine a loss value of the keypoint detection model based on a first distance of each keypoint from the corresponding predicted keypoint and a second distance of each auxiliary point from the corresponding predicted auxiliary point.

According to the embodiment of the disclosure, a key point detection device is also provided. Fig. 7 shows a block diagram of a keypoint detection apparatus 700 according to an embodiment of the disclosure. As illustrated in fig. 7, the apparatus 700 includes:

an input module 710 configured to input an image to be detected of a target object into a keypoint detection model, wherein the keypoint detection model is obtained according to a training device of the keypoint detection model of the embodiment of the present disclosure; and

an obtaining module 720 configured to obtain a plurality of key points of the target object output by the key point detection model.

According to the embodiment of the disclosure, the accurate detection of the key points can be realized.

It should be understood that the various modules or units of the apparatus 600 shown in fig. 6 may correspond to the various steps in the method 200 described with reference to fig. 2, and the various modules or units of the apparatus 700 shown in fig. 7 may correspond to the various steps in the method 500 described with reference to fig. 5. Thus, the operations, features and advantages described above with respect to method 200 are equally applicable to apparatus 600 and the modules and units included therein, and the operations, features and advantages described above with respect to method 500 are equally applicable to apparatus 700 and the modules and units included therein. Certain operations, features and advantages may not be described in detail herein for the sake of brevity.

Although specific functionality is discussed above with reference to particular modules, it should be noted that the functionality of the various modules discussed herein may be divided into multiple modules and/or at least some of the functionality of multiple modules may be combined into a single module. For example, the first determination module 630 and the second determination module 640 described above may be combined into a single module in some embodiments.

It should also be appreciated that various techniques may be described herein in the general context of software, hardware elements, or program modules. The various modules described above with respect to fig. 6, 7 may be implemented in hardware or in hardware in combination with software and/or firmware. For example, the modules may be implemented as computer program code/instructions configured to be executed in one or more processors and stored in a non-transitory computer readable storage medium. Alternatively, the modules may be implemented as hardware logic/circuitry. For example, in some embodiments, one or more of the modules 610-720 may be implemented together in a System on Chip (SoC). The SoC may include an integrated circuit chip (which includes one or more components of a Processor (e.g., a Central Processing Unit (CPU), microcontroller, microprocessor, Digital Signal Processor (DSP), etc.), memory, one or more communication interfaces, and/or other circuitry), and may optionally execute received program code and/or include embedded firmware to perform functions.

According to an embodiment of the present disclosure, there is also provided an electronic apparatus including: at least one processor; and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the above-described method of keypoint detection model training and/or method of keypoint detection.

There is also provided, in accordance with an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the above-described method for training a keypoint detection model and/or the keypoint detection method.

There is also provided, in accordance with an embodiment of the present disclosure, a computer program product, including a computer program, wherein the computer program, when executed by a processor, implements the above-described method of training a keypoint detection model and/or the keypoint detection method.

Referring to fig. 8, a block diagram of a structure of an electronic device 800, which may be a server or a client of the present disclosure, which is an example of a hardware device that may be applied to aspects of the present disclosure, will now be described. Electronic device is intended to represent various forms of digital electronic computer devices, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the disclosure described and/or claimed herein.

As shown in fig. 8, the electronic device 800 includes a computing unit 801 that can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)802 or a computer program loaded from a storage unit 808 into a Random Access Memory (RAM) 803. In the RAM803, various programs and data required for the operation of the electronic apparatus 800 can also be stored. The calculation unit 801, the ROM 802, and the RAM803 are connected to each other by a bus 804. An input/output (I/O) interface 805 is also connected to bus 804.

A number of components in the electronic device 800 are connected to the I/O interface 805, including: an input unit 806, an output unit 807, a storage unit 808, and a communication unit 809. The input unit 806 may be any type of device capable of inputting information to the electronic device 800, and the input unit 806 may receive input numeric or character information and generate key signal inputs related to user settings and/or function controls of the electronic device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a track pad, a track ball, a joystick, a microphone, and/or a remote controller. Output unit 807 can be any type of device capable of presenting information and can include, but is not limited to, a display, speakers, a video/audio output terminal, a vibrator, and/or a printer. The storage unit 808 may include, but is not limited to, a magnetic disk, an optical disk. The communication unit 809 allows the electronic apparatus 800 to pass through a computer such as the internetThe network and/or various telecommunications networks exchange information/data with other devices and may include, but are not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and/or chipsets, such as bluetooth^TMDevices, 802.11 devices, Wi-Fi devices, WiMAX devices, cellular communication devices, and/or the like.

Computing unit 801 may be a variety of general and/or special purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and the like. The computing unit 801 performs the various methods and processes described above, such as the method 200 and/or the method 500. For example, in some embodiments, method 200 and/or method 500 may be implemented as a computer software program tangibly embodied on a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and/or installed onto the electronic device 800 via the ROM 802 and/or the communication unit 809. When loaded into RAM803 and executed by computing unit 801, may perform one or more of the steps of method 200 and method 500 described above. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the method 200 and/or the method 500 in any other suitable manner (e.g., by way of firmware).

Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), Complex Programmable Logic Devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.

Program code for implementing the methods of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.

In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user may provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), and the Internet.

The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server of a distributed system, or a server with a combined blockchain.

It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present disclosure may be performed in parallel, sequentially or in different orders, and are not limited herein as long as the desired results of the technical solutions disclosed in the present disclosure can be achieved.

Although embodiments or examples of the present disclosure have been described with reference to the accompanying drawings, it is to be understood that the above-described methods, systems and apparatus are merely exemplary embodiments or examples and that the scope of the present invention is not limited by these embodiments or examples, but only by the claims as issued and their equivalents. Various elements in the embodiments or examples may be omitted or may be replaced with equivalents thereof. Further, the steps may be performed in an order different from that described in the present disclosure. Further, the various elements in the embodiments or examples may be combined in various ways. It is important that as technology evolves, many of the elements described herein may be replaced with equivalent elements that appear after the present disclosure.

Claims

1. A method for training a keypoint detection model comprises the following steps:

acquiring a sample image of a target object, wherein the target object comprises a plurality of key points and at least one auxiliary point having a preset position relationship with the key points;

inputting the sample image into the key point detection model to obtain a plurality of predicted key points output by the key point detection model;

determining at least one prediction auxiliary point based on the plurality of prediction key points and the preset position relation;

determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and

adjusting parameters of the keypoint detection model based on the loss value.

2. The method of claim 1, wherein any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints.

3. The method of claim 2, wherein any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints.

4. The method of claim 2, wherein the at least one auxiliary point is a pixel point different from the plurality of key points on a plurality of line segments formed by connecting the plurality of key points two by two.

5. The method of claim 2, wherein the sample image is labeled with neighboring relationships between the plurality of keypoints, and wherein the at least one auxiliary point is a pixel point on a plurality of line segments formed by connecting neighboring keypoints of the plurality of keypoints that is different from the plurality of keypoints.

6. The method of any of claims 1-5, wherein determining a loss value for the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point comprises:

generating a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point;

generating a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and

determining a loss value of the keypoint detection model based on the tag image and the predicted image.

7. The method of claim 6, wherein, in the label image, the pixel values of the plurality of key points and the at least one auxiliary point are a first preset value, and the pixel values of other pixel points except the plurality of key points and the at least one auxiliary point are a second preset value;

in the predicted image, the pixel values of the plurality of prediction key points and the at least one prediction auxiliary point are the first preset value, and the pixel values of other pixel points except the plurality of prediction key points and the at least one prediction auxiliary point are the second preset value.

8. The method of claim 3, wherein determining a loss value for the keypoint detection model, based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point, comprises:

determining a loss value of the keypoint detection model based on a first distance of each keypoint from a corresponding predicted keypoint and a second distance of each auxiliary point from a corresponding predicted auxiliary point.

9. A keypoint detection method comprising:

inputting an image to be detected of a target object into a key point detection model, wherein the key point detection model is obtained according to the method of any one of claims 1-8; and

and acquiring a plurality of key points of the target object output by the key point detection model.

10. A training apparatus for a keypoint detection model, comprising:

an acquisition module configured to acquire a sample image of a target object, wherein the target object includes a plurality of key points and at least one auxiliary point having a preset positional relationship with the plurality of key points;

an input-output module configured to input the sample image into the keypoint detection model to obtain a plurality of predicted keypoints output by the keypoint detection model;

a first determination module configured to determine at least one prediction assistance point based on the plurality of prediction key points and the preset positional relationship;

a second determination module configured to determine a loss value of the keypoint detection model based on the plurality of keypoints, the at least one auxiliary point, the plurality of predicted keypoints, and the at least one predicted auxiliary point; and

an adjustment module configured to adjust parameters of the keypoint detection model based on the loss values.

11. The apparatus of claim 10, wherein any one of the at least one auxiliary point is located on a straight line formed by respective two keypoints of the plurality of keypoints.

12. The apparatus of claim 11, wherein any one of the at least one auxiliary point is located at a preset position of a line segment formed by respective two keypoints of the plurality of keypoints.

13. The apparatus of claim 11, wherein the sample image is labeled with neighboring relationships between the plurality of keypoints, and wherein the at least one auxiliary point is a pixel point on a plurality of line segments formed by connecting neighboring keypoints of the plurality of keypoints that is different from the plurality of keypoints.

14. The apparatus of any of claims 10-13, wherein the second determining means comprises:

a first generating unit configured to generate a label image corresponding to the sample image based on the plurality of key points and the at least one auxiliary point;

a second generating unit configured to generate a prediction image corresponding to the sample image based on the plurality of prediction key points and the at least one prediction auxiliary point; and

a determining unit configured to determine a loss value of the key point detection model based on the tag image and the prediction image.

15. A keypoint detection apparatus comprising:

an input module configured to input an image to be detected of a target object into a keypoint detection model, wherein the keypoint detection model is obtained by the apparatus according to any one of claims 10-14; and

an obtaining module configured to obtain a plurality of key points of the target object output by the key point detection model.

16. An electronic device, comprising:

at least one processor; and

a memory communicatively coupled to the at least one processor; wherein

The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

17. A non-transitory computer readable storage medium having stored thereon computer instructions for causing a computer to perform the method of any one of claims 1-9.

18. A computer program product comprising a computer program, wherein the computer program realizes the method of any one of claims 1-9 when executed by a processor.