CN112348892A

CN112348892A - Point positioning method and related device and equipment

Info

Publication number: CN112348892A
Application number: CN202011182566.2A
Authority: CN
Inventors: 顾宇俊; 袁璟; 赵亮; 黄宁; 张少霆
Original assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Current assignee: Shanghai Sensetime Intelligent Technology Co Ltd
Priority date: 2020-10-29
Filing date: 2020-10-29
Publication date: 2021-02-09
Also published as: WO2022088729A1

Abstract

The application discloses a point positioning method and a related device and equipment, wherein the point positioning method comprises the following steps: acquiring an image to be positioned containing a target point; detecting a target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map, wherein the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, wherein the first area is larger than the second area; and combining and analyzing the coarse positioning heat degree diagram and the fine positioning heat degree diagram to obtain the position information of the target point. The scheme can be particularly applied to medical images containing lower limb force lines so as to position hip joint central points, knee joint central points, ankle joint central points and the like in the medical images, and the accuracy and the stability of point positioning can be improved.

Description

Point positioning method and related device and equipment

Technical Field

The present application relates to the field of computer vision technologies, and in particular, to a point locating method and related apparatus and device.

Background

In the field of computer vision, the method has great significance for accurately identifying the target point in the image. Taking a medical image as an example, medical staff can be assisted to a great extent by positioning certain target points in a lower limb image, for example, whether a patient has knee valgus or knee varus can be assisted and analyzed according to the relative position of a knee joint center point and a lower limb force line; or the preoperative planning and postoperative evaluation of operations such as high tibial osteotomy and high femoral osteotomy can be analyzed in an auxiliary manner according to the knee joint center point and the lower limb force line.

However, the manual positioning method is time-consuming and labor-consuming, and it is difficult to ensure the accuracy and stability of the point location because the positioning accuracy greatly depends on experience. In view of this, how to improve the accuracy and stability of point positioning becomes an extremely important issue

Disclosure of Invention

The application provides a point positioning method and a related device and equipment.

A first aspect of the present application provides a point localization method, including: acquiring an image to be positioned containing a target point; detecting a target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map, wherein the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, wherein the first area is larger than the second area; and combining and analyzing the coarse positioning heat degree diagram and the fine positioning heat degree diagram to obtain the position information of the target point.

Therefore, the coarse positioning heat map and the fine positioning heat map are obtained by detecting the target point of the acquired image to be positioned, the heat value of the first area containing the target point in the coarse positioning heat map is in the first heat value range, the heat value of the second area containing the target point in the fine positioning heat map is in the second heat value range, and the first area of the coarse positioning heat map is larger than the second area of the fine positioning heat map, so that the coarse positioning heat map can represent the target response in a larger range near the target point, and the fine positioning heat map can represent the target response in a smaller range near the target point.

Wherein, the combination analysis coarse positioning heat degree chart and the fine positioning heat degree chart obtains the position information of the target point, and the method comprises the following steps: and obtaining the confidence degrees of the position of a first target point in the rough positioning heat map and the position of a second target point and the position of the second target point in the fine positioning heat map, and processing the position of the first target point and the position of the second target point based on the confidence degree of the position of the second target point to obtain the position information of the target point.

Therefore, by acquiring the confidence degrees of the first target point position in the coarse positioning heat map and the second target point position in the fine positioning heat map, the first target point position and the second target point position are processed based on the confidence degree of the second target point position to obtain the position information of the target point, and further the position information of the target point can be determined based on the confidence degree conditions of the target points in the coarse positioning heat map and the fine positioning heat map, so that the accuracy of point positioning can be further improved.

The method includes the steps of obtaining the confidence degrees of a first target point position in a coarse positioning heat map and a second target point position in a fine positioning heat map, and processing the first target point position and the second target point position based on the confidence degree of the second target point position to obtain position information of a target point, and includes the following steps: determining the position of a first target point based on the heat value of the rough positioning heat map; determining the confidence of the position of the second target point and the position of the second target point based on the heat value of the fine positioning heat map; and based on the confidence of the position of the second target point, selecting the position of the second target point or the position of the first target point as the position information of the target point.

Therefore, the position of the first target point is determined through the coarse positioning heat map, the confidence degrees of the position of the second target point and the position of the second target point are determined through the fine positioning heat map, the position of the second target point or the position of the first target point is selected based on the confidence degree of the position of the second target point as the position information of the target point, the confidence degree of the position of the second target point is determined based on the fine positioning heat map, and the position information of the target point is selected from the position of the first target point determined from the position of the first target point and the position of the first target point determined through the coarse positioning heat map, so that the stability of point positioning can be improved.

Wherein, based on the confidence of the second target point position, selecting the second target point position or the first target point position as the position information of the target point comprises: if the confidence of the position of the second target point meets the fine positioning confidence condition, taking the position of the second target point as the position information of the target point; and if the confidence coefficient of the second target point position does not meet the fine positioning confidence condition, taking the first target point position as the position information of the target point.

Therefore, when the confidence of the second target point position meets the fine positioning confidence condition, the second target point position is used as the position information of the target point, and when the confidence of the second target point position does not meet the fine positioning confidence condition, the first target point position is used as the position information of the target point, so that the method can be favorable for selecting the position information with better confidence as the target point, and can be favorable for improving the accuracy and the stability of point positioning.

Before determining the confidence degrees of the position of the second target point and the position of the second target point based on the heat value of the fine positioning heat map, the method further comprises the following steps: determining the confidence coefficient of the position of a first target point based on the heat value of the coarse positioning heat map, and if the confidence coefficient of the position of the first target point meets the coarse positioning confidence condition, adjusting the heat value, which is positioned outside a preset distance range of the position of the first target point in the fine positioning heat map, to be a preset heat value; wherein the preset heat value is out of the second heat value range.

Therefore, when the confidence of the first target point position meets the coarse positioning confidence condition, the heat value outside the preset distance range of the first target point position in the fine positioning heat map is directly adjusted to be the preset heat value, so that after the fine positioning heat map is adjusted, the confidence of the second target point position and the second target point position is determined based on the heat value of the fine positioning heat map, the second target point position can be located near the first target point position, and the point positioning accuracy is further improved.

Wherein the coarse positioning confidence condition comprises that the confidence of the position of the first target point is greater than a first preset threshold, and the fine positioning confidence condition comprises at least one of the following: the confidence coefficient of the position of the second target point is greater than a second preset threshold value, and the confidence coefficient of the position of the second target point is greater than that of the position of the first target point.

Therefore, the coarse positioning confidence condition is set to include that the confidence of the first target point position is greater than the first preset threshold, the fine positioning confidence condition is set to include that the confidence of the second target point position is greater than at least one of the second preset threshold and the confidence of the second target point position is greater than the confidence of the first target point position, so that the position information of the target point with better confidence can be screened, and the accuracy and the stability of point positioning can be improved.

Wherein, based on the heat value of the rough positioning heat map, determining the position of a first target point, or based on the heat value of the fine positioning heat map, determining the position of a second target point, comprises: and taking the point with the maximum heat value in the positioning heat map as the target point position, or taking the area preset point in the positioning heat map as the target point position.

Therefore, the point with the largest heat value in the positioning heat map is used as the target point position, or the area preset point in the positioning heat map is used as the target point position, so that the difficulty of determining the target point position can be reduced, and the point positioning speed can be improved.

Wherein, based on the heat value of the rough positioning heat map, the confidence of the position of the first target point is determined, or based on the heat value of the fine positioning heat map, the confidence of the position of the second target point is determined, including: acquiring at least one reference heat value; for each reference heat value, acquiring the size of a reference area with the heat value larger than the reference heat value from the positioning heat map; obtaining the confidence of the position of the target point based on the size of each reference area and the heat value of the position of the target point; or, obtaining a target heat map based on the target point position of the positioning heat map; and obtaining the confidence of the position of the target point based on the similarity of the heat value distribution between the positioning heat map and the target heat map.

Therefore, by acquiring at least one reference heat value and acquiring the size of the reference area with the heat value larger than the reference heat value from the positioning heat map according to each reference heat value, the confidence of the target point position is obtained based on the size of each reference area and the heat value of the target point position, or the confidence of the target point position is obtained based on the distribution similarity of the heat values between the positioning heat map and the target heat map, so that the accuracy of the confidence can be improved, and meanwhile, the subsequent relocation and completion of the area with low confidence can be facilitated.

Wherein obtaining at least one reference calorific value comprises: acquiring at least one multiplying power, and taking the product of the at least one multiplying power and the heat value of the target point position as at least one reference heat value; the reference area comprises a target point position; the size of the reference region includes the perimeter and area of the reference region; the confidence of the position of the target point is obtained based on the size of each reference area and the heat value of the position of the target point, and the confidence comprises the following steps: and acquiring a first proportion between the area of each reference region and the square of the perimeter, and acquiring the confidence of the target point position by using the sum of the first proportions of at least one reference region, the heat value of the target point position and a preset heat peak value.

Therefore, by acquiring at least one multiplying power and taking the product of the at least one multiplying power and the heat value of the target point position as at least one reference heat value, the reference heat value can be conveniently and quickly determined, the speed of point positioning is favorably improved, and by acquiring the first proportion between the area of each reference area and the square of the perimeter, the confidence coefficient of the target point position is obtained by using the sum of the first proportion of the at least one reference area, the heat value of the target point position and the preset heat peak value, the confidence coefficient of the target point position can be accurately determined.

Wherein, the target heat map is obtained based on the target point position of the positioning heat map, and the method comprises the following steps: based on the position of a target point of a positioning heat map, acquiring the heat value of each pixel point in the target heat map by using a two-dimensional Gaussian function, wherein the index of the two-dimensional Gaussian function comprises a range parameter, the absolute value of the index and the range parameter are in a negative correlation relationship, and the range parameter in the two-dimensional Gaussian function corresponding to the coarse positioning heat map is larger than the range parameter in the two-dimensional Gaussian function corresponding to the fine positioning heat map; obtaining the confidence of the position of the target point based on the similarity of the distribution of the heat value between the positioning heat map and the target heat map, wherein the confidence comprises the following steps: and obtaining a correlation coefficient between the positioning heat map and the target heat map as the confidence of the position of the target point based on the heat value distribution between the positioning heat map and the target heat map.

Therefore, the target point position of the heat map is positioned, the heat value of each pixel point in the target heat map is obtained by utilizing the two-dimensional Gaussian function, the index of the two-dimensional Gaussian function comprises the range parameter, the absolute value of the index and the range parameter are in a negative correlation relationship, the range parameter in the two-dimensional Gaussian function corresponding to the coarse positioning heat map is larger than the range parameter in the two-dimensional Gaussian function corresponding to the fine positioning heat map, and therefore the target heat map can be conveniently and accurately obtained, and the correlation coefficient between the positioning heat map and the target heat map is obtained on the basis of the heat value distribution between the positioning heat map and the target heat map and serves as the confidence coefficient of the target point position, and therefore the confidence coefficient of the target point position can be conveniently and accurately obtained.

Before detecting a target point of an image to be positioned and obtaining a coarse positioning heat map and a fine positioning heat map, the method further comprises the following steps: preprocessing an image to be positioned; wherein the pre-treatment comprises at least one of: and normalizing the image to be positioned, and enhancing the image contrast of the image to be positioned.

Therefore, the image to be positioned is preprocessed before the image to be positioned is subjected to target point detection, and the preprocessing comprises the step of normalizing the image to be positioned and/or the step of enhancing the image contrast of the image to be positioned, so that the accuracy of subsequent target point detection can be improved.

After the coarse positioning heat map and the fine positioning heat map are analyzed in combination to obtain the position information of the target point, the method further comprises the following steps: and outputting the position information of the target point and the confidence degree of the corresponding position information.

Therefore, after the position information of the target point is obtained by analyzing the coarse positioning heat map and the fine positioning heat map in a combined manner, the position information of the target point and the confidence degree of the corresponding position information are output, so that the user can evaluate the position information of the target point obtained by positioning, and the user perception is improved.

Wherein, normalizing the image to be positioned comprises: setting a pixel value larger than the first pixel value in the image to be positioned as a first pixel value, and setting a pixel value smaller than the second pixel value in the image to be positioned as a second pixel value; among the sequentially arranged pixel values of the image to be positioned, the pixel value positioned at the first numerical value arrangement position is a first pixel value, and the pixel value positioned at the second numerical value arrangement position is a second pixel value.

Therefore, the pixel value larger than the first pixel value in the image to be positioned is set as the first pixel value, the heat value smaller than the second pixel value in the image to be positioned is set as the second pixel value, and in the pixel values of the sequencing sequence of the image to be positioned, the pixel value positioned in the first numerical value arrangement is the first pixel value, and the pixel value positioned in the second numerical value arrangement is the second pixel value, so that the interference such as extra-bright or extra-dark in the image to be positioned can be favorably eliminated, the accuracy of the detection of a subsequent target point can be favorably improved, and the accuracy of the point positioning can be improved.

Wherein, treat the location image and carry out the target point detection, obtain thick location heat degree picture and thin location heat degree picture, include: and detecting a target point of the image to be positioned by using the deep learning model to obtain a coarse positioning heat map and a fine positioning heat map.

Therefore, the target point detection is carried out on the image to be positioned by utilizing the deep learning model, the hardware can be fully utilized for parallel acceleration, and the complexity of the target point detection is reduced.

Wherein, the deep learning model is a full convolution neural network, and/or the deep learning model is obtained by training at least the following steps: acquiring a sample image, wherein the sample image is marked with real position information of a target point; generating a coarse target heat map and a fine target heat map by using the real position information of the target point; the heat value of a third area containing the target point in the coarse target heat map is in a third heat value range; the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, wherein the third area is larger than the fourth area; detecting target points of the sample images by using a deep learning model to obtain a coarse positioning heat map and a fine positioning heat map; and adjusting network parameters of the deep learning model based on the difference between the coarse target heat map and the coarse positioning heat map and the difference between the fine target heat map and the fine positioning heat map.

Therefore, by using the real position information of the target point, a coarse target heat map and a fine target heat map are generated, the heat value of a third area containing the target point in the coarse target heat map is in a third heat value range, the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, and the third area is larger than the fourth area, so that the target point detection is performed on the sample image by using the deep learning model to obtain a coarse positioning heat map and a fine positioning heat map.

Wherein, the image to be positioned is an X-ray image; in the first area and the second area, the closer to the target point, the higher the heat value; the heat value outside the first region in the coarse positioning heat map is lower than the lower limit value of the first heat value range, and the heat value outside the second region in the fine positioning heat map is lower than the lower limit value of the second heat value range.

Therefore, in the first area and the second area, the higher the heat value closer to the target point is, the lower limit value of the range of the first heat value is lower for the heat value outside the first area in the coarse positioning heat map, and the lower limit value of the range of the second heat value is lower for the heat value outside the second area in the fine positioning heat map.

A second aspect of the present application provides a spot positioning apparatus, comprising: the system comprises an image acquisition module, a target detection module and a position analysis module, wherein the image acquisition module is used for acquiring an image to be positioned containing a target point; the target detection module is used for detecting a target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map, wherein the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, wherein the first area is larger than the second area; and the position analysis module is used for analyzing the coarse positioning heat degree diagram and the fine positioning heat degree diagram in a combined manner to obtain the position information of the target point.

A third aspect of the present application provides an electronic device, comprising a memory and a processor coupled to each other, wherein the processor is configured to execute program instructions stored in the memory to implement the point location method in the first aspect.

A fourth aspect of the present application provides a computer-readable storage medium having stored thereon program instructions that, when executed by a processor, implement the point localization method of the first aspect described above.

According to the scheme, the target point detection is carried out on the acquired image to be positioned to obtain the coarse positioning heat map and the fine positioning heat map, the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range, the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, and the first area of the coarse positioning heat map is larger than the second area of the fine positioning heat map, so that the coarse positioning heat map can represent target response in a larger range near the target point, the fine positioning heat map can represent target response in a smaller range near the target point, and therefore the coarse positioning heat map and the fine positioning heat map are combined and analyzed, the positioning stability of the coarse positioning heat map and the accuracy of the fine positioning heat map can be achieved at the same time, and the accuracy and the stability of point positioning can be improved.

Drawings

FIG. 1 is a schematic flow chart diagram illustrating an embodiment of a method for locating a point of application;

FIG. 2 is a schematic view of one embodiment of the image to be located in FIG. 1;

FIG. 3 is a schematic diagram of one embodiment of the coarse and fine positioning heat maps of FIG. 1;

FIG. 4 is a schematic flow chart diagram of an embodiment of training a deep learning model;

FIG. 5 is a flowchart illustrating an embodiment of step S13 in FIG. 1;

FIG. 6 is a block diagram of an embodiment of a pointing device according to the present application;

FIG. 7 is a block diagram of an embodiment of an electronic device of the present application;

FIG. 8 is a block diagram of an embodiment of a computer-readable storage medium of the present application.

Detailed Description

The following describes in detail the embodiments of the present application with reference to the drawings attached hereto.

In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular system structures, interfaces, techniques, etc. in order to provide a thorough understanding of the present application.

The terms "system" and "network" are often used interchangeably herein. The term "and/or" herein is merely an association describing an associated object, meaning that three relationships may exist, e.g., a and/or B, may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship. Further, the term "plurality" herein means two or more than two.

Referring to fig. 1, fig. 1 is a schematic flowchart illustrating an embodiment of a point locating method according to the present application. Specifically, the method may include the steps of:

step S11: and acquiring an image to be positioned containing the target point.

The image to be positioned can be an image including facial features so as to fix target points of eyes, mouth, nose and the like of the face, so that the image to be positioned can be used for application scenes such as face recognition and the like in the following; or, the image to be positioned may also be an image including human tissues and organs so as to position the target point in the human tissues and organs. For example, the image to be positioned may be an X-ray image, i.e. a computed tomography image, in a specific implementation scenario, please refer to fig. 2, fig. 2 is a schematic diagram of an embodiment of the image to be positioned in fig. 1, the image to be positioned may be an X-ray image of a lower limb (including a left lower limb and a right lower limb) of a human body, and the included target points (black filled dots in fig. 2) may include, but are not limited to: the target points shown in fig. 2 are the 8 target points on the left lower limb and the right lower limb, namely 16 target points in total. In other application scenarios, the same can be said, and no one example is given here.

Step S12: and detecting a target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map.

The heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range, the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, and the first area is larger than the second area. The positioning heat map can reflect target response of each pixel point in the image to be positioned, in an implementation scene, the closer the heat value to the target point is, the higher the target response value is, the heat value outside a first area in the coarse positioning heat map is lower than the lower limit value of a first heat value range, the heat value outside a second area in the fine positioning heat map is lower than the lower limit value of a second heat value range, and the fine positioning heat map has a high response value in a larger range, so that the coarse positioning heat map can ensure the stability of point positioning in the subsequent point positioning process, and the fine positioning heat map can ensure the accuracy of point positioning in the subsequent point positioning process. Referring to fig. 3 in combination, fig. 3 is a schematic diagram of an embodiment of the coarse positioning heat map and the fine positioning heat map in fig. 1, as shown in fig. 3, the left side is the coarse positioning heat map, and the right side is the fine positioning heat map, for convenience of description, the coarse positioning heat map and the fine positioning heat map shown in fig. 3 represent target responses to the same target point, solid dots and white filled areas around the solid dots in the coarse positioning heat map and the fine positioning heat map respectively represent a first area and a second area including the target point, in an embodiment, for convenience of distinction, the heat values may be represented in a sequence of spectral colors, that is, "red" may be used to represent the target point with the largest heat value, and as the heat values decrease, "orange", "yellow", "green", "blue" and the like may represent points away from the target point.

In an implementation scenario, in order to fully utilize parallel acceleration of hardware and reduce the complexity of target point detection, a deep learning model may be used to perform target point detection on an image to be positioned, so as to obtain a coarse positioning heat map and a fine positioning heat map. In one implementation scenario, the deep learning model may employ a Unet network that includes an encoder, a decoder, and a hopping link structure. In an implementation scenario, when the deep learning model detects a target point of an image to be positioned, a corresponding coarse positioning heat map and a corresponding fine positioning heat map can be generated for each target point.

In an implementation scenario, in order to improve the accuracy of target point detection, the image to be positioned may be normalized before the target point detection is performed. Specifically, a pixel value larger than a first pixel value in the image to be positioned may be set as a first pixel value, and a pixel value smaller than a second pixel value in the image to be positioned may be set as a second pixel value, where, among the sequentially arranged pixel values of the image to be positioned, the pixel value located in the first numerical rank is the first pixel value, and the pixel value located in the second numerical rank is the second pixel value. For example, the pixel points of the image to be positioned are sorted from small to large according to the pixel values, the 99 th (i.e. located at 99% of the total) arranged pixel value is taken as a first pixel value, the 3 rd (i.e. located at 3% of the total) arranged pixel value is taken as a second pixel value, the pixel value larger than the first pixel value is set as the first pixel value, and the pixel value smaller than the second pixel value is set as the second pixel value. In other implementation scenarios, the first numerical rank and the second numerical rank may also be set according to a specific application, which is not limited herein.

In an implementation scenario, in order to improve the accuracy of target point detection, the image contrast of the image to be positioned may also be enhanced. Specifically, a Contrast-Limited Adaptive Histogram Equalization (CLAHE) algorithm may be used to enhance the local Contrast of the image.

Step S13: and combining and analyzing the coarse positioning heat degree diagram and the fine positioning heat degree diagram to obtain the position information of the target point.

In an implementation scenario, the first target point position in the coarse positioning heat map and the confidence level of the first target point position may be obtained, and when the confidence level of the first target point position is greater than a confidence level threshold, the first target point position is used as the position information of the target point, so that the position information of the target point can be quickly determined under the condition of low accuracy requirement. Specifically, a point with the highest heat value in the rough positioning heat map may be taken as the first target point position. The confidence of the position of the first target point is used for representing the confidence of the positioned position of the first target point, and the higher the confidence of the position of the first target point is, the higher the confidence of the position of the first target point is. In this application, for convenience of description, if not otherwise specified, the first confidence level is used to indicate the confidence level of the position of the first target point.

In another implementation scenario, a first target point position in the coarse positioning heat map may be further obtained, and a second target point position and a confidence level of the second target point position in the fine positioning heat map are obtained, so that the first target point position and the second target point position are processed based on the confidence level of the second target point position to obtain position information of the target point, and thus accuracy and stability of point positioning can be considered, for example, the first target point position or the second target point position may be selected as the position information of the target point according to the confidence level of the second target point position; or, according to the confidence of the position of the second target point, outputting position information including the position of the first target point and the position of the second target point, which is not limited herein. Specifically, a point with the highest heat value in the coarse positioning heat map may be used as the first target point position, and a point with the highest heat value in the fine positioning heat map may be used as the second target point position. The confidence level of the second target point position is used for representing the confidence level of the located second target point position, and the higher the confidence level of the second target point position is, the higher the confidence level of the second target point position is. In this application, for convenience of description, if not otherwise specified, the second confidence level is used to indicate the confidence level of the position of the second target point.

In yet another implementation scenario, location information of the target point location and a confidence level of the corresponding location information may also be output. For example, when the first target point position is taken as the position information of the target point, the confidence of the first target point may be taken as the confidence of the corresponding position information; or, when the position of the second target point is used as the position information of the target point, the confidence of the second target point can be used as the confidence of the corresponding position information, so that the user can evaluate the position information of the target point obtained by positioning, and the user perception is improved.

Referring to fig. 4, fig. 4 is a flowchart illustrating an embodiment of training a deep learning model, which may specifically include the following steps:

step S41: and acquiring a sample image, wherein the sample image is marked with the real position information of the target point.

The sample image may be an image including facial features, and the target points may include: at least one of eyes, mouth and nose of the human face; alternatively, the sample image may be a video image including human tissue and organs. For example, the sample image may be an X-ray image, i.e. a computed tomography image, in particular, the sample image may be an X-ray image of a lower limb (including a left lower limb and a right lower limb) of a human body, and the target point may include, but is not limited to: the femoral head center, the greater trochanter apex, the femoral medial malleolus, the femoral lateral malleolus, the tibial plateau medial end point, the tibial plateau lateral end point, the ankle joint gap medial end point, and the ankle joint gap lateral end point may refer to the relevant steps in the foregoing embodiments, and are not described herein again.

Step S42: and generating a coarse target heat map and a fine target heat map by using the real position information of the target point.

The heat value of a third area containing the target point in the coarse target heat map is in a third heat value range; the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, wherein the third area is larger than the fourth area.

Specifically, the heat value of each pixel point in the coarse target heat map and the fine target heat map can be obtained by using a two-dimensional gaussian function based on the real position information of the target point in the sample image, wherein the index of the two-dimensional gaussian function includes a range parameter, the absolute value of the index and the range parameter are in a negative correlation relationship, and the range parameter in the two-dimensional gaussian function corresponding to the coarse target heat map is greater than the range parameter in the two-dimensional gaussian function corresponding to the fine target heat map. In a specific implementation scenario, the heat value of each pixel point in the target heat map may be represented as:

in the above formula (1), (x, y) represents the abscissa and ordinate of the pixel point, (x)₀,y₀) Representing the abscissa and ordinate of the target point, e being a natural constant, f (x, y) representing the heat of the pixel pointAnd the value of the degree, sigma, represents a range parameter and is used for controlling the size of the response area on the coarse target heat map and the fine target heat map, and M represents a preset heat peak value and is used for controlling the heat peak value.

Step S43: and detecting target points of the sample images by using a deep learning model to obtain a coarse positioning heat map and a fine positioning heat map.

The deep learning model may be a full Convolutional Neural Network (CNN). And carrying out target detection on the sample image by using the deep learning model so as to obtain a coarse positioning heat map and a fine positioning heat map.

Step S44: and adjusting network parameters of the deep learning model based on the difference between the coarse target heat map and the coarse positioning heat map and the difference between the fine target heat map and the fine positioning heat map.

In one implementation scenario, the difference between the coarse target heat map and the coarse positioning heat map may include: a difference in position between points in the heat map where the heat value is the largest, and a difference in size between regions in the heat map where the heat value is larger than the lower limit value of the third heat range. Specifically, the difference may be processed by using a mean square error function and a cross entropy function, so as to obtain a first loss value corresponding to the difference between the coarse target heat map and the coarse positioning heat map. The difference between the fine target heat map and the fine positioning heat map may include: a difference in position between points in the heat map where the heat value is the largest, and a difference in size between regions in the heat map where the heat value is larger than the lower limit value of the fourth heat range. Specifically, the difference may be processed by using a mean square error function and a cross entropy function, so as to obtain a second loss value corresponding to the difference between the fine target heat map and the fine positioning heat map.

In an implementation scenario, the difference between the coarse target heat map and the coarse positioning heat map, and the difference between the fine target heat map and the fine positioning heat map may be weighted to obtain the total difference. Specifically, the first loss value and the second loss value may be weighted to obtain a loss value of the deep learning model.

In one implementation scenario, the network parameters of the deep learning model may be adjusted by using a Stochastic Gradient Descent (SGD), a Batch Gradient Descent (BGD), a small Batch Gradient Descent (mbi-Batch Gradient Descent, MBGD), and the like. The batch gradient descent refers to updating parameters by using all samples during each iteration; the random gradient descent means that one sample is used for parameter updating in each iteration; the small batch gradient descent means that a batch of samples is used for parameter updating at each iteration, and details are not repeated here. In a specific implementation scenario, the network parameters of the deep learning model may include: weights, biases, etc. of hidden neurons.

In one implementation scenario, a training end condition may be further set, and when the training end condition is satisfied, the training of the deep learning model may be ended. Specifically, the training end condition may include: the loss value of the deep learning model is smaller than a preset loss threshold value, and the loss value is not reduced any more; the current training times reach a preset time threshold (e.g., 500 times, 1000 times, etc.), which is not limited herein.

Unlike the foregoing embodiment, by using the actual position information of the target points, a coarse target heat map and a fine target heat map are generated, and the heat value of a third area containing the target point in the coarse target heat map is in a third heat value range, the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, the third area is larger than the fourth area, therefore, the target point detection is carried out on the sample image by utilizing the deep learning model to obtain a coarse positioning heat map and a fine positioning heat map, and the network parameters of the deep learning model are adjusted based on the difference between the coarse target heat map and the coarse positioning heat map and the difference between the fine target heat map and the fine positioning heat map, so that the deep learning model can generate the accurate coarse positioning heat map and the fine positioning heat map, and the accuracy and the stability of point positioning can be improved.

Referring to fig. 5, fig. 5 is a flowchart illustrating an embodiment of step S13 in fig. 1. Specifically, the method may include the steps of:

step S131: and determining the position of the first target point based on the heat value of the rough positioning heat map.

Specifically, a point in the rough positioning heat map where the heat value is the largest may be taken as the first target point position. In an implementation scenario, a preset point in the first region (e.g., the center of gravity of the first region) in the rough localization heat map may also be used as the first target point position, which is not limited herein.

In one implementation scenario, in determining the first target point location, a first confidence level of the first target point location may also be determined. Specifically, at least one reference heat value may be acquired, for example, 1 reference heat value, 2 reference heat values, 3 reference heat values, and the like, and for each reference heat value, the size of a reference region having a heat value greater than the reference heat value is acquired from the rough positioning heat map, so that the first confidence of the first target point position is obtained based on the size of each reference region and the heat value of the first target point position. In a specific implementation scenario, the at least one reference heat value may be obtained by calculating a product between at least one multiplying factor (e.g., 0.2, 0.4, 0.6, 0.8, etc.) and the heat value of the first target point location, respectively. In a specific implementation scenario, the size of the reference region includes a perimeter and an area of the reference region, so that a first ratio between the area of each reference region and a square of the perimeter may be obtained, and a first confidence of the first target point position is obtained by using a sum of the first ratios of at least one reference region, a heat value of the first target point position, and a preset heat peak. Specifically, it can be expressed as:

in the above formula (2), confidence represents the first confidence, M represents the heat value of the first target point, M represents the preset heat peak, K represents the number of reference regions, and s_iDenotes the area of the ith reference region, c_i ²Represents the square of the perimeter of the ith reference region.

In another implementation scenario, in order to determine the first confidence of the position of the first target point, a coarse target heat map may be obtained based on the position of the first target point in the coarse positioning heat map, and the first confidence of the position of the first target point may be obtained based on similarity of distribution of heat values between the coarse positioning heat map and the coarse target heat map. In a specific implementation scenario, the heat value of each pixel point in the coarse target heat map may be obtained by using a two-dimensional gaussian function based on the position of the first target point of the coarse positioning heat map, where an index of the two-dimensional gaussian function includes a range parameter, and an absolute value of the index and the range parameter are in a negative correlation relationship. The process of obtaining the target heat map may specifically refer to the relevant steps in the foregoing embodiments, and details are not repeated here. In another specific implementation scenario, a correlation coefficient between the coarse positioning heat map and the coarse target heat map may be obtained as the first confidence of the first target point position based on the heat value distribution between the coarse positioning heat map and the coarse target heat map.

Step S132: based on the heat value of the fine positioning heat map, the confidence of the position of the second target point and the position of the second target point is determined.

Specifically, a point with the largest heat value in the fine positioning heat map may be taken as the second target point position; alternatively, a preset point in the second region (for example, the center of gravity of the second region) in the fine positioning heat map may be used as the second target point position, which is not limited herein.

In one implementation scenario, to determine the second confidence of the second target point location, at least one reference thermal value may be obtained, and for each reference thermal value, the size of a reference region with a thermal value greater than the reference thermal value is obtained from the fine localization thermal map, so as to obtain the second confidence of the second target point location based on the size of each reference region and the second target point location. In a specific implementation scenario, at least one reference thermal value may be obtained by multiplying at least one multiplying factor by a thermal value of the second target point, which may specifically refer to the foregoing related steps, and is not described herein again. In another specific implementation scenario, a first ratio between the area of each reference region and the square of the perimeter may be obtained, and a second confidence of the position of the second target point is obtained by using the sum of the first ratios of at least one reference region, the heat value of the position of the second target point, and a preset heat peak value.

In another implementation scenario, in order to determine the second confidence of the position of the second target point, a fine target heat map may be further obtained based on the position of the second target point in the fine positioning heat map, and the confidence of the position of the second target point is obtained based on the similarity of heat value distribution between the fine positioning heat map and the fine target heat map. In a specific implementation scenario, the heat value of each pixel point in the fine target heat map may be obtained by using a two-dimensional gaussian function based on the position of the second target point of the fine positioning heat map, where an index of the two-dimensional gaussian function includes a range parameter, and an absolute value of the index and the range parameter are in a negative correlation relationship, and the range parameter in the two-dimensional gaussian function corresponding to the coarse positioning heat map is greater than the range parameter in the two-dimensional gaussian function corresponding to the fine positioning heat map. In another specific implementation scenario, a correlation coefficient between the fine positioning heat map and the fine target heat map may be obtained based on the similarity of heat value distribution between the fine positioning heat map and the fine target heat map, and is used as a second confidence of the second target point position.

In another implementation scenario, in order to position the second target point near the first target point when the first target point is positioned accurately, the confidence level of the first target point may be determined based on the heat level value of the coarse positioning heat level map, and when the confidence level of the first target point meets the coarse positioning confidence condition, the preset heat level value of the fine positioning heat level map that is located outside the preset distance range of the first target point is adjusted, and the preset heat level value is outside the second heat level range, for example, the preset heat level value is set to 0, which is not limited herein. After the fine positioning heat map is adjusted as described above, the confidence levels of the second target point position and the second target point position may be determined based on the heat values of the fine positioning heat map. The determination method of the confidence of the position of the first target point, the position of the second target point, and the confidence of the position of the second target point may specifically refer to the foregoing steps, which are not described herein again. In one particular implementation scenario, the coarse positioning confidence condition may include the confidence level of the first target point location being greater than a first preset threshold (e.g., 0.5, etc.). Therefore, when the coarse positioning confidence condition is met, the position of the first target point determined based on the coarse positioning heat map can be considered to be basically accurate, so that the positions of the first target point and the fine positioning heat map can be combined for positioning to obtain the position of the second target point and the position confidence of the second target point, so as to perform subsequent judgment, and the position of the second target point is located near the position of the first target point. In another specific implementation scenario, when the requirement on the point positioning accuracy is not high, the position of the first target point and the first confidence may also be directly output, which is not limited herein.

In an implementation scenario, the steps S131 and S132 may be executed in a sequential order, for example, the step S131 is executed first, and then the step S132 is executed, or the step S132 is executed first, and then the step S131 is executed. In another implementation scenario, the step S131 and the step S132 may also be executed simultaneously, which is not limited herein.

Step S133: and judging whether the second confidence coefficient meets the fine positioning confidence condition, if so, executing the step S134, and otherwise, executing the step S135.

In particular, the fine location confidence condition may include at least one of: the second confidence is greater than a second preset threshold (e.g., 0.5), and the second confidence is greater than the first confidence. When the second confidence degree meets the fine positioning confidence condition, the reliability of the position of the second target point may be considered to be higher, or the reliability of the position of the second target point is higher than that of the position of the first target point, so in order to improve the accuracy of point positioning, step S134 may be executed, that is, the position of the second target point is used as the position information of the target point; on the contrary, in order to ensure the stability of the point positioning, step S135 may be performed, i.e., the first target point position is adopted as the position information of the target point.

Step S134: and taking the position of the second target point as the position information of the target point.

When the second confidence degree meets the fine positioning confidence condition, the confidence degree of the second target point position may be considered to be higher, or the confidence degree of the second target point position may be higher than that of the first target point position, so in order to improve the accuracy of point positioning, the second target point position may be used as the position information of the target point. In one implementation scenario, when the second target point position is taken as the position information of the target point, the confidence of the second target point position may be taken as the confidence of the corresponding position information, and the position information of the target point and the confidence of the corresponding position information may be output.

Step S135: and taking the position of the first target point as the position information of the target point.

When the second confidence does not satisfy the fine positioning confidence condition, it may be considered that the reliability of the second target point position is low, or the reliability of the first target point position is higher than the reliability of the second target point position, so in order to ensure the stability of point positioning, the first target point position may be used as the position information of the target point. In one implementation scenario, when the first target point position is used as the position information of the target point, the confidence of the first target point position may be used as the confidence of the corresponding position information, and the position information of the target point position and the confidence of the corresponding position information may be output.

In addition, when the first confidence of the position of the first target point in the rough positioning heat map is lower than the first preset confidence threshold and the second confidence of the position of the second target point is lower than the second preset confidence threshold, it can be considered that the image to be positioned has objective reasons such as poor image quality, no target point and the like, and accurate positioning cannot be realized, so that the point positioning process can be finished. The first preset confidence threshold and the second preset confidence threshold may be set according to an actual situation, which is not limited herein. In an implementation scene, the objective reasons can be output to remind, so that error positioning of overlarge deviation can be avoided when accurate positioning cannot be performed, and meanwhile, follow-up completion can be facilitated.

Different from the foregoing embodiment, the confidence degrees of the second target point position and the second target point position are determined and obtained by determining the position of the first target point through the rough positioning heat map and by determining the heat value of the fine positioning heat map, and when the confidence degree of the second target point position satisfies the fine positioning confidence condition, the second target point position is taken as the position information of the target point, and when the confidence degree of the second target point position does not satisfy the fine positioning confidence condition, the first target point position is taken as the position information of the target point, so that the selection of the position information of the target point with better confidence degree can be facilitated, and the accuracy and the stability of point positioning can be improved.

Referring to fig. 6, fig. 6 is a schematic diagram of a frame of an embodiment of the point locating device 60 of the present application. The positioning device 60 includes: the system comprises an image acquisition module 61, a target detection module 62 and a position analysis module 63, wherein the image acquisition module 61 is used for acquiring an image to be positioned containing a target point; the target detection module 62 is configured to perform target point detection on the image to be positioned, so as to obtain a coarse positioning heat map and a fine positioning heat map, where a heat value of a first area including the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is in a second heat value range, wherein the first area is larger than the second area; the position analysis module 63 is configured to combine and analyze the coarse positioning heat map and the fine positioning heat map to obtain position information of the target point.

In some embodiments, the position analysis module 63 is specifically configured to obtain confidence degrees of a first target point position in the coarse positioning heat map and a second target point position in the fine positioning heat map, and process the first target point position and the second target point position based on the confidence degree of the second target point position to obtain position information of the target point.

Different from the foregoing embodiment, the confidence degrees of the first target point position in the coarse positioning heat map and the second target point position in the fine positioning heat map are obtained, so that the first target point position and the second target point position are processed based on the confidence degree of the second target point position to obtain the position information of the target point, and further, the position information of the target point can be determined based on the confidence degree conditions of the target points in the coarse positioning heat map and the fine positioning heat map, so that the accuracy of point positioning can be further improved.

In some embodiments, the location analysis module 63 includes a first analysis submodule for determining a first target point location based on the heat value of the coarse localization heat map, the location analysis module 63 includes a second analysis submodule for determining a second target point location and a confidence of the second target point location based on the heat value of the fine localization heat map, the location analysis module 63 includes a location selection submodule for selecting either the second target point location or the first target point location as the location information of the target point based on the confidence of the second target point location.

Different from the foregoing embodiment, the position of the first target point is determined by the coarse positioning heat map, the confidence levels of the position of the second target point and the position of the second target point are determined by the fine positioning heat map, the position of the second target point or the position of the first target point is selected based on the confidence level of the position of the second target point, and the position information of the target point is selected from the positions of the first target point determined by the first target point and the coarse positioning heat map as the position information of the target point.

The position selection submodule comprises a condition judgment unit used for judging whether the confidence coefficient of the position of the second target point meets the fine positioning confidence condition, the position selection submodule comprises a position determination unit used for taking the position of the second target point as the position information of the target point when the confidence coefficient of the position of the second target point meets the fine positioning confidence condition, and the position determination unit is also used for taking the position of the first target point as the position information of the target point when the confidence coefficient of the position of the second target point does not meet the fine positioning confidence condition.

Different from the foregoing embodiment, when the confidence of the second target point position satisfies the fine positioning confidence condition, the second target point position is used as the position information of the target point, and when the confidence of the second target point position does not satisfy the fine positioning confidence condition, the first target point position is used as the position information of the target point, so that it is beneficial to selecting the target point position with better confidence as the position information of the target point, and thus it is beneficial to improving the accuracy and stability of point positioning.

In some embodiments, the first analysis submodule is further configured to determine a confidence level of the first target point position based on the heat value of the coarse positioning heat map, and the second analysis submodule includes an adjusting unit configured to adjust, when the confidence level of the first target point position satisfies the coarse positioning confidence condition, a heat value in the fine positioning heat map, which is located outside a preset distance range of the first target point position, to a preset heat value; wherein the preset heat value is out of the second heat value range.

Different from the foregoing embodiment, when the confidence of the first target point position satisfies the coarse positioning confidence condition, the heat value outside the preset distance range of the first target point position in the fine positioning heat map is directly adjusted to the preset heat value, so that after the fine positioning heat map is adjusted, based on the heat value of the fine positioning heat map, the confidence of the second target point position and the second target point position is determined, so that the second target point position can be located near the first target point position, and the accuracy of point positioning is further improved.

In some embodiments, the coarse localization confidence condition comprises a confidence that the first target point location is greater than a first preset threshold, and the fine localization confidence condition comprises at least one of: the confidence coefficient of the position of the second target point is greater than a second preset threshold value, and the confidence coefficient of the position of the second target point is greater than that of the position of the first target point.

Different from the foregoing embodiment, the coarse positioning confidence condition is set to include that the confidence of the position of the first target point is greater than a first preset threshold, and the fine positioning confidence condition is set to include that the confidence of the position of the second target point is greater than at least one of the second preset threshold, and the confidence of the position of the second target point is greater than the confidence of the position of the first target point, which can be beneficial to screening the position information of the target point with better confidence, and thus can be beneficial to improving the accuracy and stability of point positioning.

In some embodiments, the first analysis submodule, the second analysis submodule, or the determination unit is specifically configured to: and taking the point with the maximum heat value in the positioning heat map as the target point position, or taking the area preset point in the positioning heat map as the target point position.

Different from the foregoing embodiment, by using the point with the largest heat value in the localization heat map as the target point position, or using the area preset point in the localization heat map as the target point position, the difficulty in determining the target point position can be reduced, and the speed of point localization can be increased.

In some embodiments, the first analysis sub-module, the second analysis sub-module, or the determining unit specifically includes: a reference acquisition subunit for acquiring at least one reference calorific value; the area determining subunit is used for acquiring the size of a reference area with the heat value larger than the reference heat value from the positioning heat map according to each reference heat value; the confidence calculation subunit is used for obtaining the confidence of the position of the target point based on the size of each reference area and the heat value of the position of the target point; or, specifically, includes: the target heat map acquisition subunit is used for acquiring a target heat map based on the target point position of the positioning heat map; and the confidence determining subunit is used for obtaining the confidence of the position of the target point based on the distribution similarity of the heat values between the positioning heat map and the target heat map.

Different from the foregoing embodiment, by obtaining at least one reference heat value, and for each reference heat value, obtaining the size of the reference region with the heat value greater than the reference heat value from the positioning heat map, so as to obtain the confidence level of the target point position based on the size of each reference region and the heat value of the target point position, or obtain the confidence level of the target point position based on the similarity of distribution of the heat values between the positioning heat map and the target heat map, the accuracy of the confidence level can be improved, and at the same time, the subsequent relocation and completion of the region with the lower confidence level can be facilitated.

In some embodiments, the reference obtaining subunit is specifically configured to obtain at least one magnification, and take a product between each of the at least one magnification and a heat value of the target point location as at least one reference heat value, where the reference area includes the target point location; the size of the reference region comprises the perimeter and the area of the reference region, the confidence calculation subunit is specifically configured to obtain a first ratio between the area of each reference region and the square of the perimeter, and obtain the confidence of the target point position by using the sum of the first ratios of at least one reference region, the heat value of the target point position, and the preset heat peak value.

Different from the embodiment, by obtaining at least one multiplying power and taking the product of the at least one multiplying power and the heat value of the target point position as at least one reference heat value, the reference heat value can be conveniently and quickly determined, the speed of point positioning is favorably improved, and the confidence of the target point position is obtained by obtaining the first ratio between the area of each reference region and the square of the perimeter and by using the sum of the first ratios of the at least one reference region, the heat value of the target point position and the preset heat peak value, so that the confidence of the target point position can be accurately determined.

In some embodiments, the target heat map obtaining subunit is specifically configured to obtain, based on a position of a target point of the positioning heat map, a heat value of each pixel point in the target heat map by using a two-dimensional gaussian function, where an index of the two-dimensional gaussian function includes a range parameter, an absolute value of the index and the range parameter are in a negative correlation relationship, and the range parameter in the two-dimensional gaussian function corresponding to the coarse positioning heat map is greater than the range parameter in the two-dimensional gaussian function corresponding to the fine positioning heat map; the confidence determination subunit is specifically configured to obtain a correlation coefficient between the positioning heat map and the target heat map as the confidence of the target point position based on the heat value distribution between the positioning heat map and the target heat map.

Different from the embodiment, the target point position of the heat map is positioned, the heat value of each pixel point in the target heat map is obtained by utilizing the two-dimensional Gaussian function, the index of the two-dimensional Gaussian function comprises the range parameter, the absolute value of the index and the range parameter are in a negative correlation relationship, the range parameter in the two-dimensional Gaussian function corresponding to the coarse positioning heat map is larger than the range parameter in the two-dimensional Gaussian function corresponding to the fine positioning heat map, so that the target heat map can be conveniently and accurately obtained, and the correlation coefficient between the positioning heat map and the target heat map is obtained based on the heat value distribution between the positioning heat map and the target heat map and is used as the confidence coefficient of the target point position, so that the confidence coefficient of the target point position can be conveniently and accurately obtained.

In some embodiments, the point location device 60 further comprises a preprocessing module for preprocessing the image to be located; wherein the pre-treatment comprises at least one of: and normalizing the image to be positioned, and enhancing the image contrast of the image to be positioned.

Different from the embodiment, the image to be positioned is preprocessed before the image to be positioned is subjected to target point detection, and the preprocessing comprises normalizing the image to be positioned and/or enhancing the image contrast of the image to be positioned, so that the accuracy of subsequent target point detection can be improved.

In some embodiments, the point locating device 60 further comprises an output module for outputting position information of the target point and a confidence of the corresponding position information.

Different from the embodiment, the position information of the target point obtained by positioning can be favorably evaluated by the user by outputting the position information of the target point and the confidence degree of the corresponding position information, so that the user perception is improved.

In some embodiments, the pre-processing module includes a normalization sub-module configured to set a pixel value in the image to be positioned that is greater than the first pixel value to a first pixel value, and set a pixel value in the image to be positioned that is less than the second pixel value to a second pixel value; among the sequentially arranged pixel values of the image to be positioned, the pixel value positioned at the first numerical value arrangement position is a first pixel value, and the pixel value positioned at the second numerical value arrangement position is a second pixel value.

Different from the foregoing embodiment, the pixel value larger than the first pixel value in the image to be positioned is set as the first pixel value, the heat value smaller than the second pixel value in the image to be positioned is set as the second pixel value, and in the pixel values of the ordering sequence of the image to be positioned, the pixel value positioned in the first numerical value arrangement is the first pixel value, and the pixel value positioned in the second numerical value arrangement is the second pixel value, so that interference such as extra-bright or extra-dark in the image to be positioned can be favorably eliminated, the accuracy of detection of a subsequent target point can be favorably improved, and the accuracy of point positioning can be improved.

In some embodiments, the target detection module 62 is specifically configured to perform target point detection on the image to be positioned by using a deep learning model, so as to obtain a coarse positioning heat map and a fine positioning heat map.

Different from the embodiment, the target point detection is performed on the image to be positioned by using the deep learning model, so that the parallel acceleration of hardware can be fully utilized, and the complexity of the target point detection is reduced.

In some embodiments, the deep learning model is a full convolution neural network, and the point location device 60 further includes a model training module, specifically including: the system comprises a sample acquisition submodule and a heat map generation submodule, wherein the sample acquisition submodule is used for acquiring a sample image, the sample image is marked with real position information of a target point, and the heat map generation submodule is used for generating a coarse target heat map and a fine target heat map by using the real position information of the target point; the heat value of a third area containing the target point in the coarse target heat map is in a third heat value range; the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, wherein the third area is larger than the fourth area; the device comprises a target detection submodule and a parameter adjustment submodule, wherein the target detection submodule is used for detecting a target point of a sample image by using a deep learning model to obtain a coarse positioning heat degree map and a fine positioning heat degree map, and the parameter adjustment submodule is used for adjusting network parameters of the deep learning model based on the difference between the coarse positioning heat degree map and the difference between the fine positioning heat degree map and the fine positioning heat degree map.

In some embodiments, the image to be located is an X-ray image; in the first area and the second area, the closer to the target point, the higher the heat value; the heat value outside the first region in the coarse positioning heat map is lower than the lower limit value of the first heat value range, and the heat value outside the second region in the fine positioning heat map is lower than the lower limit value of the second heat value range.

Different from the foregoing embodiment, in the first area and the second area, the closer the heat value to the target point is, the higher the heat value is, the lower limit value of the first heat value range is lower for the heat value outside the first area in the coarse positioning heat map, and the lower limit value of the second heat value range is lower for the heat value outside the second area in the fine positioning heat map, so that the target points in the coarse positioning heat map and the fine positioning heat map can be determined conveniently.

Referring to fig. 7, fig. 7 is a schematic diagram of a frame of an embodiment of an electronic device 70 according to the present application. The electronic device 70 comprises a memory 71 and a processor 72 coupled to each other, and the processor 72 is configured to execute program instructions stored in the memory 71 to implement the steps in any of the above-described embodiments of the point location method. In one particular implementation scenario, the electronic device 70 may include, but is not limited to: a microcomputer, a server, and the electronic device 70 may also include a mobile device such as a notebook computer, a tablet computer, and the like, which is not limited herein.

In particular, the processor 72 is configured to control itself and the memory 71 to implement the steps of any of the above-described embodiments of the point location method. The processor 72 may also be referred to as a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip having signal processing capabilities. The Processor 72 may also be a general purpose Processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other Programmable logic device, discrete Gate or transistor logic, discrete hardware components. A general purpose processor may be a microprocessor or the processor may be any conventional processor or the like. Additionally, the processor 72 may be collectively implemented by an integrated circuit chip.

By the scheme, the positioning stability of the coarse positioning heat map and the accuracy of the fine positioning heat map can be achieved simultaneously, and therefore the accuracy and the stability of point positioning can be improved.

Referring to fig. 8, fig. 8 is a block diagram illustrating an embodiment of a computer readable storage medium 80 according to the present application. The computer readable storage medium 80 stores program instructions 801 that can be executed by the processor, and the program instructions 801 are used for implementing the steps in any one of the above-described embodiments of the point location method.

The positioning stability of the coarse positioning heat map and the accuracy of the fine positioning heat map can be achieved simultaneously, and therefore the accuracy and the stability of point positioning can be improved.

In the several embodiments provided in the present application, it should be understood that the disclosed method and apparatus may be implemented in other ways. For example, the above-described apparatus embodiments are merely illustrative, and for example, a division of a module or a unit is merely one type of logical division, and an actual implementation may have another division, for example, a unit or a component may be combined or integrated with another system, or some features may be omitted, or not implemented. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection of devices or units through some interfaces, and may be in an electrical, mechanical or other form.

Units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on network elements. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

In addition, functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be substantially implemented or contributed to by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, a network device, or the like) or a processor (processor) to execute all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.

Claims

1. A method of point location, the method comprising:

acquiring an image to be positioned containing a target point;

detecting a target point of an image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map, wherein the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is within a second heat value range, wherein the first area is larger than the second area;

and analyzing the rough positioning heat degree diagram and the fine positioning heat degree diagram in a combined manner to obtain the position information of the target point.

2. The method of claim 1, wherein the analyzing the coarse localization heat map and the fine localization heat map in combination to obtain the position information of the target point comprises:

and acquiring the position confidence of a first target point in the coarse positioning heat map and the position confidence of a second target point in the fine positioning heat map and the position confidence of the second target point, and processing the position of the first target point and the position of the second target point based on the position confidence of the second target point to obtain the position information of the target point.

3. The method according to claim 2, wherein the obtaining confidence degrees of the first target point position in the coarse positioning heat map and the second target point position in the fine positioning heat map, and based on the confidence degree of the second target point position, processing the first target point position and the second target point position to obtain the position information of the target point comprises:

determining a first target point position based on the heat value of the rough positioning heat map; and determining a second target point position and a confidence level of the second target point position based on the heat value of the fine positioning heat map;

and selecting the position of the second target point or the position of the first target point as the position information of the target point based on the confidence of the position of the second target point.

4. The method according to claim 3, wherein the selecting the second target point position or the first target point position as the position information of the target point based on the confidence of the second target point position comprises:

if the confidence of the second target point position meets a fine positioning confidence condition, taking the second target point position as the position information of the target point;

and if the confidence of the second target point position does not meet the fine positioning confidence condition, taking the first target point position as the position information of the target point.

5. The method of claim 3, wherein prior to the determining confidence levels for a second target point location and the second target point location based on the heat values of the fine positioning heat map, the method comprises:

determining a confidence level of the position of the first target point based on the heat value of the rough positioning heat map;

if the confidence of the first target point position meets the coarse positioning confidence condition, adjusting the heat value of the fine positioning heat map, which is positioned outside a preset distance range of the first target point position, to be a preset heat value; wherein the preset heat value is outside the second heat value range.

6. The method of claim 5, wherein the coarse positioning confidence condition comprises a confidence that the first target point location is greater than a first preset threshold, and wherein the fine positioning confidence condition comprises at least one of: the confidence of the position of the second target point is greater than a second preset threshold, and the confidence of the position of the second target point is greater than the confidence of the position of the first target point.

7. The method of claim 5 or 6, wherein determining a first target point location based on the heat value of the coarse positioning heat map or determining a second target point location based on the heat value of the fine positioning heat map comprises:

taking the point with the maximum heat value in the positioning heat map as a target point position, or taking an area preset point in the positioning heat map as the target point position;

and/or, the determining the confidence of the position of the first target point based on the heat value of the coarse positioning heat map, or the determining the confidence of the position of the second target point based on the heat value of the fine positioning heat map comprises:

acquiring at least one reference heat value; for each reference heat value, acquiring the size of a reference area with the heat value larger than the reference heat value from the positioning heat map; obtaining the confidence of the position of the target point based on the size of each reference area and the heat value of the position of the target point;

or, obtaining a target heat map based on the target point position of the positioning heat map; and obtaining the confidence of the target point position based on the similarity of the heat value distribution between the positioning heat map and the target heat map.

8. The method of claim 7, wherein said obtaining at least one reference heat value comprises:

acquiring at least one multiplying power, and taking the product of the at least one multiplying power and the heat value of the target point position as at least one reference heat value;

the reference area includes the target point location; the size of the reference region comprises a perimeter and an area of the reference region; the obtaining a confidence level of the target point location based on the size of each of the reference regions and the heat value of the target point location comprises:

and acquiring a first proportion between the area of each reference region and the square of the perimeter, and acquiring the confidence of the target point position by using the sum of the first proportions of the at least one reference region, the heat value of the target point position and a preset heat peak value.

9. The method of claim 7, wherein obtaining the target heat map based on the target point locations of the positioning heat map comprises:

based on the position of a target point of a positioning heat map, acquiring the heat value of each pixel point in the target heat map by using a two-dimensional Gaussian function, wherein the index of the two-dimensional Gaussian function comprises a range parameter, the absolute value of the index and the range parameter are in a negative correlation relationship, and the range parameter in the two-dimensional Gaussian function corresponding to the coarse positioning heat map is larger than the range parameter in the two-dimensional Gaussian function corresponding to the fine positioning heat map;

the obtaining the confidence of the position of the target point based on the similarity of the distribution of the heat value between the positioning heat map and the target heat map comprises:

and obtaining a correlation coefficient between the positioning heat map and the target heat map as the confidence of the position of the target point based on the heat value distribution between the positioning heat map and the target heat map.

10. The method according to any one of claims 1 to 9, wherein before the detecting the target point of the image to be positioned to obtain the coarse positioning heat map and the fine positioning heat map, the method further comprises:

preprocessing the image to be positioned; wherein the pre-treatment comprises at least one of: normalizing the image to be positioned, and enhancing the image contrast of the image to be positioned;

and/or after the combination analysis of the coarse positioning heat map and the fine positioning heat map to obtain the position information of the target point, the method further comprises the following steps:

and outputting the position information of the target point and the confidence corresponding to the position information.

11. The method of claim 10, wherein the normalizing the image to be located comprises:

setting a pixel value larger than a first pixel value in the image to be positioned as the first pixel value, and setting a pixel value smaller than a second pixel value in the image to be positioned as the second pixel value;

among the sequentially arranged pixel values of the image to be positioned, the pixel value positioned at the first numerical value arrangement position is a first pixel value, and the pixel value positioned at the second numerical value arrangement position is a second pixel value.

12. The method according to any one of claims 1 to 11, wherein the detecting the target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map comprises:

and detecting a target point of the image to be positioned by using a deep learning model to obtain a coarse positioning heat map and a fine positioning heat map.

13. The method of claim 11, wherein the deep learning model is a full convolutional neural network, and/or wherein the deep learning model is trained by at least:

acquiring a sample image, wherein the sample image is marked with real position information of a target point;

generating a coarse target heat map and a fine target heat map by using the real position information of the target point; the heat value of a third area containing the target point in the coarse target heat map is in a third heat value range; the heat value of a fourth area containing the target point in the fine target heat map is in a fourth heat value range, wherein the third area is larger than the fourth area;

detecting a target point of the sample image by using the deep learning model to obtain a coarse positioning heat map and a fine positioning heat map;

and adjusting the network parameters of the deep learning model based on the difference between the coarse target heat map and the coarse positioning heat map and the difference between the fine target heat map and the fine positioning heat map.

14. The method according to any one of claims 1 to 13, characterized in that the image to be located is an X-ray image;

in the first area and the second area, the closer to the target point, the higher the heat value is; the heat value of the coarse positioning heat map outside the first region is lower than the lower limit value of the first heat value range, and the heat value of the fine positioning heat map outside the second region is lower than the lower limit value of the second heat value range.

15. A point locating device, comprising:

the image acquisition module is used for acquiring an image to be positioned containing a target point;

the target detection module is used for detecting a target point of the image to be positioned to obtain a coarse positioning heat map and a fine positioning heat map, wherein the heat value of a first area containing the target point in the coarse positioning heat map is in a first heat value range; the heat value of a second area containing the target point in the fine positioning heat map is within a second heat value range, wherein the first area is larger than the second area;

and the position analysis module is used for analyzing the coarse positioning heat map and the fine positioning heat map in a combined manner to obtain the position information of the target point.

16. An electronic device comprising a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the point location method of any one of claims 1 to 14.

17. A computer readable storage medium having stored thereon program instructions which, when executed by a processor, implement the point localization method according to any one of claims 1 to 14.