WO2022007342A1 - 图像处理方法及装置、电子设备、存储介质和程序产品 - Google Patents

图像处理方法及装置、电子设备、存储介质和程序产品 Download PDF

Info

Publication number
WO2022007342A1
WO2022007342A1 PCT/CN2020/138438 CN2020138438W WO2022007342A1 WO 2022007342 A1 WO2022007342 A1 WO 2022007342A1 CN 2020138438 W CN2020138438 W CN 2020138438W WO 2022007342 A1 WO2022007342 A1 WO 2022007342A1
Authority
WO
WIPO (PCT)
Prior art keywords
target
key point
heat map
image
target key
Prior art date
Application number
PCT/CN2020/138438
Other languages
English (en)
French (fr)
Inventor
顾宇俊
袁璟
赵亮
黄宁
Original Assignee
上海商汤智能科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 上海商汤智能科技有限公司 filed Critical 上海商汤智能科技有限公司
Priority to KR1020217042439A priority Critical patent/KR20220013404A/ko
Priority to JP2021576877A priority patent/JP2022542780A/ja
Publication of WO2022007342A1 publication Critical patent/WO2022007342A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0012—Biomedical image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30004—Biomedical image processing
    • G06T2207/30008—Bone

Definitions

  • the present disclosure relates to the technical field of image processing, and in particular, to an image processing method and apparatus, an electronic device, a storage medium, and a program product.
  • the force line of the knee joint is shifted to the central or even slightly lateral part, thereby reducing the pressure on the medial compartment and avoiding or delaying the high tibial osteotomy for joint replacement.
  • An important part of the high tibial osteotomy is to determine the position of the tibial osteotomy. The accuracy of the tibial osteotomy position will greatly affect the effect of the high tibial osteotomy.
  • the embodiment of the present disclosure proposes an image processing technical solution.
  • an image processing method including:
  • the acquiring the target key points of the target image includes: performing key point detection on the target image to obtain at least two target key points containing confidence.
  • At least two target keypoints containing confidence are obtained. It is possible to determine whether the target key points are accurate while acquiring the target key points. In the case that the target key points cannot be accurately predicted due to some reasons (such as poor image quality, non-existence of target key points, etc.), it can be determined based on the confidence Some low-accuracy target key points are excluded or additionally processed to complete, so as to improve the accuracy of the target key points, and then improve the accuracy of the subsequent processing parameters.
  • the obtaining the target key points of the target image includes: performing key point detection on the target image to obtain heat maps corresponding to at least two target key points respectively; obtaining at least two of the target key points. The coordinates and confidence levels of the target key points corresponding to the heat map.
  • the coordinates and confidence of the target key points can be simultaneously determined by obtaining the heat map corresponding to the target key points.
  • the processing process is simple and intuitive, which improves the accuracy and efficiency of obtaining the target key points, thereby improving the overall process of image processing. accuracy and efficiency.
  • the acquiring the target key points of the target image includes: performing key point detection on the target image to obtain a first heat map and a second heat map corresponding to at least two target key points respectively , wherein the response range of the target key point corresponding to the first heat map is greater than the response range of the target key point corresponding to the second heat map; the first key point is determined from the first heat map , according to the first heat map, obtain the first confidence of the first key point; according to the first confidence, determine the second key point from the second heat map, and combine the second heat Figure, obtain the second confidence level of the second key point; according to the second confidence level, determine the first key point or the second key point as the target key point in the target image, and Obtain the confidence corresponding to the target key point.
  • the first heat map with relatively coarse positioning results and the second heat map with relatively fine positioning results can be effectively used to comprehensively determine the position and confidence of the target key points in the target image,
  • the accuracy and stability of the target key point positioning prediction in the target image are improved, and the accuracy and stability of the subsequent image processing results are improved.
  • the second key point is determined from the second heat map according to the first confidence level, and the second key point of the second key point is obtained in combination with the second heat map.
  • Two confidence levels including: when the first confidence level is greater than a first confidence level threshold, determining a response area according to the position of the first key point in the first heat map, and determining a response area from the second heat map A second key point is determined in the response area of the map, and a second confidence level of the second key point is obtained according to the response area of the second heat map; the first confidence level is not greater than all
  • a second key point is determined from the second heat map, and a second confidence of the second key point is obtained according to the second heat map.
  • the first confidence level is greater than the first confidence level threshold, that is, the position of the first key point determined in the first heat map is relatively accurate, due to the possibility that the target key point in the target image is within the response area
  • the second key point is determined directly in the response area of the second heat map, which can reduce the amount of calculated data on the one hand, and make the determined second key point have a high degree of confidence on the other hand;
  • the first confidence level is not greater than the first confidence level threshold, that is, the position of the first key point determined in the first heat map is relatively low, since the first heat map and the second heat map are independent of each other,
  • the target key point with higher confidence can still be obtained.
  • the accuracy of the final target key points is greatly improved, thereby improving the accuracy of image processing.
  • the first key point or the second key point is determined as the target key point in the target image according to the second confidence level, and a comparison between the first key point and the second key point is obtained.
  • the confidence level corresponding to the target key point includes: in the case that the second confidence level is greater than the second confidence level threshold, determining the second key point as the target key point in the target image, and determining the second key point as the target key point in the target image.
  • the second confidence level is used as the confidence level corresponding to the target key point; in the case that the second confidence level is not greater than the second confidence level threshold, the first key point is determined as the key point in the target image.
  • the first confidence level is taken as the confidence level corresponding to the target key point.
  • the above process is further based on the comparison between the second confidence level and the second confidence level threshold to select whether the first key point or the second key point is used as the target key point of the target image, and determine the confidence level of the target key point in the target image. .
  • the accuracy and stability of target key point location prediction in the target image are improved, and the accuracy and stability of subsequent image processing results are improved.
  • the performing key point detection on the target image includes: inputting the target image into a first neural network to perform key point detection.
  • the key point detection process can be realized through the neural network, thereby effectively improving the stability, efficiency and accuracy of key point detection, and then improving the stability, efficiency and stability of image processing. precision.
  • the neural network can flexibly adjust the structure and implementation method according to the actual situation of key point detection, the flexibility of key point detection can be improved, and then the flexibility of image processing method implementation can be improved.
  • the first neural network is trained by using a training image containing target key point position annotations, and the training includes: generating a position corresponding to the target key point position according to the target key point position annotation The corresponding target heat map; input the training image to the first neural network to obtain the heat map output by the first neural network; determine the first neural network according to the output heat map and the target heat map A loss function of the network; according to the loss function, at least one parameter of the first neural network is adjusted.
  • the target heat map to supervise the heat map output by the first neural network to determine the loss function of the first neural network, and adjust at least one parameter of the first neural network based on the loss function, so that the heat generated by the first neural network can be generated.
  • the graph is as close to the target heatmap as possible, so that the trained first neural network has high accuracy. Then, the accuracy of the target key points obtained based on the trained first neural network is improved, thereby improving the accuracy of image processing.
  • the acquiring the processing type of the target object includes: determining the processing type of the target object according to the target key point.
  • the processing type of the target object is determined by the target key points, so that the acquisition method of the processing type of the target object can be flexibly changed according to the difference of the target object and the application scene.
  • the target key points are at least two
  • the at least one processing parameter includes a first processing parameter and a second processing parameter
  • the target key point the segmentation result
  • determining at least one processing parameter of the target object includes: determining, according to the processing type, a first processing parameter and a second processing parameter to be acquired by the target object; point and the segmentation result, to obtain the first processing parameter; and according to at least three of the target key points, in combination with the first processing parameter, to obtain the second processing parameter.
  • the above-mentioned first processing parameter may be a processing parameter obtained first based on the target key point and the segmentation result
  • the second processing parameter may be a processing parameter that can be further obtained in combination with the target key point after the first processing parameter is determined, thereby realizing Depending on the type of processing, different processing parameters can be obtained, thereby improving the accuracy and flexibility of the image processing method.
  • the target image includes a preprocessed image
  • the preprocessing includes image normalization and/or image enhancement.
  • the preprocessed image is obtained as the target image through image standardization and/or image enhancement, which can facilitate the subsequent acquisition and segmentation of target key points for target images with uniform specifications and good image effects, and increase the acquisition and segmentation of target key points. It can also increase the accuracy and segmentation accuracy of acquiring target key points, thereby increasing the convenience and accuracy of image processing.
  • the target object includes a tibial object
  • the treatment type includes: medial closed type, lateral closed type, medial open type or lateral open type
  • the at least one processing parameter includes an incision point, One or more of hinge point, target force line, handle angle, and handle distance.
  • the determination standard and position of the infeed point in the first processing parameter to be acquired may change accordingly;
  • the second processing parameter that needs to be acquired may include a closed angle or an open angle, a closed distance or an open distance, and the like. Therefore, the processing type and corresponding processing parameters are flexibly selected according to the actual application scenario, so that the subsequent image processing results have better processing effects.
  • an image processing apparatus comprising: a target key point acquisition module configured to acquire target key points of a target image; a segmentation module configured to perform an image processing operation on a target object in the target image segmentation to obtain the segmentation result of the target object; a processing type acquisition module, configured to acquire the processing type of the target object; a processing parameter determination module, configured to obtain the processing type according to the target key point, the segmentation result and the processing type , and determine at least one processing parameter of the target object.
  • an electronic device comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to The image processing method described above is performed.
  • a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions implementing the above-mentioned image processing method when executed by a processor.
  • a computer program product comprising computer readable code, when the computer readable code is executed in an electronic device, a processor in the electronic device executes the above image Approach.
  • the segmentation result of the target object is obtained.
  • Type to determine at least one processing parameter of the target object.
  • FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure.
  • FIG. 2 shows a schematic diagram of the principle of high tibial osteotomy according to an embodiment of the present disclosure.
  • FIG. 3 shows a schematic diagram of a treatment type of a tibial object according to an embodiment of the present disclosure.
  • FIG. 4 shows a schematic diagram of an X-ray image of a high tibial osteotomy according to an embodiment of the present disclosure.
  • FIG. 5 shows a schematic diagram of a first heat map and a second heat map according to an embodiment of the present disclosure.
  • FIG. 6 shows a schematic diagram of acquiring processing parameters according to an embodiment of the present disclosure.
  • FIG. 7 shows a schematic diagram of a Fujisawa point according to an embodiment of the present disclosure.
  • FIG. 8 shows a schematic diagram of a second processing parameter according to an embodiment of the present disclosure.
  • FIG. 9 shows a schematic diagram of automatic positioning of target key points in an application example according to the present disclosure.
  • FIG. 10 shows a schematic diagram of automatic segmentation of the tibia according to an application example of the present disclosure.
  • FIG. 11 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
  • FIG. 12 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
  • FIG. 13 shows a block diagram of an electronic device according to an embodiment of the present disclosure.
  • exemplary is intended to serve as an example, embodiment, or illustration. Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.
  • FIG. 1 shows a flowchart of an image processing method according to an embodiment of the present disclosure.
  • the method may be applied to an image processing apparatus, and the image processing apparatus may be a terminal device, a server, or other processing devices.
  • the terminal device may be user equipment (User Equipment, UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, in-vehicle device, available wearable devices, etc.
  • the image processing method may be implemented by the processor calling computer-readable instructions stored in the memory.
  • the image processing method may include:
  • Step S11 acquiring target key points of the target image.
  • Step S12 segment the target object in the target image to obtain a segmentation result of the target object.
  • Step S13 acquiring the processing type of the target object.
  • Step S14 Determine at least one processing parameter of the target object according to the target key point, the segmentation result and the processing type.
  • the target image may be any form of image containing the target object, and the implementation manner may be flexibly determined according to the actual situation of the target object.
  • the target object may be any object with processing requirements, and its implementation form may be flexibly determined according to the actual application scenario of the image processing method proposed by the embodiments of the present disclosure.
  • the method proposed by the embodiments of the present disclosure may be applied to the surgical planning process, and the target object may be a surgical object, such as certain parts, organs or tissues of the human body, and the target image may be Contains medical images of surgical objects, such as X-rays, Computed Tomography (CT, Computed Tomography) images, or Magnetic Resonance Imaging (MRI, Magnetic Resonance Imaging).
  • CT Computed Tomography
  • MRI Magnetic Resonance Imaging
  • the image processing method proposed by the embodiments of the present disclosure can be applied to the surgical planning process of high tibial osteotomy
  • the target object can be the part where high osteotomy needs to be performed, that is, the tibial object (which can be the left tibia or the It is the right tibia, which is determined according to the actual needs of the tibial osteotomy), etc.
  • the target image can be a medical image containing these objects, such as a full-body X-ray, a lower limb X-ray, or an X-ray of the tibia.
  • Fig. 2 shows a schematic diagram of the principle of high tibial osteotomy according to an embodiment of the present disclosure.
  • medial stress concentration stage 21 for the treatment of knee osteoarthritis, there are 4 stages: medial stress concentration stage 21, proximal tibial osteotomy Stage 22, lateral force line transfer stage 23 and medial pressure relief stage 24, it can be seen that the high tibial osteotomy can transfer the force line of the knee joint to the central or even slightly lateral side before the medial cartilage is severely worn away. part, thereby reducing the pressure on the medial compartment and avoiding or delaying knee replacement. That is, high tibial osteotomy may need to determine the treatment parameters according to the stress state of the entire lower limb of the human body in the standing position.
  • the target object is the tibia object and the target image is the X-ray film of the lower extremity as an example to describe the image processing process, and the case where the target object is other objects or the target image is an image of other forms can refer to
  • the subsequent disclosed embodiments are flexibly expanded, and will not be listed one by one.
  • the number and realization form of the target key points of the target image can also be flexibly determined according to the realization form of the target object and the application scene of the image processing method. It should be noted that, due to the different application scenarios of the image processing method, the target key points in the target image can be included in the target object or located outside the target object, which can be determined according to the actual situation, which is not limited here. In a possible implementation, when the target object is a tibial object and the image processing method is applied to assist tibial osteotomy, as mentioned in the above disclosed embodiments, tibial osteotomy may need to take into account the overall lower limb of the human body.
  • the target key points can include both target key points belonging to the target object and target key points located outside the target object.
  • the target key points may include the center point of the femoral head, the center point of the ankle joint (which can be defined as the midpoint between the medial end point of the ankle joint gap and the lateral end point of the ankle joint gap), the center point of the knee joint (which can be Defined as the midpoint between the medial end point of the tibial plateau and the lateral end point of the tibial plateau), the medial end point of the tibial plateau and the lateral end point of the tibial plateau, in a possible implementation, the target key point can be based on the above-mentioned target key points.
  • step S11 and step S12 are independent of each other, and the implementation order of the two is not limited. That is, the target key points of the target image can be obtained first, and then the target object in the target image can be segmented; the target object can also be segmented first and then the target key points of the target image can be obtained; or the target key points of the target image can be obtained at the same time. Segment the target object, etc., and choose flexibly according to the actual situation.
  • the processing type of the target object may also be obtained through step S13.
  • the processing parameters of the target object there may be multiple processing methods in the corresponding application scenarios. With different processing methods, the processing parameters of the target object that need to be determined will naturally change. Processing type to specify the final processing parameters to be determined.
  • the processing type of the target object can be flexibly determined according to the target object and the application scenario of the target object.
  • FIG. 3 shows a schematic diagram of the processing type of the tibial object according to an embodiment of the present disclosure. It can be seen from FIG.
  • the processing types of the tibial object may include medial closed type 31 , lateral closed type 32 , medial open type 33 Or open 34 on the outside, etc.
  • At least one processing parameter of the target object can be determined in step S14 according to the target key point, the segmentation result and the processing type.
  • the number of processing parameters and the realization form of the target object can also be flexibly determined according to the realization form of the target object and the application scenario of the image processing method.
  • FIG. 4 shows a schematic diagram of an X-ray film of a high tibial osteotomy according to an embodiment of the present disclosure, wherein (a) part is a full-length X-ray film of the lower extremity including both legs, and (b) part is a preoperative high tibial osteotomy A full-length X-ray of the lower limb of a single leg, part (c) is a full-length X-ray of the lower limb of a single leg after high tibial osteotomy, in which the line segment between the center of the femoral head 41 and the center of the ankle joint 42 is the lower limb force line .
  • the high tibial osteotomy can be achieved by performing osteotomy on the tibia to correct the alignment of the lower limb. Due to the need for osteotomy of the tibia, it is necessary to consider the location of the osteotomy and the length of the osteotomy. Therefore, in a possible implementation manner, when the target object is a tibial object and the image processing method is applied to assist the tibial osteotomy, the processing parameters of the target object may include the feed point, the hinge point, the target force One or more of line, processing angle, processing distance, etc.
  • the target force line may be the line segment corresponding to the target position of the lower limb force line in the above-mentioned disclosed embodiments, for example, it may be the line segment connecting the target point after the correction of the ankle joint and the center point of the femoral head, etc.;
  • the processing angle may be The surgical angle in tibial osteotomy, the treatment angle can vary with the treatment type, for example, if the treatment type is medial closed or lateral closed, the treatment angle can be closed, and the treatment type is medial open Or in the case of lateral open type, the treatment angle may be an open angle; similarly, the treatment distance may be the osteotomy distance in the tibial osteotomy, which may also be divided into closed distance or open distance, etc., depending on the treatment type.
  • the determination process can be flexibly determined according to the actual situation, please refer to the following disclosed embodiments for details, which will not be discussed here. Expand.
  • the segmentation result of the target object is obtained.
  • Type to determine at least one processing parameter of the target object.
  • step S11 may include: performing key point detection on the target image to obtain at least two target key points containing confidence.
  • the number of target key points can be flexibly determined according to the actual situation.
  • the target key points may at least include the femoral head Center Point, Ankle Center Point, Knee Center Point, Medial Endpoint of Tibial Plateau, and Lateral Endpoint of Tibial Plateau.
  • the method of detecting the target key points on the target image can be flexibly determined according to the actual situation. For details, please refer to the following disclosed embodiments, which will not be expanded here. Since the detection of different target key points may produce different detection results, that is, the obtained target key points may not be completely accurate, therefore, the degree of accuracy of the obtained target key points can be reflected by the confidence. How to determine the confidence of the target key points in the implementation process can be determined flexibly. In a possible implementation manner, the confidence level of each target key point can be directly determined according to the relative positions of the detected target key points, so that the target image containing the confidence level can be obtained directly by detecting the target key points of the target image. target key points. In a possible implementation manner, the confidence level of the target key point may also be determined in other manners. For details, please refer to the following disclosed embodiments, which will not be expanded here.
  • At least two target keypoints containing confidence are obtained. It is possible to determine whether the target key points are accurate while acquiring the target key points. In the case that the target key points cannot be accurately predicted due to some reasons (such as poor image quality, non-existence of target key points, etc.), it can be determined based on the confidence Some low-accuracy target key points are excluded or additionally processed to complete, so as to improve the accuracy of the target key points, and then improve the accuracy of the subsequent processing parameters.
  • the manner of additionally processing and completing the target key points may be flexibly determined according to the actual situation, which is not limited in the embodiments of the present disclosure.
  • these target key points can be completed by the missing value completion method, that is, the target key points with higher confidence are used for inference to determine the feature vector of the target key points with lower confidence , and then determine the position of the target key point with low confidence.
  • step S11 may include: performing key point detection on the target image to obtain heat maps corresponding to at least two target key points respectively; obtaining the coordinates and confidence levels of the target key points corresponding to at least two heat maps Spend.
  • the heat map can be the response heat map of the target key point.
  • the size of the heatmap can be consistent with the target image.
  • the method of performing key point detection on the target image to obtain the heat map can be flexibly determined according to the actual situation. For details, please refer to the following disclosed embodiments, which will not be expanded here.
  • the number of target key points may be one or more, and accordingly, the number of heat maps may also be flexibly determined according to the actual situation.
  • a corresponding heat map can be obtained according to each target key point, that is, each target key point corresponds to a heat map; in a possible implementation, it can also be based on all target key points point to get the corresponding heat map, that is, a heat map contains all target key points. Subsequent disclosed embodiments are described with the implementation process of each target key point corresponding to a heat map, and the implementation process of one heat map including all target key points can be expanded correspondingly with reference to the subsequent disclosed embodiments, which will not be repeated here. .
  • each target key point corresponds to a heat map
  • how to determine the coordinates of the corresponding target key point according to the heat map can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the pixel with the highest response value in the heat map can be used as the target key point, then the coordinates of the pixel with the highest response value in the heat map can be used as the target key point in the target coordinates in the image.
  • the high response area in the heat map can also be determined first based on the heat map, and the determination method of the high response area can be flexibly set according to the actual situation, for example, each pixel in the heat map can be traversed , the pixels whose response value is higher than a certain set threshold are regarded as the pixels in the high-response area, so as to determine the high-response area in the heat map.
  • the center of gravity of the high response area can be further used as the target key point, then the coordinates of the center of gravity of the high response area in the heat map can be used as the coordinates of the target key point in the target image .
  • the confidence level of the target key point can also be obtained.
  • the manner of obtaining the confidence degree of the target key point may also be flexibly determined as described in the above disclosed embodiments. Since the embodiment of the present disclosure can determine the coordinates of the target key points by obtaining the heat map corresponding to the target key points, in a possible implementation manner, the heat map can be further used to determine the confidence of the target key points. In the implementation process, how to determine the confidence of the target key points according to the heat map, the implementation form of which can also be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the process of determining the confidence level of the target key point according to the heat map may include: selecting at least one area containing the target key point from the heat map according to the response value of the target key point; The response value of the point, combined with the area parameter of at least one area containing the target key point, determines the confidence level of the target key point.
  • the selection method of the area containing the target key points selected from the heat map can be flexibly set according to the actual situation.
  • the response value of the target key point can be denoted as m. Since the pixel point in the heat map is closer to the target key point, the higher the response value, so the heat map can be traversed, and the response value can be selected by traversing the heat map.
  • a i *m pixels the area formed by these pixels can naturally contain target key points.
  • a number of different regions containing target key points can be obtained .
  • the value of a i and the number of selected regions containing target key points can be flexibly determined according to the actual situation, not limited to the following disclosures Example.
  • four regions containing target key points may be selected, and these four regions containing target key points may correspond to four values of a i , respectively denoted as a 0 , a 1 , a 2 and a 3.
  • the area parameters of these areas can be determined, and the confidence of the target key points can be determined according to the determined area parameters and the response values of the target key points.
  • the area parameters of the area containing the target key points can be flexibly determined according to the actual situation.
  • the area parameter may be the perimeter c i of the area, and in a possible implementation manner, the area parameter may also be the area s i of the area. In a possible implementation manner, the area parameter may also be a parameter jointly determined according to the perimeter and area of the area, such as s i / ci 2 .
  • the method of determining the confidence of the target key points can also be flexibly changed.
  • the area parameters are jointly determined according to the perimeter and the area of The calculation method of the confidence level of the point can be the following formula (1):
  • Confidence is the confidence of the target key point
  • ⁇ is the pi ratio
  • m is the response value of the target key point
  • M is the target response value of the preset target key point
  • s i is the area containing the target key point
  • c i is the perimeter of the area containing the target keypoint.
  • the confidence level of the target key point may be determined by the area parameters of a plurality of areas including the target key point.
  • the confidence level of the target key point may also be determined in other manners.
  • the process of determining the confidence of the target key points according to the heat map may include: generating a target heat map corresponding to the position of the target key points according to the coordinates of the target key points; The heat map is normalized to obtain the first probability distribution; the target heat map is normalized to obtain the second probability distribution; the correlation coefficient between the first probability distribution and the second probability distribution is used as the confidence of the target key point.
  • the heat map corresponding to the target key points is the heat map obtained by performing key point detection on the target image
  • the target heat map is the heat map generated according to the coordinates of the target key points, that is, according to the heat map determined
  • the target key point coordinates can be reversed to generate a heat map as the target heat map.
  • the manner of generating the target heat map according to the coordinates of the target key points can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the target heat map can be generated by combining the coordinates of the target key points with the two-dimensional Gaussian function.
  • the realization form of the two-dimensional Gaussian function can be flexibly determined according to the actual situation.
  • the target heat map generated by the coordinates can be determined by the following formula (2):
  • f(x,y) is the two-dimensional Gaussian distribution function corresponding to the target heat map
  • x is the abscissa of the pixel in the target heat map
  • y is the ordinate of the pixel in the target heat map
  • M is the preset target.
  • the target response value of the key point x 0 is the abscissa of the target key point
  • y 0 is the ordinate of the target key point
  • e is a natural constant
  • ⁇ is the preset response range of the target key point.
  • the heat map and the target heat map can be normalized respectively to obtain the first probability distribution of the heat map and the second probability distribution of the target heat map, and the first probability and the second The correlation coefficient between the probability distributions is used as the confidence of the target key points.
  • heat maps corresponding to at least two target key points are obtained, and then the coordinates and confidence levels of the target key points corresponding to at least two heat maps are obtained.
  • the heat maps can be obtained at the same time
  • the coordinates and confidence levels of the target key points the processing process is simple and intuitive, which improves the accuracy and efficiency of acquiring the target key points, thereby improving the accuracy and efficiency of the overall image processing process.
  • step S11 may include:
  • Step S111 perform key point detection on the target image to obtain a first heat map and a second heat map corresponding to at least two target key points respectively, wherein the response range of the target key points corresponding to the first heat map is larger than that of the second heat map The response range of the corresponding target key point;
  • Step S112 determining the first key point from the first heat map, and obtaining the first confidence level of the first key point according to the first heat map;
  • Step S113 according to the first confidence level, determine the second key point from the second heat map, and combine the second heat map to obtain the second confidence level of the second key point;
  • Step S114 determine the first key point or the second key point as the target key point in the target image, and obtain the confidence level corresponding to the target key point.
  • the first heat map and the second heat map may be two heat maps corresponding to the target key points generated by performing key point detection on the target image.
  • the number of target key points may be multiple. Therefore, in a possible implementation manner, for each target key point, the first popularity corresponding to the target key point may be separately generated map and the second heat map, so that for each target key point, the position and confidence of the target key point can be determined based on its corresponding two heat maps respectively. In a possible implementation manner, some of the target key points may also be selected, and then based on the selected target key points, a first heat map and a second heat map corresponding to the target key points are respectively generated.
  • an overall first heat map and a second heat map may also be generated for all target key points, that is, the first heat map and the second heat map contain the response positions of all target key points, so as to be based on
  • the two overall first heat map and second heat map are used to determine the location and confidence of each or part of the target key points. Subsequent disclosed embodiments are described by taking each target key point to generate a first heat map and a second heat map corresponding to the target key point as an example. Repeat.
  • the response range of the target key point corresponding to the first heat map is greater than the response range of the target key point corresponding to the second heat map, that is, between the first heat map and the target key point
  • the possible location range of the target key point indicated by the first heat map is larger than the location range indicated by the second heat map.
  • 5 shows a schematic diagram of a first heat map and a second heat map according to an embodiment of the present disclosure, wherein part (a) is the first heat map, and part (b) is the second heat map.
  • the first heat map Both the heat map and the second heat map can indicate that the target key point is located in the middle-right position of the heat map, but the range of the target key point delineated by the first heat map is larger than the range of the target key point delineated by the second heat map. That is, the first heat map can perform relatively rough positioning of target key points, while the second heat map can perform relatively fine positioning of target key points.
  • the first key point may be a target key point determined according to the first heat map, and its confidence may be recorded as the first confidence.
  • the second key point may be the target key point determined according to the second heat map, and its confidence may be recorded as the second confidence.
  • the determined first key point and second key point both correspond to the same target key point in the target image, but because they are based on the To generate a heat map and a second heat map, the coordinates of the first key point and the second key point, and the corresponding confidence levels, may have some differences. Therefore, the position and confidence of the target key point in the target image can be finally obtained based on the position and confidence of the first key point and the second key point.
  • the first heat map with relatively coarse positioning results and the second heat map with relatively fine positioning results can be effectively used to comprehensively determine the position and confidence of the target key points in the target image,
  • the accuracy and stability of the target key point positioning prediction in the target image are improved, and the accuracy and stability of the subsequent image processing results are improved.
  • the method of obtaining the first key point including the first confidence degree based on the first heat map in step S112 may refer to the method of determining the coordinates and confidence degree of the target key point based on the heat map in the above-mentioned disclosed embodiments, It is not repeated here.
  • the second key point may be determined from the second heat map and the second confidence level may be obtained through step S113 based on the first confidence level.
  • the implementation manner of step S113 may be flexibly determined according to the actual situation. In a possible implementation manner, step S113 may include:
  • the response area is determined according to the position of the first key point in the first heat map, the second key point is determined from the response area of the second heat map, and the second key point is determined according to the second key point in the first heat map.
  • the response area of the heat map is used to obtain the second confidence of the second key point; if the first confidence is not greater than the first confidence threshold, the second key point is determined from the second heat map, according to the second heat map. , get the second confidence of the second key point.
  • the second key point may be determined in the second heat map in different ways based on the comparison between the first confidence level and the first confidence level threshold and the corresponding second confidence level.
  • the value of the first confidence threshold is not limited in this embodiment of the present disclosure, and may be flexibly determined according to actual conditions. In an example, the first confidence threshold may be set to 0.5.
  • the first confidence level is greater than the first confidence level threshold, it can be shown that the position of the first key point determined in the first heat map is relatively accurate.
  • the position of the response area is determined, and then in the second heat map, the second key point is determined according to the position of the response area, and the second confidence level of the second key point is obtained.
  • the response area may be a preset range delineated according to the position of the first key point in the first heat map, and the size of the preset range may be flexibly determined according to the actual situation, which is not limited in the embodiments of the present disclosure. Since the first heat map and the second heat map are heat maps for the same key point, the position of the response area in the first heat map and the second heat map is consistent, that is, the response area of the first heat map can directly correspond to to the second heatmap. In a possible implementation manner, the response values of the pixels located outside the response area in the second heat map may be set to 0, so that only the response area in the second heat map is retained, and the remaining areas are excluded.
  • the second key point may be determined based on the response area of the second heat map, and the second confidence level of the second key point may be obtained according to the response area of the second heat map.
  • the manner of obtaining the second key point and the second confidence level during implementation can also refer to the above disclosed embodiments, and details are not described herein again.
  • the first confidence level is not greater than the first confidence level threshold, it may indicate that the location accuracy of the first key point determined in the first heat map is low.
  • the second key point and the second confidence level can be directly determined according to the second heat map, and the method of determining the second key point and the second confidence level according to the second heat map is the same. Reference may be made to the above disclosed embodiments, which will not be repeated here.
  • the response area is determined according to the position of the first key point, and then the second key point and the second confidence level are determined according to the response area of the second heat map.
  • the second key point and the second confidence level are directly determined according to the second heat map.
  • the amount of calculated data can be reduced, and on the other hand, the determined second key point can have a high degree of confidence.
  • the first confidence level is not greater than the first confidence level threshold, that is, the position of the first key point determined in the first heat map is relatively low, since the first heat map and the second heat map are independent of each other, By directly determining the second key point according to the second heat map, the target key point with higher confidence can still be obtained. Thus, the accuracy of the final target key points is greatly improved, thereby improving the accuracy of image processing.
  • step S113 may also have other implementation manners, for example, regardless of the size of the first confidence level, the second key point and the second confidence level are directly obtained according to the second heat map.
  • step S114 that is, according to the second confidence level, the first key point or the The second key point is determined as the target key point in the target image, and the confidence corresponding to the target key point is obtained.
  • the implementation manner of step S114 may also be flexibly determined according to the actual situation.
  • step S114 may include: when the second confidence level is greater than the second confidence level threshold, determining the second key point as the target For the target key point in the image, the second confidence level is taken as the confidence level corresponding to the target key point; when the second confidence level is not greater than the second confidence level threshold, the first key point is determined as the target in the target image. key points, and take the first confidence level as the confidence level corresponding to the target key point.
  • the first key point or the second key point may be selected as the target image based on the comparison of the second confidence level and the second confidence level threshold. target key points.
  • the value of the second confidence threshold is not limited in the embodiments of the present disclosure, and can be flexibly determined according to the actual situation. In one example, the second confidence threshold may be set to 0.5; The value of the first confidence is set as the second confidence threshold.
  • the second confidence level is greater than the second confidence level threshold, it can indicate that the position of the second key point determined in the second heat map is relatively accurate. Therefore, the second key point can be used as the target key point of the target image, and the The second confidence of the second key point is used as the confidence of the target key point in the target image. In the case where the second confidence level is not greater than the second threshold, it can indicate that the accuracy of the second key point determined by the second heat map is low. In this case, the first key point can be selected as the target image. For the target key point, the first confidence level of the first key point is used as the confidence level of the target key point in the target image.
  • the first heat map can locate the target key points relatively coarsely, and the second heat map can locate the target key points relatively finely, through the above process, the result of relatively fine positioning of the target key points can be obtained. If the accuracy is relatively accurate, select the fine positioning result to determine the key points of the target, and if the accuracy of the fine positioning result is low, select the coarse positioning result to determine the key points of the target, so that the final target key can be improved as much as possible. The accuracy of the point, and then improve the accuracy of image processing.
  • the target key points or the heat map of the target key points may be obtained by performing key point detection on the target image.
  • the method of performing key point detection on the target image can be flexibly determined according to the actual situation.
  • the detection of target key points can also be achieved through a neural network. Therefore, in a possible implementation manner, performing key point detection on the target image may include: inputting the target image into the first neural network to perform key point detection.
  • the first neural network may be a network with a key point detection function, and its actual implementation function may be flexibly changed according to different implementations of step S11.
  • the first neural network may directly Generate target key point coordinates and target key point confidence according to the target image; in a possible implementation, the first neural network can also generate a plurality of heat maps corresponding to each target key point according to the target image.
  • the heat map generated by the first neural network is post-processed to obtain the target key point coordinates and confidence; in a possible implementation, the first neural network can also generate a plurality of target key points corresponding to each target key point according to the target image.
  • the coordinates and confidence of the target key points are obtained by post-processing the first heat map and the second heat map.
  • the implementation form of the first neural network can also be flexibly determined according to its function and actual situation, and is not limited to the following disclosed embodiments.
  • the first neural network can be implemented by a Unet neural network including an encoder, a decoder and a skip link structure.
  • the first neural network can also be implemented by other neural network models such as Vnet and the like.
  • the key point detection process can be realized through the neural network, thereby effectively improving the stability, efficiency and accuracy of key point detection, and then improving the stability, efficiency and stability of image processing. precision.
  • the neural network can flexibly adjust the structure and implementation method according to the actual situation of key point detection, the flexibility of key point detection can be improved, and then the flexibility of image processing method implementation can be improved.
  • the training method of the first neural network can also be flexibly changed.
  • the first neural network under the condition that the first neural network can realize the function of generating multiple heat maps corresponding to each target key point according to the target image, the first neural network can generate a plurality of heat maps corresponding to each target key point according to the target image.
  • the training image with the location label is used for training, and the training process may include: generating a target heat map corresponding to the target key point position according to the target key point location label; inputting the training image to the first neural network to obtain the heat output of the first neural network.
  • Figure according to the output heat map and the target heat map, determine the loss function of the first neural network; according to the loss function, adjust at least one parameter of the first neural network.
  • the target key point position annotation can indicate the actual position of the target key point in the training image, and the target heat map generated by the target key point position annotation can accurately reflect the response of the target key point.
  • the method of generating the target heat map according to the position annotation of the target key points reference may be made to the process of generating the target heat map according to the coordinates of the target key points in the above disclosed embodiments, which will not be repeated here.
  • the loss function of the first neural network can be determined according to the target heat map and the heat map output by the first neural network based on the training image.
  • the manner of determining the loss function of the first neural network can be flexibly selected according to the actual situation, and is not limited to the following disclosed embodiments.
  • the loss function of the first neural network can be obtained through a mean square error loss function.
  • at least one parameter of the first neural network can be adjusted according to the loss function.
  • the method of adjusting the parameters is also flexible and is not limited to the following embodiments. Propagation and stochastic gradient descent are used to reverse adjust the parameters of the first neural network.
  • the target heat map to supervise the heat map output by the first neural network to determine the loss function of the first neural network, and adjust at least one parameter of the first neural network based on the loss function, so that the heat generated by the first neural network can be generated.
  • the graph is as close to the target heatmap as possible, so that the trained first neural network has high accuracy. Then, the accuracy of the target key points obtained based on the trained first neural network is improved, thereby improving the accuracy of image processing.
  • the training process of the first neural network will also change, which can be flexibly expanded according to the above disclosed embodiments, which will not be repeated here.
  • the target heat map generated according to the target key point position annotation during the training process can be the first heat map.
  • the first target heat map and the second target heat map, the first target heat map and the second target heat map can all be generated by the two-dimensional Gaussian function mentioned in the above disclosed embodiments.
  • the value of ⁇ can be adjusted so that the response range of the target key point in the first heat map is larger than the response range of the target key point in the second heat map, that is, in one example, the generation of the first target heat map
  • the value of ⁇ in the function may be greater than the value of ⁇ in the generating function of the second target heat map, and the other parameter values may remain the same.
  • step S12 the target object in the target image is segmented, and the implementation manner of obtaining the segmentation result of the target object can also be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the target object can be segmented by a method based on pixel gray value; in a possible implementation, the target can be segmented by a method based on level set, active contour model or region growing objects are segmented, etc.
  • the segmentation of the target object can also be achieved through a neural network with a segmentation function. Therefore, in an example, step S12 may include: inputting the target image into the second neural network to segment the target object , to get the segmentation result of the target object.
  • the second neural network may be a neural network with a target object prediction and segmentation function, and its implementation form may be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the second neural network can be implemented with a fully convolutional neural network named Tiramisu, which has downsampling paths, upsampling paths, and skip connections, and uses dense connections in the DenseNet structure As a convolutional block, Dense Block can have better feature reuse effect and obtain more robust features.
  • the densely connected block contains concatenated convolutional layers, and the input of each convolutional layer will be matched with it. The outputs are combined as the input to the next convolutional layer.
  • the training method of the second neural network can be flexibly determined according to the actual situation, and is not limited to the following disclosed embodiments.
  • the segmentation results generated by the second neural network can be supervised by the cross-entropy loss function, and trained using backpropagation and stochastic gradient descent, so that the segmentation results generated by the second neural network are as close as possible. Manually annotated femur segmentation criteria.
  • the segmentation result of the target object can be obtained, which can effectively improve the segmentation accuracy and robustness of the target object, and then improve the accuracy and robustness of image processing.
  • the target object may be a tibia object and other objects with structures on both sides in the human body. Therefore, after the target object is segmented, it may be obtained that the left For the overall segmentation results of the tibia and the right tibia, for the needs of subsequent image processing, the obtained segmentation results may be further post-processed to segment the left and right segmentation results.
  • the way of post-processing the segmentation results can be flexibly determined according to the actual situation.
  • the connected domain can be calculated according to the tibia segmentation result generated by the second neural network, and the two connected domains with the largest area are reserved. The connected domain on the left is the segmentation result of the left tibia, and the connected domain whose center of gravity is on the right is the segmentation result of the right tibia.
  • the second neural network can also directly implement the function of segmenting the left and right tibias, that is, after the target image is input into the second neural network, the second neural network can automatically identify the left tibia object or the right tibia. object, and segment the left tibia object and the right tibia object respectively, thereby outputting the left and right tibia segmentation results respectively.
  • step S13 may include: determining the processing type of the target object according to the target key points.
  • the processing type of the target object can be flexibly determined according to the target object and the application scenario of the target object. Accordingly, the acquisition method of the processing type of the target object can also be determined according to the difference of the target object and the application scenario. Flexibility to change.
  • the processing type of the target object may be determined according to the actual state of the target object.
  • the high tibial osteotomy needs to consider the overall stress state of the lower limb of the human body. Therefore, the selection of the type of high tibial osteotomy also needs to consider the current stress state or stress situation of the human lower limb.
  • the current stress state of the lower limbs of the human body can be determined through the target key points, and then the appropriate treatment type can be determined.
  • the realization process can be flexibly determined according to the actual situation.
  • the calculation parameters related to the processing type can be calculated according to the position of the target key point, and these calculation parameters can be fed back to the human-computer interaction interface.
  • the parameters determine the processing type through experience, and the selected processing type is transmitted to the image processing device through the human-computer interaction interface, so as to realize the acquisition of the processing type.
  • the calculation parameters related to the processing type may be calculated according to the position of the target key point, and the processing type may be directly determined by further calculation based on the calculation parameters.
  • the calculation can also be performed according to the target key points to partially determine the treatment type, such as excluding some impossible treatment types. For example, in the case of determining the treatment type of high tibial osteotomy , in the case that the lower limbs of the human body are determined to be genu varus (such as O-shaped legs) through the target key points, it can be determined that the treatment type will not be medial closed and lateral open.
  • the treatment type can also include There are medial open and lateral closed, and even other surgical methods, such as medial open distal femoral osteotomy or lateral closed distal femoral osteotomy, etc.; After part of the processing types, the remaining selectable processing types can also be fed back to the human-computer interaction interface in the manner proposed by the above disclosed embodiments, so that the processing types can be finally obtained according to the selection of the relevant personnel.
  • At least one processing parameter of the target object may be determined according to the at least one target key point and the segmentation result in step S13.
  • the implementation manner of step S13 can be flexibly determined according to the actual situation of the target object and the processing parameters to be determined, and is not limited to the following disclosed embodiments.
  • there may be multiple target key points at least one processing parameter may include a first processing parameter and a second processing parameter, and step S13 may include:
  • Step S131 according to the processing type, determine the first processing parameter and the second processing parameter to be obtained by the target object;
  • Step S132 obtaining the first processing parameter according to the at least two target key points and the segmentation result
  • Step S132 according to the at least three target key points, combined with the first processing parameters, to obtain the second processing parameters.
  • the first processing parameter may be a processing parameter first obtained based on the target key point and the segmentation result, and which parameters may be included can be flexibly determined according to the actual situation.
  • the image processing method is applied to assist the tibial osteotomy
  • the feed point and the hinge point can be determined according to the target key points and segmentation results
  • the first processing parameter may include the hinge point and/or the feed point.
  • the connecting line between the hinge point and the feed point may constitute an osteotomy line, it is based on the hinge point and the feed point.
  • the cutting point can directly determine the length of the osteotomy line, therefore, in a possible implementation manner, the first processing parameter may include one or more of the hinge point, the cutting point and the length of the osteotomy line.
  • the second processing parameter may be a processing parameter that can be further obtained in combination with the target key point after the first processing parameter is determined, and which parameters may be included can also be flexibly determined according to the actual situation.
  • the image processing method is applied to assist the tibial osteotomy, after the hinge point and the infeed point are determined, it can be further determined based on the determined result and the detection results of the target key points.
  • Target Alignment of Tibial Osteotomy may include a target alignment; after the target alignment of the tibial osteotomy is determined, according to the obtained various target key points, an open or closed alignment can be further obtained. angle, and the distance of opening or closing, etc.
  • the second processing parameter may include one or more of the target force line, the processing angle, and the processing distance.
  • the types of processing parameters that need to be acquired may change flexibly.
  • it may be determined in step S131 which first processing parameters and second processing parameters need to be acquired.
  • the treatment type may include closed inside, closed outside, open inside, and open outside.
  • the determination standard and position of the infeed point in the first processing parameter to be acquired may change accordingly; in one example, as the processing type is Depending on whether it is closed or open, the second processing parameter that needs to be acquired may include a closed angle or an open angle, a closed distance or an open distance, and the like.
  • the first processing parameter may change.
  • the feed point under the inner closed type needs to be based on the hinge point and the closing angle. It is jointly determined that, in this case, the first processing parameter may only contain the hinge point, and the feed point is acquired as the second processing parameter.
  • step S13 can also be flexibly changed, for example, all processing parameters can be obtained directly according to the segmentation result and target key points; Or first obtain some processing parameters according to the segmentation results, and then obtain the remaining processing parameters according to the obtained partial processing parameters in combination with the target key points; or first obtain some processing parameters according to the target key points, and then combine the obtained partial processing parameters with The remaining processing parameters and the like are obtained from the segmentation result, and the implementation process can be flexibly expanded and changed with reference to the following disclosed embodiments based on the actual situation of the target object, which will not be repeated one by one.
  • step S132 may be flexibly determined according to the actual situation of the target object, and is not limited to the following disclosed embodiments.
  • step S132 may include: determining the scope of the first processing parameter according to the target key points ; Combine the scope of the first processing parameter with the segmentation result to obtain the first processing parameter.
  • the approximate range of the first processing parameter may be determined based on the target key points, and then the target object may be classified based on the segmentation result.
  • the location constraints are combined with the determined approximate ranges to obtain the first processing parameters.
  • the first processing parameter may include the hinge point and/or the infeed point, so in a
  • the range to which the hinge point and the feed point belong can be first determined according to the target key point, and then the actual positions of the hinge point and the feed point can be obtained by combining the determined range according to the segmentation result.
  • 6 shows a schematic diagram of acquiring processing parameters according to an embodiment of the present disclosure.
  • the implementation process of determining the hinge point may be: The range of 10 millimeters (mm) below is used as the range of the hinge point, and then in this range, find the intersection point with the outer contour of the tibia in the segmentation result, and move 10mm to the inner side of the tibia based on this intersection point, and get the first A hinge point in a processing parameter, as shown in FIG. 6 , the hinge point 64 is the left end point of the line segment at the proximal end of the left tibia.
  • mm millimeters
  • the process of determining the infeed point may be: taking the acquired range of 15 mm below the medial end point of the tibial plateau as the range of the infeed point, and then in this In the scope, find the intersection point with the inner contour of the tibia in the segmentation result, and use this intersection as the infeed point in the first processing parameter.
  • the infeed point 67 is the right end point of the left proximal tibia line segment. .
  • the hinge point and the infeed point can also be connected, and the connected line segment is used as the osteotomy line, that is, the connection line segment of the proximal left tibia in FIG. 6 .
  • step S133 may include: associating the first processing parameter with at least three target key The points are combined to obtain the target force line; according to the target force line, the first processing parameter and at least one target key point are combined to obtain the processing angle and/or the processing distance.
  • the target force line may be the target position of the lower limb force line in the above disclosed embodiments.
  • how to obtain the target alignment can be flexibly determined according to the different correction purposes of the tibial osteotomy.
  • the correction purpose of the tibial osteotomy may be that the corrected lower limb alignment (that is, the target alignment) passes through the center point of the knee joint.
  • the tibia The purpose of the surgery may also be to correct the alignment of the lower extremity through the Fujisawa point.
  • the Fujisawa point may be a point in the line segment connecting the medial end point of the tibia and the lateral end point of the tibia, and the length of the line segment from the medial end point of the tibia accounts for 62% of the line segment.
  • FIG. 7 shows a schematic diagram of the Fujisawa point according to an embodiment of the present disclosure. As can be seen from FIG. 7 , the position pointed by the arrow is the position of the Fujisawa point. Subsequent disclosed embodiments are described by taking the target line of force passing through the center point of the knee joint as an example, and the implementation manner of the target line of force passing through the Fujisawa point may be expanded with reference to the subsequent disclosed embodiments, which will not be repeated.
  • the realization process of obtaining the target force line may be:
  • the center point 61 of the femoral head makes a ray to the midpoint of the medial end point 62 of the tibial plateau and the lateral end point 63 of the tibial plateau, that is, the center point of the knee joint, and takes the hinge point 64 as the center of the circle, the hinge point 64 and the medial end point 65 of the ankle joint space and the ankle joint.
  • the center of the outer end point 66 of the joint space that is, the distance from the center point of the ankle joint is the radius to make an arc, then the intersection of the arc and the distal end of the above-mentioned ray can be used as the correction target of the center point of the ankle joint.
  • the line segment connecting the bone center points 61 constitutes the target force line.
  • the opening angle and opening distance can be further obtained.
  • the opening angle can be a ray from the hinge point to the center point of the ankle joint, and the ray from the hinge point to the center point of the ankle joint.
  • the angle formed by these two rays can be For the base length of an isosceles triangle with the opening angle as the top angle and the waist length as the length of the osteotomy line, FIG. 8 shows a schematic diagram of the second processing parameter according to an embodiment of the present disclosure, as shown in FIG.
  • the lower limb force line can pass through the center of the knee joint, that is, the position of the target force line is reached.
  • the open angle can be changed to a closed angle, so that an isosceles with an apex angle formed with the hinge point on the medial tibia profile is an isosceles closed angle
  • the two points of the triangle are used to determine the infeed point.
  • the segmentation results can be used, and further, the feed point, hinge point, opening (or closing) angle, and opening (or closing) distance of high tibial osteotomy can be automatically obtained with the help of the acquired target key points, so as to achieve
  • the image processing process with a higher degree of automation is used to assist the tibial osteotomy and improve the efficiency of medical-engineering interaction.
  • the image processing method proposed by the embodiments of the present disclosure may further include the step of image preprocessing before acquiring target key points and/or segmentation results, that is, in a possible implementation manner, the target image may include Preprocessed images, where preprocessing may include image normalization and/or image enhancement.
  • the target image may be a medical image including a target object, such as a whole body X-ray, a lower limb X-ray, or an X-ray of the tibia.
  • a target object such as a whole body X-ray, a lower limb X-ray, or an X-ray of the tibia.
  • a target object such as a whole body X-ray, a lower limb X-ray, or an X-ray of the tibia.
  • a target object such as a whole body X-ray, a lower limb X-ray, or an X-ray of the tibia.
  • image standardization may be performed on the medical images to obtain preprocessed target images.
  • image enhancement may also be performed on the medical image.
  • the image normalization may include one or more of background normalization, pixel pitch normalization, and pixel value normalization.
  • the background normalization method can set the background of the image to be the same color, and the color to be set is not limited.
  • the background of the medical image can be set to be black, and the foreground of the medical image can be set to white.
  • the way to standardize the pixel spacing can be to set the pixel spacing in the medical image to a specified value, and the value of the specified value can be flexibly set according to the actual situation.
  • the pixel spacing can be set to (0.68mm, 0.68 mm).
  • Pixel value normalization can normalize the pixel values in the image to a certain value range, such as between [0, 1], etc.
  • the normalization method is not limited.
  • the medical The pixel values of the pixels in the image are sorted from small to large, and the pixel value at the 3% position is set as the lowest pixel value, the pixel value at the 99% position is set as the highest pixel value, and then the pixel value below the lowest pixel value is set The value of the pixel point is changed to the lowest pixel value, and the value of the pixel point higher than the highest pixel value is the highest pixel value. After the pixel value is changed, the pixel value is normalized to [0,1]. Thus, the normalization of pixel values is completed.
  • CLAHE contrast-limited adaptive histogram equalization algorithm
  • the preprocessed image is obtained as the target image through image standardization and/or image enhancement, which can facilitate the subsequent acquisition and segmentation of target key points for target images with uniform specifications and good image effects, and increase the acquisition and segmentation of target key points. It can also increase the accuracy and segmentation accuracy of acquiring target key points, thereby increasing the convenience and accuracy of image processing.
  • High tibial osteotomy is an effective method for the treatment of early knee osteoarthritis. It can preserve the original joint, correct the gravity line of the knee joint, prevent further wear of cartilage, increase joint stability, relieve pain, improve knee joint function, thereby avoiding or delaying joint replacement as much as possible. Among them, the accuracy of tibial osteotomy position will greatly affect the effect of high tibial osteotomy.
  • the surgical planning method in the related art usually uses the method of manual labeling to locate key points, and the surgical planning process is complicated; the automatic positioning method of key points is difficult to have both high stability and high accuracy, and does not give the prediction confidence of key points. If the key points cannot be accurately predicted due to some reasons (such as poor image quality, non-existence of key points, etc.), the algorithm gives a wrong prediction with a large deviation. At the same time, this method needs to manually set the feed point and hinge point to calculate the subsequent opening (closing) angle and opening (closing) distance, and the efficiency of medical-engineering interaction is not high.
  • the application example of the present disclosure proposes an image processing method.
  • a deep learning model is used to automatically locate key points in a full-length X-ray film of a lower extremity, and the model can use both a coarse localization heat map and a fine localization heat map for localization prediction, and also It has high stability and high accuracy, and can give the prediction confidence of key points;
  • the deep learning model is used to automatically predict the tibial segmentation in the full-length X-ray of the lower extremity, and further use the lateral and medial endpoints of the tibial plateau.
  • the embodiment of the present disclosure proposes an image processing method, which can determine the osteotomy position in the tibial osteotomy process based on the X-ray film of the lower limb.
  • the image processing process can be as follows:
  • the first step is automatic localization of target key points in lower extremity X-rays.
  • knee joint center point can be defined as the midpoint between the medial end point of the tibial plateau and the lateral end point of the tibial plateau
  • ankle joint center point can be defined as the midpoint between the medial end point of the ankle joint space and the lateral end point of the ankle joint space
  • the predicted target key points should at least include the center point of the femoral head, the center point of the ankle joint, the center point of the knee joint, the medial end point of the tibial plateau, and the lateral end point of the tibial plateau, etc.
  • the locating process of target key points can be divided into the following steps:
  • image preprocessing 92 is performed on the input image 91 in sequence according to the following steps: uniformly process the X-ray image into a black background and a white foreground; the pixel spacing of the unified image is (0.68mm, 0.68mm); the pixel values are normalized , first set the values below the 3rd percentile and above the 99th percentile as the 3rd percentile and the 99th percentile, respectively, and then normalize the values to [0,1 ]; and then use the CLAHE method to enhance the local contrast of the image.
  • the preprocessed image is input into the fully convolutional neural network 93 (ie, the first neural network in the above disclosed embodiment).
  • a Unet including an encoder-decoder and a skip link structure can be used.
  • a network is used to generate a coarse positioning heat map (ie, the first heat map in the above disclosed embodiment) and a fine positioning heat map (ie, the second heat map in the above disclosed embodiment) for each target key point.
  • the heat map of the coarse positioning target and the fine positioning target corresponding to the target key point can be calculated according to the actual value (ie the label value) of the position of each target key point in each input training image.
  • Heat map and then supervise the heat map 94 generated by the first neural network through the mean square error loss function, and use backpropagation and stochastic gradient descent to train, so that the heat map generated by the first neural network is as close as possible to the aforementioned target heat map.
  • Both the coarse positioning target heat map and the fine positioning target heat map can be expressed in the form of a two-dimensional Gaussian function shown in the above formula (2). As shown in FIG.
  • the value of ⁇ in the first heat map that is, the heat map of the coarse positioning target is larger than that of the heat map of the fine positioning, so there is a high response value in a larger range.
  • the target heat map in the implementation scheme can also be composed of similar properties (the position closer to the target key point has a larger response value, and the coarse positioning heat map has a higher response value in a larger range than the fine positioning heat map). It is not limited to the form proposed in this application example.
  • post-processing 95 can be performed on them to obtain the positioning results 96 of the target key points.
  • the post-processing of the coarse positioning heat map and the fine positioning heat map can be divided into The following steps:
  • the rough positioning coordinates that is, the coordinates of the first key point in the above disclosed embodiments
  • the rough positioning reliability that is, the first confidence in the above disclosed embodiments
  • the rough location reliability can be calculated by the above formula (1).
  • the fine positioning coordinates that is, the coordinates of the second key point in the above disclosed embodiment
  • the fine position reliability that is, the second confidence in the above disclosed embodiment
  • the coarse positioning If the confidence level is greater than 0.5, it can be considered that the coarse positioning is basically accurate, and the response values within a certain range (that is, the response area in the above-mentioned disclosed embodiment) near the coarse positioning coordinates on the fine positioning heat map can be reserved, and the fine positioning heat map exceeds the The value of the response area range is set to 0, so that the fine positioning coordinates are always near the coarse positioning coordinates; if the coarse positioning reliability is less than or equal to 0.5, the original fine positioning heat map is retained. Then, by a method similar to the coarse positioning heat map, the fine positioning coordinates and the fine positioning reliability are calculated according to the processed or original fine positioning heat map.
  • the prediction result of the final target key point can be selected.
  • the fine positioning coordinates and the fine position reliability are selected as the final output target key point positioning coordinates and confidence; otherwise, the coarse positioning coordinates and the coarse position reliability are selected as the final output target key point positioning coordinates and confidence.
  • Confidence and the like can be calculated according to the correlation coefficient between the predicted heat map and the target heat map corresponding to the positioning coordinates.
  • Fig. 10 shows the flow of automatic segmentation.
  • the process of automatic segmentation of the tibia can be realized by the following steps:
  • the preprocessed image is then fed into a fully convolutional neural network 103 (i.e. the second neural network in the above disclosed embodiment) that predicts tibial segmentation.
  • a fully convolutional neural network named Tiramisu can be used to segment the femur.
  • the Tiramisu network is similar to the Unet network and has a down-sampling path, an up-sampling path, and skip connections.
  • the Tiramisu network structure uses the dense connection block in the DenseNet structure to replace the convolution block composed of concatenated convolution layers, so as to achieve better feature reuse and obtain more robust features.
  • the densely connected block not only contains cascaded convolutional layers, but the input of each convolutional layer is combined with its output as the input of the next convolutional layer.
  • the cross-entropy loss function can be used to supervise the tibia segmentation result 134 generated by the second neural network, and back-propagation and stochastic gradient descent can be used to train the segmentation results generated by the second neural network as much as possible. Close to the manually annotated tibia segmentation standard.
  • the second neural network can realize the prediction of bilateral tibia segmentation, and in a possible implementation manner, the second neural network can also directly realize the segmentation prediction of the left tibia and/or the right tibia.
  • post-processing 105 may be performed on the tibia segmentation result, and the process of post-processing the segmentation result may be as follows: calculating a connected domain for the tibia segmentation result generated by the second neural network, and retaining the two largest connections among them domain, take the connected domain with the center of gravity on the left as the left tibia segmentation result 106, and take the connected domain with the center of gravity on the right as the right tibia segmentation result 107; take the boundaries of these two connected domains as the left tibia respectively Outline and right tibia outline.
  • the third step is to determine various processing parameters of the tibial osteotomy based on the position of the target key points and the contour of the tibia.
  • the infeed point and the hinge point can be obtained by using the target key point and the tibia contour, as well as the geometric position relationship between the infeed point and the hinge point.
  • a point 10mm below the lateral end point of the tibial plateau is obtained on the lateral contour of the tibia, and this point can be used as a hinge point at 10 mm medial to the tibia; 15mm below the medial end point of the tibial plateau is found on the medial contour of the tibia
  • the point can be used as the infeed point, and the line segment connecting the hinge point and the infeed point is the osteotomy line. As shown in FIG. 6 in the above disclosed embodiment, the line segment at the proximal end of the left tibia is the osteotomy line, the left end point is the hinge point 64 , and the right end point is the incision point 67 .
  • the target force line can be determined according to the correction target of the high tibial osteotomy (the lower limb force line can be the line segment connecting the center of the femoral head and the center of the ankle joint, and the target force line can be the correction target of the lower limb force line).
  • the target of correction is the target force line passing through the center of the knee joint
  • a ray can be drawn from the center of the femoral head to the center of the knee joint, with the hinge point as the center of the circle and the distance between the hinge point and the center of the ankle joint as the radius to make an arc.
  • intersection of the arc and the distal end of the ray can be used as the target for the center correction of the ankle joint, and the connecting line segment between the target for the center correction of the ankle joint and the center of the femoral head can be used as the target force line.
  • the opening angle and opening distance can be further determined.
  • the opening angle may be the angle formed by the ray from the hinge point to the center point of the ankle joint and the ray from the hinge point to the center of the ankle joint to correct the target point.
  • the opening distance can be the base length of an isosceles triangle with the opening angle as the top angle and the waist length equal to the length of the osteotomy line.
  • the treatment type of the high tibial osteotomy in the application example of the present disclosure can be replaced from medial open to medial closed, lateral open or lateral closed, then the opening angle, entry point, closing
  • the recommended method of the page point can be changed accordingly.
  • the closed angle can be used instead of the open angle
  • the position of the infeed point can be passed through
  • the apex angle formed with the hinge point on the inner contour of the tibia is The closed angle is determined in this way by the two points of an isosceles triangle.
  • the correction target in the application example of the present disclosure can also be flexibly changed.
  • the correction target can be changed to the target force line passing through the Fujisawa point proposed in the above disclosed embodiment, and the calculation of the target force line and the ankle joint center correction target The method can vary accordingly.
  • the first neural network can be used to automatically locate the target key points in the lower extremity X-ray film, which reduces the process of manually marking the target key points, thereby simplifying the surgical planning process and improving the efficiency of medical-engineering interaction; the first neural network It can use both the coarse positioning heat map and the fine positioning heat map for the positioning prediction of target key points, which has both high stability and high accuracy; at the same time, it can give the confidence of each target key point prediction, so as to reduce the target key points as much as possible For some reasons (such as poor image quality, non-existence of target key points, etc.), a wrong prediction result with excessive deviation can be obtained when it cannot be accurately predicted, and at the same time, it is convenient to use the missing value completion method to predict the failed target key points. to complete.
  • the application example of the present disclosure can use the second neural network to automatically realize the segmentation of the tibia in the X-ray of the lower extremity, and further use the lateral and medial endpoints of the tibial plateau to automatically determine the entry point, hinge point, opening of the high tibial osteotomy. (closed) angle, open (closed) distance.
  • the prediction of various processing parameters in high tibial osteotomy with high automation and high accuracy can be realized, the operation planning process can be simplified, and the efficiency of medical-engineering interaction can be improved.
  • the application example of the present disclosure can provide high-stability, high-accuracy, and high-consistency automatic positioning of key points for full-length X-ray films of the lower extremity, and solves the problem of time-consuming and labor-intensive manual labeling of key points by doctors in the traditional reading process and the problem of junior doctors.
  • the labeling consistency may be poor; this scheme can automatically segment the tibia on the full-length X-ray of the lower extremity, and automatically recommend the incision according to the type of high tibial osteotomy and the correction target selected by the physician using key points and tibial contours Points, hinge points, opening or closing angles, opening or closing distances, eliminating the need for physicians to manually perform surgical planning steps through complex drawing, geometric operations, and measurements, with a high degree of automation.
  • the application example of the present disclosure can simplify the surgical planning process of high tibial osteotomy, and improve the efficiency of medical-engineering interaction.
  • the image processing method of the embodiment of the present disclosure is not limited to be applied to the above-mentioned lower limb X-ray image processing, nor is it limited to only determining the position of the tibial osteotomy, but can be applied to any image processing and any related processing In the process of parameter determination, this embodiment of the present disclosure does not limit this.
  • the embodiments of the present disclosure also provide image processing apparatuses, electronic devices, computer-readable storage media, and program products, all of which can be used to implement any image processing method provided by the embodiments of the present disclosure.
  • image processing apparatuses electronic devices, computer-readable storage media, and program products, all of which can be used to implement any image processing method provided by the embodiments of the present disclosure.
  • program products all of which can be used to implement any image processing method provided by the embodiments of the present disclosure.
  • FIG. 11 shows a block diagram of an image processing apparatus according to an embodiment of the present disclosure.
  • the image processing apparatus may be a terminal device, a server, or other processing devices.
  • the terminal device may be user equipment (User Equipment, UE), mobile device, user terminal, terminal, cellular phone, cordless phone, Personal Digital Assistant (PDA), handheld device, computing device, in-vehicle device, available wearable devices, etc.
  • the image processing apparatus may be implemented by a processor invoking computer-readable instructions stored in a memory. As shown in FIG.
  • the image processing apparatus 110 may include: a target key point acquisition module 111, configured to acquire target key points of the target image; and a segmentation module 112, configured to segment the target object in the target image to obtain the target image.
  • the processing type acquisition module 113 is configured to acquire the processing type of the target object;
  • the processing parameter determination module 114 is configured to determine at least one processing parameter of the target object according to the target key point, the segmentation result and the processing type.
  • the target key point acquiring module 111 is configured to: perform key point detection on the target image to obtain at least two target key points containing confidence.
  • the target key point acquisition module 111 is configured to: perform key point detection on the target image to obtain heat maps corresponding to at least two target key points respectively; obtain target key points corresponding to at least two heat maps coordinates and confidence.
  • the target key point acquiring module 111 is configured to: perform key point detection on the target image, and obtain a first heat map and a second heat map corresponding to at least two target key points respectively, wherein the first heat map
  • the response range of the target key point corresponding to the heat map is larger than the response range of the target key point corresponding to the second heat map; the first key point is determined from the first heat map, and the first key point of the first key point is obtained according to the first heat map.
  • the target key point obtaining module 111 is further configured to: in the case that the first confidence level is greater than the first confidence level threshold, determine the response area according to the position of the first key point in the first heat map , determine the second key point from the response area of the second heat map, and obtain the second confidence level of the second key point according to the response area of the second heat map; when the first confidence level is not greater than the first confidence level threshold In this case, the second key point is determined from the second heat map, and the second confidence level of the second key point is obtained according to the second heat map.
  • the target key point obtaining module 111 is further configured to: when the second confidence level is greater than the second confidence level threshold, determine the second key point as the target key point in the target image, and set the second key point as the target key point in the target image.
  • the second confidence level is used as the confidence level corresponding to the target key point; when the second confidence level is not greater than the second confidence level threshold, the first key point is determined as the target key point in the target image, and the first confidence level is determined as the target key point in the target image. as the confidence corresponding to the target keypoint.
  • the target key point acquiring module 111 is configured to: input the target image into the first neural network for key point detection.
  • the first neural network is trained by training images containing target key point position annotations, and the training includes: generating a target heat map corresponding to the target key point position according to the target key point position annotation; The image is input to the first neural network, and the heat map output by the first neural network is obtained; the loss function of the first neural network is determined according to the output heat map and the target heat map; at least one parameter of the first neural network is adjusted according to the loss function .
  • the processing type obtaining module 113 is configured to: determine the processing type of the target object according to the target key point.
  • At least two target key points there are at least two target key points, at least one processing parameter includes a first processing parameter and a second processing parameter, and the processing parameter determining module 114 is configured to: determine the target object to be acquired according to the processing type The first processing parameter and the second processing parameter; the first processing parameter is obtained according to the at least two target key points and the segmentation result; the second processing parameter is obtained according to the at least three target key points in combination with the first processing parameter.
  • the target image includes a preprocessed image
  • the preprocessing includes image normalization and/or image enhancement.
  • the target object includes a tibial object
  • the treatment type includes: medial closed, lateral closed, medial open, or lateral open
  • at least one treatment parameter includes a feed point, a hinge point, a target force One or more of Line, Process Angle, and Process Distance.
  • Embodiments of the present disclosure further provide a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the foregoing method is implemented.
  • the computer-readable storage medium may be a non-volatile computer-readable storage medium.
  • An embodiment of the present disclosure further provides an electronic device, comprising: a processor; a memory configured to store instructions executable by the processor; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.
  • the electronic device may be provided as a terminal, server or other form of device.
  • Embodiments of the present disclosure also provide a computer program product, including computer-readable codes.
  • a processor in the device executes the image processing method for implementing the image processing method provided by any of the above embodiments. instruction.
  • Embodiments of the present disclosure further provide another computer program product configured to store computer-readable instructions, which, when executed, cause the computer to perform the operations of the image processing method provided by any of the foregoing embodiments.
  • FIG. 12 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure.
  • electronic device 800 may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, fitness device, and personal digital assistant, among other terminals.
  • an electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 , and communication component 816 .
  • a processing component 802 a memory 804
  • a power supply component 806 a multimedia component 808, an audio component 810, an input/output (I/O) interface 812 , sensor component 814 , and communication component 816 .
  • I/O input/output
  • the processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations.
  • the processing component 802 can include one or more processors 820 to execute instructions to perform all or some of the steps of the methods described above.
  • processing component 802 may include one or more modules that facilitate interaction between processing component 802 and other components.
  • processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.
  • Memory 804 is configured to store various types of data to support operation at electronic device 800 . Examples of such data include instructions for any application or method configured to operate on electronic device 800, contact data, phonebook data, messages, pictures, videos, and the like.
  • Memory 804 may be implemented by any type of volatile or non-volatile storage device or combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM) , Electrically Erasable Programmable Read-Only Memory), Erasable Programmable Read-Only Memory (EPROM, Erasable Programmable Read-Only Memory), Programmable Read-Only Memory (PROM, Programmable Read-Only Memory), Read-Only Memory (ROM, Read Only Memory), magnetic memory, flash memory, magnetic disk or optical disk.
  • SRAM Static Random-Access Memory
  • EEPROM Electrically Erasable Programmable Read-Only Memory
  • EPROM Erasable Programmable Read-Only Memory
  • PROM Programmable Read-Only Memory
  • Power supply assembly 806 provides power to various components of electronic device 800 .
  • Power supply components 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800 .
  • Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user.
  • the screen may include a liquid crystal display (LCD, Liquid Crystal Display) and a touch panel (TP, TouchPanel).
  • the screen may be implemented as a touch screen to receive input signals from a user.
  • the touch panel includes one or more touch sensors to sense touch, swipe, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or swipe action, but also detect the duration and pressure associated with the touch or swipe action.
  • the multimedia component 808 includes a front-facing camera and/or a rear-facing camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera may receive external multimedia data.
  • Each of the front and rear cameras can be a fixed optical lens system or have focal length and optical zoom capability.
  • Audio component 810 is configured to output and/or input audio signals.
  • the audio component 810 includes a microphone (MIC, Microphone) that is configured to receive external audio signals when the electronic device 800 is in operating modes, such as calling mode, recording mode, and voice recognition mode.
  • the received audio signal may be further stored in memory 804 or transmitted via communication component 816 .
  • audio component 810 also includes a speaker for outputting audio signals.
  • the I/O interface 812 provides an interface between the processing component 802 and a peripheral interface module, which may be a keyboard, a click wheel, a button, or the like. These buttons may include, but are not limited to: home button, volume buttons, start button, and lock button.
  • Sensor assembly 814 includes one or more sensors for providing status assessment of various aspects of electronic device 800 .
  • the sensor assembly 814 can detect the open/closed state of the electronic device 800 and the relative positioning of the components, such as the display and the keypad of the electronic device 800, and the sensor assembly 814 can also detect the electronic device 800 or one of the electronic device 800. Changes in the position of components, presence or absence of user contact with the electronic device 800 , orientation or acceleration/deceleration of the electronic device 800 and changes in the temperature of the electronic device 800 .
  • Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects in the absence of any physical contact.
  • the sensor assembly 814 may also include a light sensor, such as a Complementary Metal-Oxide-Semiconductor (CMOS) or Charge Coupled Device (CCD) image sensor, configured for use in imaging applications.
  • CMOS Complementary Metal-Oxide-Semiconductor
  • CCD Charge Coupled Device
  • the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.
  • Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices.
  • the electronic device 800 can access a wireless network based on a communication standard, such as Wireless Fidelity (Wi-Fi, Wireless Fidelity), the second generation mobile communication technology (2G, The 2nd Generation,) or the third generation mobile communication technology (3G, The 3nd Generation,) or a combination thereof.
  • the communication component 816 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel.
  • the communication component 816 further includes a Near Field Communication (NFC, Near Field Communication) module to facilitate short-range communication.
  • NFC Near Field Communication
  • the NFC module may be based on Radio Frequency Identification (RFID, Radio Frequency Identification) technology, Infrared Data Association (IrDA, Infrared Data Association) technology, Ultra Wide Band (UWB, Ultra Wide Band) technology, Bluetooth (BT, Blue Tooth) technology and other technologies to achieve.
  • RFID Radio Frequency Identification
  • IrDA Infrared Data Association
  • UWB Ultra Wide Band
  • Bluetooth Bluetooth
  • electronic device 800 may be implemented by one or more Application Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD) Processing Device), Programmable Logic Device (PLD, Programmable Logic Device), Field Programmable Gate Array (FPGA, Field Programmable Gate Array), controller, microcontroller, microprocessor, or other electronic component implementation for performing the above method.
  • ASIC Application Specific Integrated Circuit
  • DSP Digital Signal Processor
  • DSPD Digital Signal Processing Device
  • PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • controller microcontroller, microprocessor, or other electronic component implementation for performing the above method.
  • a non-volatile computer-readable storage medium such as a memory 804 comprising computer program instructions executable by the processor 820 of the electronic device 800 to perform the above method is also provided.
  • FIG. 13 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure.
  • the electronic device 1900 may be provided as a server.
  • electronic device 1900 includes a processing component 1922, which may include one or more processors, and a memory resource, represented by memory 1932, for storing instructions executable by processing component 1922, such as applications.
  • An application program stored in memory 1932 may include one or more modules, each corresponding to a set of instructions.
  • the processing component 1922 is configured to execute instructions to perform the above-described methods.
  • the electronic device 1900 may also include a power supply assembly 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an I/O interface 1958.
  • Electronic device 1900 may operate based on an operating system stored in memory 1932, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSD TM, or similar systems.
  • a non-volatile computer-readable storage medium such as memory 1932 comprising computer program instructions executable by processing component 1922 of electronic device 1900 to perform the above-described method.
  • Embodiments of the present disclosure may be systems, methods and/or computer program products.
  • the computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the embodiments of the present disclosure.
  • a computer-readable storage medium may be a tangible device that can hold and store instructions for use by the instruction execution device.
  • the computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing.
  • Computer-readable storage media may include: portable computer disks, hard disks, random access memory (RAM, Random Access Memory), read-only memory, erasable programmable read-only memory (EPROM or flash memory), static random access memory ( SRAM), Portable Compact Disc Read-Only Memory (CD-ROM, Compact Disc Read-Only Memory), Digital Versatile Disc (DVD, Digital Video Disc), Memory Stick, Floppy Disk, Mechanical Encoding Device, such as one on which instructions are stored Punched cards or raised structures in grooves, and any suitable combination of the above.
  • RAM random access memory
  • EPROM or flash memory erasable programmable read-only memory
  • SRAM static random access memory
  • CD-ROM Portable Compact Disc Read-Only Memory
  • CD-ROM Compact Disc Read-Only Memory
  • DVD Digital Versatile Disc
  • Memory Stick Floppy Disk
  • Mechanical Encoding Device such as one on which instructions are stored Punched cards or raised structures in grooves, and any suitable combination of the above.
  • Computer-readable storage media are not to be construed as transient signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (eg, light pulses through fiber optic cables), or through electrical wires transmitted electrical signals.
  • the computer readable program instructions described herein may be downloaded to various computing/processing devices from a computer readable storage medium, or to an external computer or external storage device over a network such as the Internet, a local area network, a wide area network, and/or a wireless network.
  • the network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers.
  • a network adapter card or network interface in each computing/processing device receives computer-readable program instructions from a network and forwards the computer-readable program instructions for storage in a computer-readable storage medium in each computing/processing device .
  • Computer program instructions for carrying out operations of the present disclosure may be assembly instructions, Industry Standard Architecture (ISA) instructions, machine instructions, machine-dependent instructions, pseudocode, firmware instructions, state setting data, or in one or more source or object code written in any combination of programming languages, including object-oriented programming languages such as Smalltalk, C++, etc., as well as conventional procedural programming languages such as C or similar programming languages.
  • the computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server implement.
  • the remote computer may be connected to the user's computer through any kind of network, including a Local Area Network (LAN) or Wide Area Network (WAN), or may be connected to an external computer (eg, using Internet service provider to connect via the Internet).
  • LAN Local Area Network
  • WAN Wide Area Network
  • electronic circuits such as programmable logic circuits, field programmable gate arrays, or programmable logic arrays, that can execute computer readable program instructions are personalized by utilizing state information of computer readable program instructions , thereby implementing various aspects of the embodiments of the present disclosure.
  • These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer or other programmable data processing apparatus to produce a machine that causes the instructions when executed by the processor of the computer or other programmable data processing apparatus , resulting in means for implementing the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • These computer readable program instructions can also be stored in a computer readable storage medium, these instructions cause a computer, programmable data processing apparatus and/or other equipment to operate in a specific manner, so that the computer readable medium on which the instructions are stored includes An article of manufacture comprising instructions for implementing various aspects of the functions/acts specified in one or more blocks of the flowchart and/or block diagrams.
  • Computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other equipment to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other equipment to produce a computer-implemented process , thereby causing instructions executing on a computer, other programmable data processing apparatus, or other device to implement the functions/acts specified in one or more blocks of the flowcharts and/or block diagrams.
  • each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more functions for implementing the specified logical function(s) executable instructions.
  • the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.
  • each block of the block diagrams and/or flowchart illustrations, and combinations of blocks in the block diagrams and/or flowchart illustrations can be implemented in dedicated hardware-based systems that perform the specified functions or actions , or can be implemented in a combination of dedicated hardware and computer instructions.
  • the computer program product can be implemented in hardware, software or a combination thereof.
  • the computer program product may be embodied as a computer storage medium, and in another optional embodiment, the computer program product may be embodied as a software product, such as a software development kit (SDK, Software Development Kit), etc. Wait.
  • SDK software development kit
  • Software Development Kit Software Development Kit
  • Embodiments of the present disclosure relate to an image processing method and apparatus, an electronic device, a storage medium, and a program product.
  • the method includes: obtaining target key points of a target image; segmenting a target object in the target image to obtain a segmentation result of the target object; obtaining a processing type of the target object; The segmentation result and the processing type determine at least one processing parameter of the target object.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Software Systems (AREA)
  • Evolutionary Computation (AREA)
  • Computational Linguistics (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Biomedical Technology (AREA)
  • Mathematical Physics (AREA)
  • Data Mining & Analysis (AREA)
  • Artificial Intelligence (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Medical Informatics (AREA)
  • Nuclear Medicine, Radiotherapy & Molecular Imaging (AREA)
  • Radiology & Medical Imaging (AREA)
  • Quality & Reliability (AREA)
  • Image Analysis (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

本公开实施例涉及一种图像处理方法及装置、电子设备、存储介质和程序产品。所述方法包括：获取目标图像的目标关键点；对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；获取所述目标对象的处理类型；根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。

Description

图像处理方法及装置、电子设备、存储介质和程序产品
相关申请的交叉引用
本公开基于申请号为202010646714.5、申请日为2020年07月07日的中国专利申请提出，并要求该中国专利申请的优先权，该中国专利申请的全部内容在此以全文引入的方式引入本公开。
技术领域
本公开涉及图像处理技术领域，尤其涉及一种图像处理方法及装置、电子设备、存储介质和程序产品。
背景技术
随着患有膝关节疾病的人越来越多，将膝关节受力力线转移到中央甚至稍偏外侧的部分，从而减少内侧间室的压力，避免或延缓关节置换的胫骨高位截骨术越发重要。胫骨高位截骨中的一个重要环节在于确定胫骨截骨的位置，胫骨截骨位置的准确性将大大影响胫骨高位截骨的效果。
发明内容
本公开实施例提出了一种图像处理技术方案。
根据本公开实施例的一方面，提供了一种图像处理方法，包括：
获取目标图像的目标关键点；对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；获取所述目标对象的处理类型；根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。
在一种可能的实现方式中，所述获取目标图像的目标关键点，包括：对所述目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。
通过对目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。可以在获取目标关键点的同时，确定这些目标关键点是否准确，在因某些原因(如图像质量过差、目标关键点不存在等)无法精确预测目标关键点的情况下，可以基于置信度对一些准确率较低的目标关键点进行排除或是额外处理补全，从而提升目标关键点的准确性，继而提升后续得到的处理参数的准确性。
在一种可能的实现方式中，所述获取目标图像的目标关键点，包括：对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图；获取至少两个所述热度图对应的所述目标关键点的坐标以及置信度。
通过上述过程，可以通过得到目标关键点对应的热度图的形式来同时确定目标关键点的坐标和置信度，处理过程简单直观，提升了获取目标关键点的精度和效率，从而提升图像处理整体过程的精度和效率。
在一种可能的实现方式中，所述获取目标图像的目标关键点，包括：对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的第一热度图和第二热度图，其中，所述第一热度图对应的所述目标关键点的响应范围大于所述第二热度图对应的所述目标关键点的响应范围；从所述第一热度图中确定第一关键点，根据所述第一热度图，得到所述第一关键点的第一置信度；根据所述第一置信度，从所述第二热度图中确定第二关键点，结合所述第二热度图，得到所述第二关键点的第二置信度；根据所述第二置信度，将所述第一关键点或所述第二关键点确定为所述目标图像中的目标关键点，并得到与所述目标关键点对应的置信度。
通过分别得到与目标关键点对应的第一热度图和第二热度图，并从第一热度图中确定第一关键点及对应的第一置信度，从第二热度图中确定第二关键点及对应的置信度，通过上述过程，可以有效地利用具有较粗糙定位结果的第一热度图和具有较精细定位结果的第二热度图，综合确定目标图像中目标关键点的位置和置信度，提升了目标图像中目标关键点定位预测的准确性和稳定性，继而提升了后续图像处理结果的准确度和稳定性。
在一种可能的实现方式中，所述根据所述第一置信度，从所述第二热度图中确定第二关键点，结合所述第二热度图，得到所述第二关键点的第二置信度，包括：在所述第一置信度大于第一置信度阈值的情况下，根据所述第一关键点在所述第一热度图中的位置确定响应区域，从所述第二热度图的所述响应区域内中确定第二关键点，根据所述第二热度图的所述响应区域，得到所述第二关键点的第二置信度；在所述第一置信度不大于所述第一置信度阈值的情况下，从所述第二热度图中确定第二关键点，根据所述第二热度图，得到所述第二关键点的第二置信度。
通过上述过程，在第一置信度大于第一置信度阈值，即第一热度图中确定的第一关键点的位置 比较准确的情况下，由于目标图像中的目标关键点在响应区域内的可能性比较大，直接在第二热度图的响应区域中来确定第二关键点，一方面可以减小计算的数据量，另一方面也可以使得确定的第二关键点具有较高的置信度；而在第一置信度不大于第一置信度阈值，即第一热度图中确定第一关键点的位置准确的较低的情况下，由于第一热度图和第二热度图之间相互独立，直接根据第二热度图来确定第二关键点，可以仍得到具有较高置信度的目标关键点。从而大大提升了最终得到的目标关键点的准确程度，继而提升了图像处理的精度。
在一种可能的实现方式中，所述根据所述第二置信度，将所述第一关键点或所述第二关键点确定为所述目标图像中的目标关键点，并得到与所述目标关键点对应的置信度，包括：在所述第二置信度大于第二置信度阈值的情况下，将所述第二关键点确定为所述目标图像中的目标关键点，将所述第二置信度作为与所述目标关键点对应的置信度；在所述第二置信度不大于所述第二置信度阈值的情况下，将所述第一关键点确定为所述目标图像中的目标关键点，将所述第一置信度作为与所述目标关键点对应的置信度。
上述过程进一步基于第二置信度与第二置信度阈值的比较情况，来选定将第一关键点还是第二关键点作为目标图像的目标关键点，并确定目标图像中目标关键点的置信度。从而提升了目标图像中目标关键点定位预测的准确性和稳定性，继而提升了后续图像处理结果的准确度和稳定性。
在一种可能的实现方式中，所述对所述目标图像进行关键点检测，包括：将所述目标图像输入至第一神经网络进行关键点检测。
通过将目标图像输入至第一神经网络进行关键点检测，可以通过神经网络实现关键点的检测过程，从而有效提升关键点检测的稳定性、效率和精度，继而提升图像处理的稳定性、效率和精度。同时，由于神经网络可以根据关键点检测的实际情况灵活调整结构和实现方式，因此，可以提升关键点检测的灵活性，继而提升图像处理方法实现的灵活性。
在一种可能的实现方式中，所述第一神经网络通过包含目标关键点位置标注的训练图像进行训练，所述训练包括：根据所述目标关键点位置标注，生成与所述目标关键点位置对应的目标热度图；将所述训练图像输入至第一神经网络，得到所述第一神经网络输出的热度图；根据所述输出的热度图与所述目标热度图，确定所述第一神经网络的损失函数；根据所述损失函数，调整所述第一神经网络的至少一个参数。
利用目标热度图，对第一神经网络输出的热度图进行监督，来确定第一神经网络的损失函数，并基于损失函数调整第一神经网络的至少一个参数，可以使得第一神经网络生成的热度图尽量接近目标热度图，从而使得训练后的第一神经网络具有较高的精度。继而提升基于此训练后的第一神经网络获得的目标关键点的精度，从而提升图像处理的精度。
在一种可能的实现方式中，所述获取所述目标对象的处理类型，包括：根据所述目标关键点，确定所述目标对象的处理类型。
通过目标关键点确定目标对象的处理类型，从而使得目标对象的处理类型的获取方式可以根据目标对象以及应用场景的不同也灵活的产生变化。
在一种可能的实现方式中，所述目标关键点为至少两个，所述至少一个处理参数包括第一处理参数和第二处理参数，所述根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数，包括：根据所述处理类型，确定所述目标对象待获取的第一处理参数和第二处理参数；根据至少两个所述目标关键点以及所述分割结果，获取所述第一处理参数；根据至少三个所述目标关键点，结合所述第一处理参数，获取所述第二处理参数。
上述第一处理参数可以是基于目标关键点以及分割结果首先获得的处理参数，第二处理参数可以是在确定了第一处理参数后，结合目标关键点可以进一步获取的处理参数，从而实现了随着处理类型的不同，获取不同的处理参数，继而提升图像处理方法实现的准确性和灵活性。
在一种可能的实现方式中，所述目标图像包括经过预处理的图像，所述预处理包括图像标准化和/或图像增强。
通过图像标准化和/或图像增强来得到经过预处理的图像作为目标图像，可以便于后续对具有统一规格且具有较好图像效果的目标图像进行目标关键点获取和分割，增加目标关键点获取和分割的实现便捷程度，也可以增加获取目标关键点的精度和分割精度，继而增加图像处理的实现便捷性和精度。
在一种可能的实现方式中，所述目标对象包括胫骨对象；所述处理类型包括：内侧闭合式、外侧闭合式、内侧开放式或外侧开放式；所述至少一个处理参数包括进刀点、合页点、目标力线、处理角度以及处理距离中的一个或多个。
这样，在目标对象为胫骨对象的情况下，随着处理类型是在内侧还是外侧的不同，所需要获取的第一处理参数中进刀点的确定标准与位置可能随之发生变化；随着处理类型是闭合式还是开放式的不同，所需要获取的第二处理参数可能包含的是闭合角度或开放角度、闭合距离或开放距离等。从而根据实际的应用场景灵活选择处理类型和对应的处理参数，使得后续的图像处理结果具有更好的处理效果。
根据本公开实施例的一方面，提供了一种图像处理装置，包括：目标关键点获取模块，配置为获取目标图像的目标关键点；分割模块，配置为对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；处理类型获取模块，配置为获取所述目标对象的处理类型；处理参数确定模块，配置为根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。
根据本公开实施例的一方面，提供了一种电子设备，包括：处理器；配置为存储处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器存储的指令，以执行上述图像处理方法。
根据本公开实施例的一方面，提供了一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述图像处理方法。
根据本公开实施例的一方面，提供了一种计算机程序产品，包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备中的处理器执行上述图像处理方法。
在本公开实施例中，通过获取目标图像的目标关键点和目标对象的处理类型，并对目标图像中的目标对象进行分割，得到目标对象的分割结果，从而根据目标关键点、分割结果以及处理类型来确定目标对象的至少一个处理参数。通过上述过程，可以利用分割结果所表明的目标对象在目标图像中的位置，与获取的目标关键点进行结合，来得到在当前的处理类型下，针对目标对象更为准确的处理参数，大大提升了图像处理的精度和准确性。
应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，而非限制本公开实施例。
根据下面参考附图对示例性实施例的详细说明，本公开实施例的其它特征及方面将变得清楚。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，这些附图示出了符合本公开实施例的实施例，并与说明书一起用于说明本公开实施例的技术方案。
图1示出根据本公开一实施例的图像处理方法的流程图。
图2示出根据本公开一实施例的胫骨高位截骨的原理示意图。
图3示出根据本公开一实施例的胫骨对象的处理类型示意图。
图4示出根据本公开一实施例的胫骨高位截骨的X光片示意图。
图5示出根据本公开一实施例的第一热度图和第二热度图的示意图。
图6示出根据本公开一实施例的获取处理参数的示意图。
图7示出根据本公开一实施例的藤泽点示意图。
图8示出根据本公开一实施例的第二处理参数示意图。
图9示出根据本公开一应用示例中目标关键点的自动定位示意图。
图10示出根据本公开一应用示例中胫骨的自动分割示意图。
图11示出根据本公开一实施例的图像处理装置的框图。
图12示出根据本公开实施例的一种电子设备的框图。
图13示出根据本公开实施例的一种电子设备的框图。
具体实施方式
以下将参考附图详细说明本公开实施例的各种示例性实施例、特征和方面。附图中相同的附图标记表示功能相同或相似的元件。尽管在附图中示出了实施例的各种方面，但是除非特别指出，不必按比例绘制附图。
在这里专用的词“示例性”意为用作例子、实施例或说明性。这里作为“示例性”所说明的任何实施例不必解释为优于或好于其它实施例。
本文中术语“和/或”，仅仅是一种描述关联对象的关联关系，表示可以存在三种关系，例如，A和/或B，可以表示：单独存在A，同时存在A和B，单独存在B这三种情况。另外，本文中术语“至少一种”表示多种中的任意一种或多种中的至少两种的任意组合，例如，包括A、B、C中的至少一种，可以表示包括从A、B和C构成的集合中选择的任意一个或多个元素。
另外，为了更好地说明本公开实施例，在下文的具体实施方式中给出了众多的实现细节。本领 域技术人员应当理解，没有某些细节，本公开实施例同样可以实施。在一些实例中，对于本领域技术人员熟知的方法、手段、元件和电路未作详细描述，以便于凸显本公开实施例的主旨。
图1示出根据本公开一实施例的图像处理方法的流程图，该方法可以应用于图像处理装置，图像处理装置可以为终端设备、服务器或者其他处理设备等。其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该图像处理方法可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。如图1所示，所述图像处理方法可以包括：
步骤S11，获取目标图像的目标关键点。
步骤S12，对目标图像中的目标对象进行分割，得到目标对象的分割结果。
步骤S13，获取目标对象的处理类型。
步骤S14，根据目标关键点、分割结果以及处理类型，确定目标对象的至少一个处理参数。
其中，目标图像可以是包含有目标对象的任意形式的图像，其实现方式可以根据目标对象的实际情况灵活决定。
目标对象可以是具有处理需求的任意对象，其实现形式可以根据本公开实施例提出的图像处理方法的实际应用场景所灵活决定。在一种可能的实现方式中，本公开实施例提出的方法可以应用于手术规划过程中，则目标对象可以是手术对象，比如人体的某些部位、器官或是组织等，目标图像则可以是包含手术对象的医学图像，比如X光片、计算机体层摄影(CT，Computed Tomography)图像或是磁共振成像(MRI，Magnetic Resonance Imaging)等。在一个示例中，本公开实施例提出的图像处理方法可以应用于胫骨高位截骨的手术规划过程中，则目标对象可以是需要执行高位截骨的部位，即胫骨对象(可以是左胫骨也可以是右胫骨，根据胫骨截骨的实际需求确定)等，目标图像则可以是包含有这些对象的医学图像，比如全身X光片、下肢X光片或是胫骨部位的X光片等。
图2示出根据本公开一实施例的胫骨高位截骨的原理示意图，如图2所示，针对膝关节骨关节炎的治疗，有4个阶段：内侧应力集中阶段21、胫骨近端截骨阶段22、力线向外侧转移阶段23和内侧压力缓解阶段24，可以看出，胫骨高位截骨可以在内侧的软骨出现严重磨损之前，将膝关节受力力线转移到中央甚至稍偏外侧的部分，从而减少内侧间室的压力，避免或延缓膝关节的置换。即胫骨高位截骨可能需要依据人体整个下肢处于站立位的受力状态来确定处理参数。因此，在本公开各公开实施例均以目标对象为胫骨对象，目标图像为下肢X光片为例对图像处理过程进行阐述，目标对象为其他对象或是目标图像为其他形式图像的情形可以参考后续各公开实施例进行灵活扩展，不再一一列举。
目标图像的目标关键点的数量和实现形式同样可以根据目标对象的实现形式以及图像处理方法的应用场景灵活决定。需要注意的是，由于图像处理方法的应用场景不同，因此目标图像中的目标关键点，可以包含在目标对象内，也可以位于目标对象以外，根据实际情况决定即可，在此不做限制。在一种可能的实现方式中，在目标对象为胫骨对象且图像处理方法应用于辅助进行胫骨截骨的情况下，由于上述公开实施例已提到，胫骨截骨可能需要考虑到人体下肢整体的受力状态来确定处理参数，因此获取的目标关键点可以既包含属于目标对象的目标关键点，也包含位于目标对象以外的目标关键点。在一种可能的实现方式中，目标关键点可以包含有股骨头中心点、踝关节中心点(可以定义为踝关节间隙内侧端点与踝关节间隙外侧端点的中点)、膝关节中心点(可以定义为胫骨平台内侧端点与胫骨平台外侧端点的中点)、胫骨平台内侧端点以及胫骨平台外侧端点，在一种可能的实现方式中，目标关键点可以在包含有上述各目标关键点的基础上，额外包含有股骨大转子尖点、股骨内侧髁点、股骨外侧髁点、踝关节间隙内侧端点以及踝关节间隙外侧端点中的一个或多个等。在实施过程中如何获取目标图像的目标关键点，其实现形式可以根据实际情况灵活决定，在此先不做展开，详见后续各公开实施例。
目标对象的分割结果在本公开实施例中不做限制，可以根据分割的实际实现形式灵活决定，在此先不做展开，详见下述各公开实施例。需要注意的是，在本公开实施例中，步骤S11与步骤S12相互独立，二者的实现顺序不受限制。即可以先获取目标图像的目标关键点，再对目标图像中的目标对象进行分割；也可以先对目标对象进行分割再获取目标图像的目标关键点；或是同时获取目标图像的目标关键点并对目标对象进行分割等，根据实际情况灵活选择即可。
除了通过步骤S11获取目标图像的目标关键点以及通过步骤S12获取目标对象的分割结果以外，在本公开实施例中，还可以通过步骤S13获取目标对象的处理类型。对于一些目标对象来说，其在对应的应用场景中可能存在多种处理方式，随着处理方式的不同，需要确定的目标对象的处理参数 自然也会发生变化，因此可以通过获取目标对象的实际处理类型来明确最终要确定的处理参数。目标对象的处理类型可以根据目标对象以及目标对象的应用场景所共同灵活确定，图3示出根据本公开一实施例的胫骨对象的处理类型示意图，从图3中可以看出，在一个示例中，在目标对象为胫骨对象且图像处理方法应用于辅助进行胫骨截骨的情况下，如图3所示，胫骨对象的处理类型可以包括有内侧闭合式31、外侧闭合式32、内侧开放式33或外侧开放式34等。在实施过程中如何获取目标对象的处理方式，其实现形式可以根据实际情况灵活决定，详见下述各公开实施例，在此先不做展开。
在得到了目标图像的目标关键点、目标对象的分割结果以及目标对象的处理类型以后，可以根据目标关键点、分割结果以及处理类型，通过步骤S14来确定目标对象的至少一个处理参数。其中，目标对象的处理参数的数量以及实现形式同样可以根据目标对象的实现形式以及图像处理方法的应用场景灵活决定。
图4示出根据本公开一实施例的胫骨高位截骨的X光片示意图，其中(a)部分为包含双腿的下肢全长X光片，(b)部分为胫骨高位截骨术前的单腿下肢全长X光片，(c)部分为胫骨高位截骨术后的单腿下肢全长X光片，其中股骨头中心41与踝关节中心42之间的连线段为下肢力线。从图4中可以看出，胫骨高位截骨可以通过对胫骨进行截骨，对下肢力线进行矫正实现。由于需要对胫骨进行截骨，需要考虑到截骨的位置以及截骨的长度等。因此，在一种可能的实现方式中，在目标对象为胫骨对象且图像处理方法应用于辅助进行胫骨截骨的情况下，目标对象的处理参数可以包括有进刀点、合页点、目标力线、处理角度以及处理距离中的一个或多个等。其中，目标力线可以是上述公开实施例中下肢力线的目标位置所对应的线段，比如可以是踝关节矫正后的目标点与股骨头中心点之间的连线线段等；处理角度可以是胫骨截骨中的手术角度，处理角度可以随着处理类型的不同而发生变化，比如处理类型为内侧闭合式或外侧闭合式的情况下，处理角度可以为闭合的角度，处理类型为内侧开放式或外侧开放式的情况下，处理角度可以为开放的角度；同样地，处理距离可以是胫骨截骨中的截骨距离，其也可以随着处理类型被划分为闭合距离或开放距离等。
在实施过程中如何根据目标关键点、分割结果与处理类型，来确定上述一个或多个处理参数，其确定过程可以根据实际情况灵活决定，详见下述各公开实施例，在此先不做展开。
在本公开实施例中，通过获取目标图像的目标关键点和目标对象的处理类型，并对目标图像中的目标对象进行分割，得到目标对象的分割结果，从而根据目标关键点、分割结果以及处理类型来确定目标对象的至少一个处理参数。通过上述过程，可以利用分割结果所表明的目标对象在目标图像中的位置，与获取的目标关键点进行结合，来得到在当前的处理类型下，针对目标对象更为准确的处理参数，大大提升了图像处理的精度和准确性。
如上述公开实施例所述，步骤S11中获取目标图像目标关键点的方式可以根据实际情况灵活决定。在一种可能的实现方式中，步骤S11可以包括：对目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。
其中，如上述各公开实施例所述，目标关键点的数量可以根据实际情况灵活决定，在一种可能的实现方式中，在目标对象为胫骨的情况下，目标关键点可以至少包含有股骨头中心点、踝关节中心点、膝关节中心点、胫骨平台内侧端点以及胫骨平台外侧端点。
对目标图像进行目标关键点检测的方式可以根据实际情况灵活决定，详见下述各公开实施例，在此先不做展开。由于不同的目标关键点检测可能会产生不同的检测结果，即得到的目标关键点可能不是完全准确，因此，可以通过置信度来反应得到的目标关键点的准确程度。在实施过程中如何确定目标关键点的置信度，其确定方式可以灵活决定。在一种可能的实现方式中，可以直接根据检测到的各目标关键点的相对位置来确定每个目标关键点的置信度，从而直接通过对目标图像进行目标关键点检测来得到包含置信度的目标关键点。在一种可能的实现方式中，也可以通过其他方式来确定目标关键点的置信度，详见下述各公开实施例，在此先不做展开。
通过对目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。可以在获取目标关键点的同时，确定这些目标关键点是否准确，在因某些原因(如图像质量过差、目标关键点不存在等)无法精确预测目标关键点的情况下，可以基于置信度对一些准确率较低的目标关键点进行排除或是额外处理补全，从而提升目标关键点的准确性，继而提升后续得到的处理参数的准确性。
对目标关键点进行额外处理补全的方式可以根据实际情况灵活决定，在本公开实施例中不做限制。在一种可能的实现方式中，可以通过缺失值补全方法来补全这些目标关键点，即利用置信度较高的目标关键点进行推测，来确定置信度较低的目标关键点的特征向量，继而确定置信度较低的目标关键点位置。
在一种可能的实现方式中，步骤S11可以包括：对目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图；获取至少两个热度图对应的目标关键点的坐标以及置信度。
其中，热度图可以是目标关键点的响应热度图。在一种可能的实现方式中，热度图的大小可以与目标图像一致。在热度图中，距离目标关键点的位置越近的像素点，可以具有更高的响应值，从而可以通过热度图中各像素点的响应值来确定目标关键点在目标图像中的位置。对目标图像进行关键点检测来得到热度图的方式可以根据实际情况灵活决定，详见下述各公开实施例，在此不做展开。
如上述各公开实施例所述，目标关键点的数量可以为一个也可以为多个，因此，相应地，热度图的数量也可以根据实际情况灵活决定。在一种可能的实现方式中，可以分别根据每一个目标关键点得到对应的热度图，即每个目标关键点分别对应一个热度图；在一种可能的实现方式中，也可以根据所有目标关键点来得到对应的热度图，即一个热度图中包含所有目标关键点。后续各公开实施例均以每个目标关键点分别对应一个热度图的实现过程来进行说明，一个热度图中包含所有目标关键点的实现过程可以参考后续各公开实施例进行相应扩展，不再赘述。
在每个目标关键点分别对应一个热度图的情况下，如何根据热度图来确定对应目标关键点的坐标的方式可以根据实际情况灵活决定，不局限于下述各公开实施例。
在一种可能的实现方式中，可以将热度图中具有最高响应值的像素点作为目标关键点，则该具有最高响应值的像素点在热度图中的坐标，即可以作为目标关键点在目标图像中的坐标。在一种可能的实现方式中，也可以首先基于热度图来确定热度图中的高响应区域，高响应区域的确定方式可以根据实际情况灵活设定，比如可以遍历热度图中的每个像素点，将响应值高于某一设定阈值的像素点均作为高响应区域中的像素点，从而在热度图中确定出高响应区域。在确定了热度图的高响应区域后，可以进一步将高响应区域的重心作为目标关键点，则该高响应区域的重心在热度图中的坐标，即可以作为目标关键点在目标图像中的坐标。
在另一种可能的实现方式中，在得到目标关键点坐标的同时，还可以得到目标关键点的置信度。置信度的定义可以参考上述各公开实施例，得到目标关键点的置信度的方式也可以如上述各公开实施例所述灵活决定。由于本公开实施例可以通过得到目标关键点对应的热度图的形式来确定目标关键点的坐标，因此在一种可能的实现方式中，可以进一步利用热度图来确定目标关键点的置信度。在实施过程中如何根据热度图来确定目标关键点的置信度，其实现形式也可以根据实际情况灵活决定，不局限于下述各公开实施例。
在一种可能的实现方式中，根据热度图确定目标关键点的置信度的过程可以包括：根据目标关键点的响应值，从热度图中选定至少一个包含目标关键点的区域；根据目标关键点的响应值，结合至少一个包含目标关键点的区域的区域参数，确定目标关键点的置信度。
其中，从热度图中选定的包含目标关键点的区域的选定方式可以根据实际情况灵活设定。在一种可能的实现方式中，可以将目标关键点的响应值记为m，由于热度图中越接近目标关键点的像素点响应值越高，因此可以通过遍历热度图，选定其中响应值大于a i*m的像素点，则这些像素点构成的区域自然可以包含有目标关键点。通过更改a i的值，可以得到多个不同的包含目标关键点的区域，a i的值与选定的包含目标关键点的区域的数量可以根据实际情况灵活决定，不局限于下述各公开实施例。在一个示例中，可以选定四个包含目标关键点的区域，则这四个包含目标关键点的区域可以分别对应四个a i的值，分别记为a 0、a 1、a 2和a 3，在一个示例中，这四个a i的值可以设定为：a 0＝0.8；a 1＝0.6；a 2＝0.4；a 3＝0.2。
在得到了这些包含目标关键点的区域后，可以确定这些区域的区域参数，并根据确定的区域参数与目标关键点的响应值来确定目标关键点的置信度。包含目标关键点的区域的区域参数，其实现形式可以根据实际情况灵活决定。在一种可能的实现方式中，区域参数可以是区域的周长c i，在一种可能的实现方式中，区域参数也可以是区域的面积s i。在一种可能的实现方式中，区域参数也可以是根据区域的周长与面积共同确定的参数，如s i/c i 2。
随着区域参数实现形式的不同，确定目标关键点置信度的方式也可以灵活发生变化，在一个示例中，在区域参数为根据区域的周长与面积所共同确定的参数的情况下，目标关键点的置信度的计算方式可以为如下公式(1)：
Figure PCTCN2020138438-appb-000001
其中，Confidence为目标关键点的置信度，π为圆周率，m为目标关键点的响应值，M为预设的目标关键点的目标响应值，s i为包含目标关键点的区域的面积，c i为包含目标关键点的区域的周长。
通过上述公开实施例可以看出，在一种可能的实现方式中，可以通过多个包含有目标关键点的区域的区域参数来确定目标关键点的置信度。在一种可能的实现方式中，也可以通过其他的方式来确定目标关键点的置信度。在一种可能的实现方式中，根据热度图确定目标关键点的置信度的过程可以包括：根据目标关键点的坐标，生成与目标关键点位置对应的目标热度图；对与目标关键点对应的热度图进行归一化，得到第一概率分布；对目标热度图进行归一化，得到第二概率分布；将第一概率分布与第二概率分布的相关系数，作为目标关键点的置信度。
其中，与目标关键点对应的热度图即通过对目标图像进行关键点检测所得到的热度图，而目标热度图则是根据目标关键点坐标所生成的热度图，即根据该热度图中确定的目标关键点坐标，可以反向再生成一个热度图作为目标热度图。根据目标关键点坐标生成目标热度图的方式可以根据实际情况灵活决定，不局限于下述公开实施例。在一种可能的实现方式中，可以通过目标关键点坐标，结合二维高斯函数，生成目标热度图，二维高斯函数的实现形式可以根据实际情况灵活决定，在一个示例中，根据目标关键点坐标生成的目标热度图的方式可以通过如下公式(2)：
Figure PCTCN2020138438-appb-000002
其中，f(x,y)为目标热度图对应的二维高斯分布函数，x为目标热度图中像素点的横坐标，y为目标热度图中像素点的纵坐标，M为预设的目标关键点的目标响应值，x 0为目标关键点的横坐标，y 0为目标关键点的纵坐标，e为自然常数，σ为预设的目标关键点的响应范围。
在得到了目标热度图后，可以分别对热度图和目标热度图进行归一化，来得到热度图的第一概率分布以及目标热度图的第二概率分布，并将第一概率分别和第二概率分布之间的相关系数，来作为目标关键点的置信度。
通过对目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图，继而获取至少两个热度图对应的目标关键点的坐标以及置信度，通过上述过程，可以基于热度图同时获取目标关键点的坐标和置信度，处理过程简单直观，提升了获取目标关键点的精度和效率，从而提升图像处理整体过程的精度和效率。
在一种可能的实现方式中，步骤S11可以包括：
步骤S111，对目标图像进行关键点检测，得到至少两个目标关键点分别对应的第一热度图和第二热度图，其中，第一热度图对应的目标关键点的响应范围大于第二热度图对应的目标关键点的响应范围；
步骤S112，从第一热度图中确定第一关键点，根据第一热度图，得到第一关键点的第一置信度；
步骤S113，根据第一置信度，从第二热度图中确定第二关键点，结合第二热度图，得到第二关键点的第二置信度；
步骤S114，根据第二置信度，将第一关键点或第二关键点确定为目标图像中的目标关键点，并得到与目标关键点对应的置信度。
其中，第一热度图和第二热度图可以是通过对目标图像进行关键点检测生成的与目标关键点对应的两个热度图。如上述各公开实施例所述，目标关键点的数量可以为多个，因此，在一种可能的实现方式中，可以针对每个目标关键点，分别生成与该目标关键点对应的第一热度图和第二热度图，从而对每个目标关键点，可以分别基于其对应的两个热度图来确定该目标关键点的位置与置信度。在一种可能的实现方式中，也可以选取其中部分目标关键点，然后基于选取的这些目标关键点分别生成与该目标关键点对应的第一热度图和第二热度图。在一种可能的实现方式中，也可以针对所有目标关键点生成总体的第一热度图和第二热度图，即第一热度图和第二热度图包含所有目标关键点的响应位置，从而基于这两个总体的第一热度图和第二热度图来确定每个或部分目标关键点的位置与置信度。后续各公开实施例均以每个目标关键点分别生成与该目标关键点对应的第一热度图和第二热度图为例进行说明，其余的情况可以参考后续各公开实施例进行扩展，不再赘述。
如上述公开实施例所述，在一种可能的实现方式中，第一热度图对应的目标关键点的响应范围大于第二热度图对应的目标关键点的响应范围，即在第一热度图与第二热度图对应同一目标关键点的情况下，第一热度图表明的目标关键点可能所在的位置范围要大于第二热度图所表明的位置范围。图5示出根据本公开一实施例的第一热度图和第二热度图的示意图，其中(a)部分为第一热度图，(b)部分为第二热度图，可以看出，第一热度图和第二热度图都可以表明目标关键点位于所在热度图的中间偏右的位置，但是第一热度图圈定的目标关键点的范围要大于第二热度图圈定的目标关键点的范围，即第一热度图可以对目标关键点进行较为粗糙的定位，而第二热度图可以对目标关键点进行 较为精细的定位。
第一关键点可以是根据第一热度图所确定的目标关键点，其置信度可以记为第一置信度。第二关键点则可以是根据第二热度图所确定的目标关键点，其置信度可以记为第二置信度。对于同一目标关键点所分别生成的第一热度图和第二热度图来说，其确定的第一关键点和第二关键点均对应目标图像中的同一目标关键点，不过由于其分别基于第一热度图和第二热度图来生成，第一关键点和第二关键点的坐标，以及对应的置信度，可能会有一些差异。因此，可以进一步基于第一关键点和第二关键点的位置和置信度，来最终得到目标图像中目标关键点的位置和置信度。
通过分别得到与目标关键点对应的第一热度图和第二热度图，并从第一热度图中确定第一关键点及对应的第一置信度，从第二热度图中确定第二关键点及对应的置信度，通过上述过程，可以有效地利用具有较粗糙定位结果的第一热度图和具有较精细定位结果的第二热度图，综合确定目标图像中目标关键点的位置和置信度，提升了目标图像中目标关键点定位预测的准确性和稳定性，继而提升了后续图像处理结果的准确度和稳定性。
在一些实施方式中，步骤S112中基于第一热度图来得到包含第一置信度的第一关键点的方式可以参考上述公开实施例中基于热度图确定目标关键点的坐标及置信度的方式，在此不再赘述。在确定了第一关键点和第一置信度后，可以基于第一置信度，通过步骤S113来从第二热度图中确定第二关键点并得到第二置信度。步骤S113的实现方式可以根据实际情况灵活决定，在一种可能的实现方式中，步骤S113可以包括：
在第一置信度大于第一置信度阈值的情况下，根据第一关键点在第一热度图中的位置确定响应区域，从第二热度图的响应区域中确定第二关键点，根据第二热度图的响应区域，得到第二关键点的第二置信度；在第一置信度不大于第一置信度阈值的情况下，从第二热度图中确定第二关键点，根据第二热度图，得到第二关键点的第二置信度。
通过上述公开实施例可以看出，在一种可能的实现方式中，可以基于第一置信度与第一置信度阈值的比较情况，来以不同的方式在第二热度图中确定第二关键点及对应的第二置信度。其中，第一置信度阈值的取值在本公开实施例中不做限制，可以根据实际情况灵活决定，在一个示例中，可以将第一置信度阈值设置为0.5。
在第一置信度大于第一置信度阈值的情况下，可以表明第一热度图中确定的第一关键点的位置比较准确，因此，可以进一步根据第一热度图中表明的第一关键点的位置，来确定响应区域，继而在第二热度图中，根据响应区域的位置来确定第二关键点，并得到第二关键点的第二置信度。
其中，响应区域可以是根据第一热度图中第一关键点位置所圈定的一个预设范围，这一预设范围的大小可以根据实际情况灵活决定，在本公开实施例不做限定。由于第一热度图与第二热度图是针对同一关键点的热度图，因此响应区域在第一热度图与第二热度图中的位置是一致的，即第一热度图的响应区域可以直接对应到第二热度图中。在一种可能的实现方式中，可以将第二热度图中位于响应区域以外的像素点的响应值均设置为0，从而仅保留第二热度图中的响应区域，而将其余区域进行排除。
在将响应区域对应到第二热度图后，可以基于第二热度图的响应区域来确定第二关键点，并根据第二热度图的响应区域，得到第二关键点的第二置信度。在实施中得到第二关键点和第二置信度的方式同样可以参考上述各公开实施例，在此不再赘述。
在第一置信度不大于第一置信度阈值的情况下，可以表明第一热度图中确定的第一关键点的位置准确度较低。此时，可以不考虑第一热度图的对应结果，直接根据第二热度图来确定第二关键点和第二置信度，根据第二热度图确定第二关键点和第二置信度的方式同样可以参考上述各公开实施例，在此不再赘述。
通过在第一置信度大于第一置信度阈值的情况下根据第一关键点的位置确定响应区域，继而根据第二热度图的响应区域来确定第二关键点和第二置信度，在第一置信度不大于第一置信度阈值的情况下直接根据第二热度图来确定第二关键点和第二置信度。通过上述过程，在第一置信度大于第一置信度阈值，即第一热度图中确定的第一关键点的位置比较准确的情况下，由于目标图像中的目标关键点在响应区域内的可能性比较大，直接在第二热度图的响应区域中来确定第二关键点，一方面可以减小计算的数据量，另一方面也可以使得确定的第二关键点具有较高的置信度。而在第一置信度不大于第一置信度阈值，即第一热度图中确定第一关键点的位置准确的较低的情况下，由于第一热度图和第二热度图之间相互独立，直接根据第二热度图来确定第二关键点，可以仍得到具有较高置信度的目标关键点。从而大大提升了最终得到的目标关键点的准确程度，继而提升了图像处理的精度。
在一种可能的实现方式中，步骤S113也可以具有其他的实现方式，比如不考虑第一置信度的大小，而是直接根据第二热度图得到第二关键点和第二置信度。
在分别得到了第一关键点和对应的第一置信度，以及第二关键点和对应的第二置信度以后，可以通过步骤S114，即根据第二置信度，来将第一关键点或第二关键点确定为目标图像中的目标关键点，并得到与目标关键点对应的置信度。步骤S114的实现方式也可以根据实际情况灵活决定，在一种可能的实现方式中，步骤S114可以包括：在第二置信度大于第二置信度阈值的情况下，将第二关键点确定为目标图像中的目标关键点，将第二置信度作为与目标关键点对应的置信度；在第二置信度不大于第二置信度阈值的情况下，将第一关键点确定为目标图像中的目标关键点，将第一置信度作为与目标关键点对应的置信度。
通过上述公开实施例可以看出，在一种可能的实现方式中，可以基于第二置信度与第二置信度阈值的比较情况，来选定将第一关键点还是第二关键点作为目标图像的目标关键点。其中，第二置信度阈值的取值在本公开实施例中不做限制，可以根据实际情况灵活决定，在一个示例中，可以将第二置信度阈值设置为0.5；在一个示例中，也可以将第一置信度的值设置为第二置信度阈值。
在第二置信度大于第二置信度阈值的情况下，可以表明第二热度图中确定的第二关键点的位置比较准确，因此，可以将第二关键点作为目标图像的目标关键点，将第二关键点的第二置信度作为目标图像中目标关键点的置信度。在第二置信度不大于第二阈值的情况下，则可以表明第二热度图确定的第二关键点的准确率较低，在这种情况下，可以选择将第一关键点作为目标图像的目标关键点，将第一关键点的第一置信度作为目标图像中目标关键点的置信度。
由于第一热度图可以对目标关键点进行较为粗糙的定位，第二热度图可以对目标关键点进行较为精细的定位，因此，通过上述过程，可以在对目标关键点进行较为精细的定位的结果比较准确的情况下，选用精细的定位结果确定目标关键点，在精细的定位结果准确度较低的情况下选择较为粗糙的定位结果确定目标关键点，从而可以尽可能地提升最终得到的目标关键点的准确性，继而提升图像处理的精度。
在一些可能的实施例中，无论是上述何种实现步骤S11的方式，均可以通过对目标图像进行关键点检测的方式来得到目标关键点或目标关键点的热度图。在一些实施方式中，对目标图像进行关键点检测的方式可以根据实际情况灵活决定，在一种可能的实现方式中，可以通过特定的关键点检测算法或是关键点热度图生成算法来实现，在一种可能的实现方式中，也可以通过神经网络来实现目标关键点的检测。因此，在一种可能的实现方式中，对目标图像进行关键点检测，可以包括：将目标图像输入至第一神经网络进行关键点检测。
其中，第一神经网络可以是具有关键点检测功能的网络，其实际实现功能可以根据步骤S11的实现方式的不同而灵活发生变化，比如在一种可能的实现方式中，第一神经网络可以直接根据目标图像生成目标关键点坐标和目标关键点置信度；在一种可能的实现方式中，第一神经网络也可以根据目标图像生成多个分别与每个目标关键点对应的热度图，通过对第一神经网络生成的热度图进行后处理得到目标关键点坐标和置信度；在一种可能的实现方式中，第一神经网络也可以根据目标图像生成多个分别与每个目标关键点对应的第一热度图和第二热度图，通过对第一热度图和第二热度图进行后处理得到目标关键点的坐标和置信度等。
第一神经网络的实现形式也可以根据其功能和实际情况灵活决定，不局限于下述公开实施例。在一种可能的实现方式中，第一神经网络可以通过一个包含编码器、解码器以及跳跃链接结构的Unet神经网络来实现。在一种可能的实现方式中，第一神经网络也可以通过其他的神经网络模型如Vnet等来实现。
通过将目标图像输入至第一神经网络进行关键点检测，可以通过神经网络实现关键点的检测过程，从而有效提升关键点检测的稳定性、效率和精度，继而提升图像处理的稳定性、效率和精度。同时，由于神经网络可以根据关键点检测的实际情况灵活调整结构和实现方式，因此，可以提升关键点检测的灵活性，继而提升图像处理方法实现的灵活性。
随着第一神经网络实现功能与实现形式的不同，第一神经网络的训练方式也可以灵活发生变化。在一种可能的实现方式中，在第一神经网络可以实现根据目标图像生成多个分别与每个目标关键点对应的热度图这一功能的情况下，第一神经网络可以通过包含目标关键点位置标注的训练图像进行训练，训练过程可以包括：根据目标关键点位置标注，生成与目标关键点位置对应的目标热度图；将训练图像输入至第一神经网络，得到第一神经网络输出的热度图；根据输出的热度图与目标热度图，确定第一神经网络的损失函数；根据损失函数，调整第一神经网络的至少一个参数。
其中，目标关键点位置标注可以表明训练图像中目标关键点的实际位置，通过目标关键点位置 标注生成的目标热度图，则可以准确地反应目标关键点的响应情况。根据目标关键点位置标注生成目标热度图的方式可以参考上述公开实施例中根据目标关键点坐标生成的目标热度图的过程，在此不再赘述。
在得到目标热度图后，可以根据目标热度图和第一神经网络基于训练图像输出的热度图，来确定第一神经网络的损失函数。确定第一神经网络的损失函数的方式可以根据实际情况灵活选择，不局限于下述公开实施例。在一个示例中，可以通过均方误差损失函数，来得到第一神经网络的损失函数。在得到了第一神经网络的损失函数后，可以根据损失函数调整第一神经网络的至少一个参数，调整参数的方式同样灵活，不局限于下述实施例，在一个示例中，可以通过反向传播及随机梯度下降法来反向调整第一神经网络的参数。
利用目标热度图，对第一神经网络输出的热度图进行监督，来确定第一神经网络的损失函数，并基于损失函数调整第一神经网络的至少一个参数，可以使得第一神经网络生成的热度图尽量接近目标热度图，从而使得训练后的第一神经网络具有较高的精度。继而提升基于此训练后的第一神经网络获得的目标关键点的精度，从而提升图像处理的精度。
在第一神经网络实现的功能发生变化的情况下，第一神经网络的训练过程也将发生变化，可以根据上述各公开实施例灵活扩展，在此不再一一赘述。需要注意的是，在第一神经网络可以实现根据目标图像生成第一热度图和第二热度图这一功能的情况下，其训练过程中根据目标关键点位置标注生成的目标热度图可以为第一目标热度图和第二目标热度图，第一目标热度图和第二目标热度图均可以通过上述公开实施例提到的二维高斯函数进行生成。在一个示例中，可以通过调整σ的值，使得第一热度图中目标关键点的响应范围大于第二热度图中目标关键点的响应范围，即在一个示例中，第一目标热度图的生成函数内σ的值可以大于第二目标热度图的生成函数内σ的值，其余参数值则可以均保持相同。
步骤S12中对目标图像中的目标对象进行分割，得到目标对象的分割结果的实现方式也可以根据实际情况灵活决定，不局限于下述各公开实施例。在一种可能的实现方式中，可以通过基于像素灰度值的方法对目标对象进行分割；在一种可能的实现方式中，可以通过基于水平集、主动轮廓模型或区域生长的方法来对目标对象进行分割等。在一种可能的实现方式中，也可以通过具有分割功能的神经网络来实现目标对象的分割，因此，在一个示例中，步骤S12可以包括：将目标图像输入至第二神经网络进行目标对象分割，得到目标对象的分割结果。
其中，第二神经网络可以是具有目标对象预测分割功能的神经网络，其实现形式可以根据实际情况灵活决定，不局限于下述公开实施例。在一种可能的实现方式中，第二神经网络可以采用名称为Tiramisu的全卷积神经网络进行实现，Tiramisu神经网络具有降采样路径、升采样路径和跳跃连接，并采用DenseNet结构中的密集连接块(Dense Block)作为卷积块，可以具有更好的特征复用效果，得到更鲁棒的特征，其中，密集连接块中包含级联卷积层，且每一个卷积层的输入将与其输出合并作为下一个卷积层的输入。
第二神经网络的训练方式可以根据实际情况灵活决定，不局限于下述公开实施例。在一种可能的实现方式中，可以通过交叉熵损失函数对第二神经网络生成的分割结果进行监督，并使用反向传播及随机梯度下降法训练，使第二神经网络生成的分割结果尽量接近人工标注的股骨分割标准。
通过将目标图像输入至第二神经网络，得到目标对象的分割结果，可以有效提升目标对象的分割精度和鲁棒性，继而提升图像处理的精度和鲁棒性。
如上述各公开实施例所述，在一种可能的实现方式中，目标对象可以为胫骨对象等在人体中具有两侧结构的对象，因此，在对目标对象进行分割后，可能得到的是左胫骨与右胫骨的总体分割结果，出于后续图像处理的需要，还可以对得到的分割结果进一步进行后处理，来将左右两个分割结果进行切分。对分割结果进行后处理的方式可以根据实际情况灵活决定，在一种可能的实现方式中，可以根据第二神经网络生成的胫骨分割结果计算连通域，保留面积最大的两个连通域，将重心在左侧的连通域作为左侧胫骨分割结果，将重心在右侧的连通域作为右侧胫骨分割结果。
在一种可能的实现方式中，第二神经网络也可以直接实现左右胫骨的分割功能，即将目标图像输入第二神经网络后，第二神经网络可以自动识别其中的左侧胫骨对象或右侧胫骨对象，并分别对左侧胫骨对象和右侧胫骨对象进行分割，从而分别输出左侧胫骨分割结果和右侧胫骨分割结果。
除了获取目标对象的目标关键点以及目标对象的分割结果以外，还可以通过步骤S13获取目标对象的处理类型。目标对象处理类型的获取方式可以根据实际情况灵活决定，在一种可能的实现方式中，步骤S13可以包括：根据目标关键点，确定目标对象的处理类型。
上述公开实施例已经提到，目标对象的处理类型可以根据目标对象以及目标对象的应用场景所 共同灵活确定，因此相应地，目标对象的处理类型的获取方式可以根据目标对象以及应用场景的不同也灵活的产生变化。
在一种可能的实现方式中，在目标对象为胫骨对象且图像处理方法应用于辅助进行胫骨截骨的情况下，可以根据目标对象的实际状态来确定目标对象的处理类型。上述公开实施例已经提到，胫骨高位截骨需要考虑到人体下肢整体的受力状态，因此胫骨高位截骨类型的选择，也需要考虑到人体下肢当前的受力状态或受力情况，在一种可能的实现方式中，可以通过目标关键点来确定人体下肢当前的受力状态，继而确定合适的处理类型。
如何根据目标关键点确定目标对象的处理类型，其实现过程可以根据实际情况灵活决定。在一种可能的实现方式中，可以根据目标关键点位置计算出与处理类型相关的计算参数，并将这些计算参数反馈到人机交互界面中，相关人员可以根据人机交互界面中反馈的计算参数通过经验确定处理类型，并通过人机交互界面将选定的处理类型传递至图像处理装置中，从而实现处理类型的获取。在一种可能的实现方式中，可以根据目标关键点位置计算出与处理类型相关的计算参数，并基于计算参数直接进一步计算确定处理类型。在一种可能的实现方式中，也可以根据目标关键点进行计算从而部分确定处理类型，比如排除掉部分不可能实现的处理类型，举例来说，在确定胫骨高位截骨的处理类型的情况下，在通过目标关键点确定人体下肢为膝内翻(如O形腿)的情况下，则可以确定处理类型不会为内侧闭合式与外侧开放式，在这种情况下，处理类型还可以包括有内侧开放式与外侧闭合式，甚至还可以包含有其他的手术方式，比如内侧开放式股骨远端截骨或外侧闭合式股骨远端截骨等；在一些可能的实施例中，在排除掉部分处理类型以后，还可以将剩余可选择的处理类型，通过上述公开实施例提出的方式反馈到人机交互界面，从而根据相关人员的选择，来最终获取处理类型。在一种可能的实现方式中，还可以通过包含目标关键点与处理类型标注的训练图像，训练得到一个具有确定处理类型功能的神经网络，将获取了目标关键点的目标图像输入至该神经网络中，可以输出与该目标图像对应的处理类型等。
在通过上述任意公开实施例得到目标关键点以及目标对象的分割结果后，可以通过步骤S13根据至少一个目标关键点，结合分割结果，确定目标对象的至少一个处理参数。其中，步骤S13的实现方式，可以根据目标对象以及需要确定的处理参数的实际情况灵活决定，不局限于下述各公开实施例。在一种可能的实现方式中，目标关键点可以为多个，至少一个处理参数可以包括第一处理参数和第二处理参数，步骤S13可以包括：
步骤S131，根据处理类型，确定目标对象待获取的第一处理参数以及第二处理参数；
步骤S132，根据至少两个目标关键点以及分割结果，获取第一处理参数；
步骤S132，根据至少三个目标关键点，结合第一处理参数，获取第二处理参数。
其中，第一处理参数可以是基于目标关键点以及分割结果首先获得的处理参数，可能包含哪些参数可以根据实际情况灵活决定。在目标对象为胫骨对象图像处理方法应用于辅助进行胫骨截骨的情况下，由于进刀点与合页点可以首先根据目标关键点和分割结果所确定，因此在一种可能的实现方式中，第一处理参数可以包括合页点和/或进刀点，在一些可能的实施例中，由于合页点和进刀点二者的连线可以构成截骨线，因此基于合页点和进刀点可以直接确定截骨线的长度，因此，在一种可能的实现方式中，第一处理参数可以包含合页点、进刀点以及截骨线长度中的一个或多个。
第二处理参数可以是在确定了第一处理参数后，结合目标关键点可以进一步获取的处理参数，可能包含哪些参数也可以根据实际情况灵活决定。在目标对象为胫骨对象图像处理方法应用于辅助进行胫骨截骨的情况下，在确定了合页点与进刀点以后，还可以基于确定的结果，结合目标关键点的检测结果，来进一步确定胫骨截骨的目标力线。在一种可能的实现方式中，第二处理参数可以包括目标力线；在确定了胫骨截骨的目标力线后，根据已经获得的各类目标关键点，还可以进一步得出开放或闭合的角度，以及开放或闭合的距离等。在一种可能的实现方式中，第二处理参数可以包括目标力线、处理角度以及处理距离中的一个或多个。
如上述各公开实施例所述，随着处理类型的不同，需要获取的处理参数的类型可能灵活发生变化。在一种可能的实现方式中，在通过步骤S132确定第一处理参数以及根据步骤S133确定第二处理参数之前，可以首先通过步骤S131确定需要获取哪些第一处理参数和第二处理参数。举例来说，由于处理类型可以包括内侧闭合式、外侧闭合式、内侧开放式以及外侧开放式。在一个示例中，随着处理类型是在内侧还是外侧的不同，所需要获取的第一处理参数中进刀点的确定标准与位置可能随之发生变化；在一个示例中，随着处理类型是闭合式还是开放式的不同，所需要获取的第二处理参数可能包含的是闭合角度或开放角度、闭合距离或开放距离等。在一种可能的实现方式中，在处理类型为内侧闭合式的情况下，第一处理参数可能会发生变化，在一个示例中，内侧闭合式下的进 刀点需要根据合页点和闭合角度所共同确定，在这种情况下，第一处理参数可以仅包含合页点，而进刀点则作为第二处理参数进行获取。
后续各公开实施例均以处理类型为内侧开放式为例进行说明，在其余处理类型下，S13中各步骤的实现方式可以参考后续各公开实施例进行灵活扩展，不再一一举例说明。
在目标对象为其他对象，图像处理方法应用于辅助其他类型的手术过程的情况下，步骤S13的实现形式也可以灵活发生变化，比如可以直接根据分割结果和目标关键点来得到所有的处理参数；或是先根据分割结果得到部分处理参数，再根据得到的部分处理参数，结合目标关键点获得其余的处理参数；或是先根据目标关键点得到部分处理参数，再根据得到的部分处理参数，结合分割结果得到其余的处理参数等，其实现过程可以基于目标对象的实际情况，参考下述各公开实施例来灵活扩展变化，不再一一赘述。
步骤S132的实现过程可以根据目标对象的实际情况灵活决定，不局限于下述公开实施例，在一种可能的实现方式中，步骤S132可以包括：根据目标关键点确定第一处理参数的所属范围；将第一处理参数的所属范围与分割结果结合，得到第一处理参数。
通过上述公开实施例可以看出，在一种可能的实现方式中，在获取第一处理参数的过程中，可以首先基于目标关键点确定第一处理参数的大致范围，再基于分割结果对目标对象位置的限制，将其与确定的大致范围结合，从而得到第一处理参数。
如上述各公开实施例所述，在目标对象为胫骨对象图像处理方法应用于辅助进行胫骨截骨的情况下，第一处理参数可以包含有合页点和/或进刀点，因此在一种可能的实现方式中，可以先根据目标关键点确定合页点与进刀点的所属范围，再根据分割结果结合确定的所属范围，得到合页点与进刀点的实际位置。图6示出根据本公开一实施例的获取处理参数的示意图，在一个示例中，在处理类型为内侧开放式的情况下，确定合页点的实现过程可以为：将获取的胫骨平台外侧端点下方10毫米(mm)这一范围作为合页点的所属范围，再在这一所属范围中，找到其与分割结果中胫骨外侧轮廓的交点，并基于这一交点向胫骨内侧移动10mm，得到第一处理参数中的合页点，如图6所示合页点64为左侧胫骨近端线段的左端点。在一个示例中，在处理类型为内侧开放式的情况下，确定进刀点的实现过程可以为：将获取的胫骨平台内侧端点下方15mm这一范围作为进刀点的所属范围，再在这一所属范围中，找到其与分割结果中胫骨内侧轮廓的交点，将这一交点作为第一处理参数中的进刀点，如图6所示进刀点67为左侧胫骨近端线段的右端点。在一个示例中，在确定了合页点与进刀点后，还可以连接合页点与进刀点，将连接的线段作为截骨线，即图6中左侧胫骨近端的连接线段。
在得到了包含合页点与进刀点的第一处理参数后，可以根据目标关键点和第一处理参数，通过步骤S133进一步确定第二处理参数。步骤S133的实现过程同样可以根据目标对象的实际情况灵活决定，不局限于下述公开实施例，在一种可能的实现方式中，步骤S133可以包括：将第一处理参数与至少三个目标关键点进行结合，得到目标力线；根据目标力线，结合第一处理参数和至少一个目标关键点，得到处理角度和/或处理距离。
如上述各公开实施例所述，目标力线可以是上述公开实施例中下肢力线的目标位置。在一些实施方式中，目标力线如何获取，可以根据胫骨截骨手术的矫正目的不同而灵活决定。举例来说，在一种可能的实现方式中，胫骨截骨手术的矫正目的可能为矫正后的下肢力线(即目标力线)通过膝关节中心点，在一种可能的实现方式中，胫骨手术的矫正目的也可能为矫正后的下肢力线通过藤泽点等。其中，藤泽点可以为胫骨内侧端点与胫骨外侧端点连线的线段中，从胫骨内侧端点起长度占连线线段62％的点。图7示出根据本公开一实施例的藤泽点示意图，从图7中可以看出，其中箭头指向的位置即藤泽点的位置。后续各公开实施例均以目标力线通过膝关节中心点为例进行说明，目标力线通过藤泽点的实现方式可以参考后续各公开实施例进行扩展，不再赘述。
如图6所示，在一个示例中，在目标力线为通过膝关节中心点的情况下，根据第一处理参数与至少三个目标关键点结合，得到目标力线的实现过程可以为：从股骨头中心点61向胫骨平台内侧端点62与胫骨平台外侧端点63的中点即膝关节中心点作射线，并以合页点64为圆心，合页点64与踝关节间隙内侧端点65与踝关节间隙外侧端点66的中心即踝关节中心点的距离为半径作弧线，则该弧线与上述射线远端的交点可以作为踝关节中心点的矫正目标，踝关节中心点的矫正目标与股骨头中心点61的连线线段构成目标力线。
在确定了目标力线后，则可以进一步得到开放角度和开放距离。在一个示例中，开放角度可以为从合页点向踝关节中心点的射线，以及从合页点向踝关节中心点的矫正目标的射线，这两个射线所形成的夹角，开放距离可以为以开放角度为顶角，腰长为截骨线长度的等腰三角形的底边长度， 图8示出根据本公开一实施例的第二处理参数示意图，如图8所示，在左侧截骨线以下的胫骨整体图像绕合页点81旋转上述开放角度的情况下，下肢力线可以经过膝关节中心，即达到了目标力线的位置。
在一种可能的实现方式中，在处理类型为内侧闭合式的情况下，开放角度可以变为闭合角度，从而可以基于在胫骨内侧轮廓上，与合页点形成顶角为闭合角度的等腰三角形的两点来确定进刀点。
通过上述过程，可以利用分割结果，并进一步借助获取的目标关键点自动得到胫骨高位截骨术的进刀点、合页点、开放(或闭合)角度以及开放(或闭合)距离等，从而实现自动化程度更高的图像处理过程，用以对胫骨截骨进行辅助，提升医工交互效率。
除了上述过程以外，本公开实施例提出的图像处理方法，在获取目标关键点和/或分割结果之前，还可以包括图像预处理的步骤，即在一种可能的实现方式中，目标图像可以包括经过预处理的图像，其中，预处理可以包括图像标准化和/或图像增强。
上述公开实施例已经提出，目标图像可以是包含目标对象的医学图像，比如全身X光片、下肢X光片或是胫骨部位的X光片等。对于不同设备拍摄出的医学图像来说，其可能存在一些差异，比如有些医学图像采用白色背景黑色前景，有些医学图像则采用白色前景黑色背景，不同设备拍摄的医学图像其像素间距可能存在差异等。因此，为了便于对这些医学图像统一进行目标关键点获取或是分割，在一种可能的实现方式中，可以对医学图像进行图像标准化，来得到经过预处理的目标图像。在一种可能的实现方式中，为了使得后续得到的图像处理结果具有较好的处理效果，还可以对医学图像进行图像增强。
图像标准化的实现方式可以根据实际情况灵活决定，不局限于下述各公开实施例。在一种可能的实现方式中，图像标准化可以包括背景标准化、像素间距标准化以及像素值归一化中的一个或多个。背景标准化的方式可以将图像的背景设定为同一颜色，设定为何种颜色不受限制，在一个示例中，可以将医学图像的背景均设置为黑色，前景均设置为白色等。像素间距标准化的方式可以为将医学图像中的像素间距均设置为规定值，规定值的取值可以根据实际情况灵活设定，在一个示例中，可以将像素间距均设置为(0.68mm,0.68mm)。像素值归一化可以将图像中像素值均归一化某一数值范围内，比如[0,1]之间等，其中归一化的方式不受限定，在一个示例中，可以先对医学图像中的像素点像素值从小到大进行排序，将其中位于3％位置的像素值设定为最低像素值，位于99％位置的像素值设定为最高像素值，然后将低于最低像素值的像素点的值更改为最低像素值，高于最高像素值的像素点的值更高为最高像素值，在完成像素值更改后再将像素值归一化到[0,1]之间，从而完成像素值归一化。
图像增强的方式也可以根据实际情况灵活决定，不局限于下述实施例，在一个示例中，可以通过限制对比度自适应直方图均衡化算法(CLAHE，Contrast Limited Adaptive Histogram Equalization)来增强医学图像的局部对比度，从而实现图像增强。
通过图像标准化和/或图像增强来得到经过预处理的图像作为目标图像，可以便于后续对具有统一规格且具有较好图像效果的目标图像进行目标关键点获取和分割，增加目标关键点获取和分割的实现便捷程度，也可以增加获取目标关键点的精度和分割精度，继而增加图像处理的实现便捷性和精度。
胫骨高位截骨术是一种治疗早期膝关节骨关节炎的有效手段，它可以保留原有的关节，通过矫正膝关节负重力线、阻止软骨的进一步磨损、增加关节稳定性、缓解疼痛、改善膝关节功能，从而避免或尽可能的推迟关节置换。其中，胫骨截骨位置的准确性将大大影响胫骨高位截骨的效果。
相关技术中的手术规划方法通常采用手工标注的方法来进行关键点定位，手术规划流程复杂；关键点自动定位方法难以兼具高稳定性和高精确性，且并不给出关键点的预测置信度，存在关键点因某些原因(如图像质量过差、关键点不存在等)无法被精确预测时算法给出过大偏差的错误预测的问题。同时该方法需要手动设置进刀点、合页点，才能进行后续的开放(闭合)角度、开放(闭合)距离的计算，医工交互效率不高。
本公开应用示例提出一种图像处理方法，一方面利用深度学习模型自动定位下肢全长X线片中的关键点，并且该模型能够同时利用粗定位热度图和细定位热度图进行定位预测，兼具高稳定性和高精确性，且能够给出关键点的预测置信度；另一方面利用深度学习模型自动预测下肢全长X线片中的胫骨分割，并进一步借助胫骨平台外侧端点和内侧端点推荐胫骨高位截骨术的进刀点、合页点、开放(闭合)角度、开放(闭合)距离。本公开实施例提出了一种图像处理方法，这一处理方法可以基于下肢X光片来确定胫骨截骨过程中的截骨位置，该图像处理的过程可以为：
第一步，下肢X光片中目标关键点的自动定位。图9示出目标关键点自动定位的流程，从图9 中可以看出，本公开应用示例可以实现下肢X光片中左侧和右侧的股骨头中心、大转子尖、股骨内侧髁、股骨外侧髁、胫骨平台内侧端点、胫骨平台外侧端点、踝关节间隙内侧端点、踝关节间隙外侧端点、膝关节中心、踝关节中心，共10*2＝20种目标关键点的自动定位(左右各10种目标关键点)。其中其中膝关节中心点可以定义为胫骨平台内侧端点和胫骨平台外侧端点的中点，踝关节中心点可以定义为踝关节间隙内侧端点和踝关节间隙外侧端点的中点，在一个示例中，可以先基于定义定位各中心点左右两侧的端点，再通过计算定位的两个端点的中点来得到上述中心点；在一个示例中，也可以直接定位上述中心点的位置。为实现后续胫骨截骨各处理参数的确定，所预测的目标关键点应至少包含股骨头中心点、踝关节中心点、膝关节中心点、胫骨平台内侧端点以及胫骨平台外侧端点等。
在一些实施方式中，目标关键点的定位过程可以分为以下几个步骤：
首先对输入图像91依次按以下步骤进行图像预处理92：将X光图像统一处理为背景为黑色，前景为白色；统一图像的像素间距为(0.68mm,0.68mm)；将像素数值归一化，先将低于第3百分位数和高于第99百分位数的值分别置为第3百分位数、第99百分位数，再将数值归一化到[0,1]之间；再利用CLAHE方法增强图像的局部对比度。
然后将预处理后的图像输入全卷积神经网络93(即上述公开实施例中的第一神经网络)，在本公开应用示例中，可以采用一个含编码器-解码器及跳跃链接结构的Unet网络，来针对每一种目标关键点分别生成粗定位热度图(即上述公开实施例中的第一热度图)和细定位热度图(即上述公开实施例中的第二热度图)。
在第一神经网络的训练阶段，可以根据每幅输入的训练图像中每个目标关键点位置的真实值(即标注值)，来计算该目标关键点对应的粗定位目标热度图及细定位目标热度图，再通过均方误差损失函数对第一神经网络生成的热度图94进行监督，使用反向传播及随机梯度下降法训练，使第一神经网络生成的热度图尽量接近前述目标热度图。粗定位目标热度图和细定位目标热度图可以均以上述公式(2)所示的二维高斯函数的形式进行表示。如上述公开实施例中的图5所示，第一热度图即粗定位目标热度图中σ的值较细定位热度图中更大，因此在更大的范围上有高响应值。实现方案中的目标热度图也可以由有类似性质(越靠近目标关键点的位置上有越大的响应值，粗定位热度图较细定位热度图在更大的范围上有高响应值)的函数来实现，不局限于本应用示例提出的形式。
在生成粗定位热度图和细定位热度图后，可以对其进行后处理95，来得到目标关键点的定位结果96，其中，对粗定位热度图和细定位热度图的后处理总体可以分为以下步骤：
首先进行粗定位坐标(即上述公开实施例中的第一关键点的坐标)和粗定位置信度(即上述公开实施例中的第一置信度)的计算，在本公开应用示例中，可以将粗定位热度图上最大值的坐标作为粗定位坐标；然后计算粗定位热度图上数值大于最大值的a i倍率的区域(即上述公开实施例中的包含目标关键点的区域)的周长c i和面积s i，在本公开应用示例中，可以选用4个a i取值，分别记为：a 0＝0.8；a 1＝0.6；a 2＝0.4；a 3＝0.2。则粗定位置信度可以通过上述公式(1)计算。
然后进行细定位坐标(即上述公开实施例中第二关键点的坐标)和细定位置信度(即上述公开实施例中的第二置信度)的计算，在本公开应用示例中，若粗定位置信度大于0.5，可以认为粗定位基本准确，则可以保留细定位热度图上粗定位坐标附近一定范围(即上述公开实施例中的响应区域)内的响应值，并将细定位热度图中超出响应区域范围的值设置为0，从而使得细定位坐标总在粗定位坐标附近；若粗定位置信度小于等于0.5，则保留原有的细定位热度图。然后通过与粗定位热度图中类似的方法，根据处理后或原有的细定位热度图，计算细定位坐标和细定位置信度。
在分别完成了粗定位热度图和细定位热度图的相关计算后，可以选定最终目标关键点的预测结果，实现过程可以为：在细定位置信度大于0.5或细定位置信度大于粗定位置信度的情况下，选用细定位坐标及细定位置信度作为最终输出的目标关键点定位坐标和置信度；否则选用粗定位坐标及粗定位置信度作为最终输出的目标关键点定位坐标和置信度。
在本公开应用示例中，粗、细定位坐标和细定位置信度的具体计算方法也可以采用其他的计算方式，例如可根据热度图上高响应区域的重心确定热度图中的定位坐标，或是可以根据预测热度图与定位坐标对应的目标热度图的相关系数来计算置信度等。
第二步，下肢X光片中胫骨的自动分割。图10示出自动分割的流程，从图10中可以看出，本公开应用示例中，胫骨自动分割的过程可以通过下述步骤来实现：
首先对输入图像101进行图像预处理102：本公开应用示例中，可以采用与前述目标关键点自动定位过程相同的图像预处理步骤，在此不再赘述。
然后将预处理后的图像输入预测胫骨分割的全卷积神经网络103(即上述公开实施例中的第二 神经网络)。本公开应用示例中，可以采用一种名为Tiramisu的全卷积神经网络来进行股骨分割，该Tiramisu网络与Unet网络类似，具有降采样路径、升采样路径和跳跃连接。同时该Tiramisu网络结构使用了DenseNet结构中的密集连接块替换了由级联卷积层组成的卷积块，从而实现更好的特征复用，并得到更鲁棒的特征。密集连接块中不但包含了级联卷积层，且每一个卷积层的输入将与其输出合并作为下一个卷积层的输入。
在第二神经网络的训练阶段，可以通过交叉熵损失函数对第二神经网络生成的胫骨分割结果134进行监督，使用反向传播及随机梯度下降法训练，使第二神经网络生成的分割结果尽量接近人工标注的胫骨分割标准。本公开应用示例中，第二神经网络可以实现双侧胫骨分割预测，在一种可能的实现方式中，也可以通过第二神经网络直接实现左侧胫骨分割预测和/或右侧胫骨分割预测。
在得到上述胫骨分割结果104后，可以对胫骨分割结果进行后处理105，对分割结果后处理的过程可以为：对第二神经网络生成的胫骨分割结果计算连通域，保留其中最大的两个连通域，将其中重心在左侧的连通域作为左侧胫骨分割结果106，将其中重心在右侧的连通域作为右侧胫骨分割结果107；取出这两个连通域的边界，分别作为左侧胫骨轮廓和右侧胫骨轮廓。
第三步，基于目标关键点位置和胫骨轮廓，确定胫骨截骨的各项处理参数。
本公开应用示例以获取内侧开放式的胫骨高位截骨的各项处理参数为例进行说明：
首先，可以利用目标关键点和胫骨轮廓以及进刀点、合页点间的几何位置关系获取进刀点与合页点。在一些实施方式中：在胫骨外侧轮廓上获取胫骨平台外侧端点下方10mm的点，该点再向胫骨内侧10mm处的点即可以作为合页点；在胫骨内侧轮廓上找到胫骨平台内侧端点下方15mm的点即可以作为进刀点，合页点与进刀点的连线段即截骨线。如上述公开实施例中的图6所示，左侧胫骨近端线段为截骨线，其左侧端点为合页点64，其右侧端点为进刀点67。
接着可以根据胫骨高位截骨的矫正目标来确定目标力线(下肢力线可以为股骨头中心与踝关节中心的连线段，目标力线可以为下肢力线的矫正目标)。例如在矫正目标为目标力线经过膝关节中心的情况下，可以从股骨头中心向膝关节中心作射线，以合页点为圆心、合页点与踝关节中心的距离为半径作弧线，则该弧线与射线远端的交点可以作为踝关节中心矫正目标，踝关节中心矫正目标与股骨头中心的连线段可以作为目标力线。
在确定了目标力线后，可以进一步确定开放角度和开放距离。其中，开放角度可以为从合页点向踝关节中心点的射线，与从合页点向踝关节中心矫正目标点的射线所形成的夹角。开放距离可以为开放角度为顶角，腰长等于截骨线长度的等腰三角形的底边长度。
在一些可能的实施例中，本公开应用示例中胫骨高位截骨术的处理类型可以从内侧开放式被替换为内侧闭合式、外侧开放式或外侧闭合式，则开放角度、进刀点、合页点的推荐方法可以相应发生变化，例如处理类型为内侧闭合式的情况下，可以用闭合角度代替开放角度，进刀点的位置可以通过，在胫骨内侧轮廓上与合页点形成顶角为闭合角度的等腰三角形的两点这种方式所确定。本公开应用示例中的矫正目标也可以灵活变化，在一个示例中，矫正目标可以变为目标力线经过上述公开实施例中提出的藤泽点，则目标力线及踝关节中心矫正目标的计算方法可以相应地发生变化。
通过上述公开应用示例，可以利用第一神经网络自动定位下肢X线片中的目标关键点，减少了手工标注目标关键点的过程，从而简化手术规划流程、提升医工交互效率；第一神经网络能够同时利用粗定位热度图和细定位热度图进行目标关键点的定位预测，兼具高稳定性和高精确性；同时能够给出各目标关键点预测的置信度，从而尽可能减少目标关键点因某些原因(如图像质量过差、目标关键点不存在等)无法被精确预测时得到过大偏差的错误预测结果的情况，同时便于后续使用缺失值补全方法对预测失败的目标关键点进行补全。
同时，本公开应用示例可以利用第二神经网络自动实现下肢X线片中的胫骨分割，并进一步借助胫骨平台外侧端点和内侧端点自动确定胫骨高位截骨术的进刀点、合页点、开放(闭合)角度、开放(闭合)距离。从而实现自动化高、精确度高的胫骨高位截骨中各项处理参数的预测，简化手术规划流程、提升医工交互效率。
本公开应用示例能够对下肢全长X线片提供高稳定性、高准确度、高一致性的关键点自动定位，解决了传统阅片流程中医生手动标注关键点费时费力的问题以及初级医师的标注一致性可能较差的问题；本方案能够对下肢全长X线片自动进行胫骨分割，并根据医师选定的胫骨高位截骨术种类及矫正目标，利用关键点及胫骨轮廓自动推荐进刀点、合页点、开放或闭合角度、开放或闭合距离，免去医师手工通过复杂的作图、几何运算、测量来进行手术规划的步骤，自动化程度高。总体上，本公开应用示例能够简化胫骨高位截骨术的手术规划流程，提升医工交互效率。
需要说明的是，本公开实施例的图像处理方法不限于应用在上述下肢X光片图像的处理中，也 不限于仅确定胫骨截骨的位置，可以应用于任意的图像处理，以及任意相关处理参数的确定过程中，本公开实施例对此不作限定。
可以理解，本公开实施例提及的上述各个方法实施例，在不违背原理逻辑的情况下，均可以彼此相互结合形成结合后的实施例，限于篇幅，本公开实施例不再赘述。本领域技术人员可以理解，在具体实施方式的上述方法中，各步骤的执行顺序应当以其功能和可能的内在逻辑确定。
此外，本公开实施例还提供了图像处理装置、电子设备、计算机可读存储介质、程序产品，上述均可用来实现本公开实施例提供的任一种图像处理方法，相应技术方案和描述和参见方法部分的相应记载，不再赘述。
图11示出根据本公开实施例的图像处理装置的框图。该图像处理装置可以为终端设备、服务器或者其他处理设备等。其中，终端设备可以为用户设备(User Equipment，UE)、移动设备、用户终端、终端、蜂窝电话、无绳电话、个人数字处理(Personal Digital Assistant，PDA)、手持设备、计算设备、车载设备、可穿戴设备等。在一些可能的实现方式中，该图像处理装置可以通过处理器调用存储器中存储的计算机可读指令的方式来实现。如图11所示，所述图像处理装置110可以包括：目标关键点获取模块111，配置为获取目标图像的目标关键点；分割模块112，配置为对目标图像中的目标对象进行分割，得到目标对象的分割结果；处理类型获取模块113，配置为获取目标对象的处理类型；处理参数确定模块114，配置为根据目标关键点、分割结果以及处理类型，确定目标对象的至少一个处理参数。
在一种可能的实现方式中，目标关键点获取模块111配置为：对目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。
在一种可能的实现方式中，目标关键点获取模块111配置为：对目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图；获取至少两个热度图对应的目标关键点的坐标以及置信度。
在一种可能的实现方式中，目标关键点获取模块111配置为：对目标图像进行关键点检测，得到至少两个目标关键点分别对应的第一热度图和第二热度图，其中，第一热度图对应的目标关键点的响应范围大于第二热度图对应的目标关键点的响应范围；从第一热度图中确定第一关键点，根据第一热度图，得到第一关键点的第一置信度；根据第一置信度，从第二热度图中确定第二关键点，结合第二热度图，得到第二关键点的第二置信度；根据第二置信度，将第一关键点或第二关键点确定为目标图像中的目标关键点，并得到与目标关键点对应的置信度。
在一种可能的实现方式中，目标关键点获取模块111进一步配置为：在第一置信度大于第一置信度阈值的情况下，根据第一关键点在第一热度图中的位置确定响应区域，从第二热度图的响应区域内中确定第二关键点，根据第二热度图的响应区域，得到第二关键点的第二置信度；在第一置信度不大于第一置信度阈值的情况下，从第二热度图中确定第二关键点，根据第二热度图，得到第二关键点的第二置信度。
在一种可能的实现方式中，目标关键点获取模块111进一步配置为：在第二置信度大于第二置信度阈值的情况下，将第二关键点确定为目标图像中的目标关键点，将第二置信度作为与目标关键点对应的置信度；在第二置信度不大于第二置信度阈值的情况下，将第一关键点确定为目标图像中的目标关键点，将第一置信度作为与目标关键点对应的置信度。
在一种可能的实现方式中，目标关键点获取模块111配置为：将目标图像输入至第一神经网络进行关键点检测。
在一种可能的实现方式中，第一神经网络通过包含目标关键点位置标注的训练图像进行训练，训练包括：根据目标关键点位置标注，生成与目标关键点位置对应的目标热度图；将训练图像输入至第一神经网络，得到第一神经网络输出的热度图；根据输出的热度图与目标热度图，确定第一神经网络的损失函数；根据损失函数，调整第一神经网络的至少一个参数。
在一种可能的实现方式中，处理类型获取模块113配置为：根据目标关键点，确定目标对象的处理类型。
在一种可能的实现方式中，目标关键点为至少两个，至少一个处理参数包括第一处理参数和第二处理参数，处理参数确定模块114配置为：根据处理类型，确定目标对象待获取的第一处理参数和第二处理参数；根据至少两个目标关键点以及分割结果，获取第一处理参数；根据至少三个目标关键点，结合第一处理参数，获取第二处理参数。
在一种可能的实现方式中，目标图像包括经过预处理的图像，预处理包括图像标准化和/或图像增强。
在一种可能的实现方式中，目标对象包括胫骨对象；处理类型包括：内侧闭合式、外侧闭合式、内侧开放式或外侧开放式；至少一个处理参数包括进刀点、合页点、目标力线、处理角度以及处理距离中的一个或多个。
本公开实施例还提出一种计算机可读存储介质，其上存储有计算机程序指令，所述计算机程序指令被处理器执行时实现上述方法。计算机可读存储介质可以是非易失性计算机可读存储介质。
本公开实施例还提出一种电子设备，包括：处理器；配置为存储处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器存储的指令，以执行上述方法。电子设备可以被提供为终端、服务器或其它形态的设备。
本公开实施例还提供了一种计算机程序产品，包括计算机可读代码，当计算机可读代码在设备上运行时，设备中的处理器执行用于实现如上任一实施例提供的图像处理方法的指令。
本公开实施例还提供了另一种计算机程序产品，配置为存储计算机可读指令，指令被执行时使得计算机执行上述任一实施例提供的图像处理方法的操作。
图12示出根据本公开实施例的一种电子设备800的框图。例如，电子设备800可以是移动电话、计算机、数字广播终端、消息收发设备、游戏控制台、平板设备、医疗设备、健身设备和个人数字助理等终端。参照图12，电子设备800可以包括以下一个或多个组件：处理组件802，存储器804，电源组件806，多媒体组件808，音频组件810，输入/输出(Input/Output，I/O)的接口812，传感器组件814，以及通信组件816。
处理组件802通常控制电子设备800的整体操作，诸如与显示、电话呼叫、数据通信、相机操作和记录操作相关联的操作。处理组件802可以包括一个或多个处理器820来执行指令，以完成上述的方法的全部或部分步骤。此外，处理组件802可以包括一个或多个模块，便于处理组件802和其他组件之间的交互。例如，处理组件802可以包括多媒体模块，以方便多媒体组件808和处理组件802之间的交互。
存储器804被配置为存储各种类型的数据以支持在电子设备800的操作。这些数据的示例包括配置为在电子设备800上操作的任何应用程序或方法的指令，联系人数据，电话簿数据，消息，图片，视频等。存储器804可以由任何类型的易失性或非易失性存储设备或者它们的组合实现，如静态随机存取存储器(SRAM，Static Random-Access Memory)，电可擦除可编程只读存储器(EEPROM，Electrically Erasable Programmable Read-Only Memory)，可擦除可编程只读存储器(EPROM，Erasable Programmable Read-Only Memory)，可编程只读存储器(PROM，Programmable Read-Only Memory)，只读存储器(ROM，Read Only Memory)，磁存储器，快闪存储器，磁盘或光盘。
电源组件806为电子设备800的各种组件提供电力。电源组件806可以包括电源管理系统，一个或多个电源，及其他与为电子设备800生成、管理和分配电力相关联的组件。
多媒体组件808包括在所述电子设备800和用户之间的提供一个输出接口的屏幕。在一些实施例中，屏幕可以包括液晶显示器(LCD，Liquid Crystal Display)和触摸面板(TP，TouchPanel)。在屏幕包括触摸面板的情况下，屏幕可以被实现为触摸屏，以接收来自用户的输入信号。触摸面板包括一个或多个触摸传感器以感测触摸、滑动和触摸面板上的手势。所述触摸传感器可以不仅感测触摸或滑动动作的边界，而且还检测与所述触摸或滑动操作相关的持续时间和压力。在一些实施例中，多媒体组件808包括一个前置摄像头和/或后置摄像头。在电子设备800处于操作模式，如拍摄模式或视频模式的情况下，前置摄像头和/或后置摄像头可以接收外部的多媒体数据。每个前置摄像头和后置摄像头可以是一个固定的光学透镜系统或具有焦距和光学变焦能力。
音频组件810被配置为输出和/或输入音频信号。例如，音频组件810包括一个麦克风(MIC，Microphone)，在电子设备800处于操作模式，如呼叫模式、记录模式和语音识别模式的情况下，麦克风被配置为接收外部音频信号。所接收的音频信号可以被进一步存储在存储器804或经由通信组件816发送。在一些实施例中，音频组件810还包括一个扬声器，用于输出音频信号。
I/O接口812为处理组件802和外围接口模块之间提供接口，上述外围接口模块可以是键盘、点击轮、按钮等。这些按钮可包括但不限于：主页按钮、音量按钮、启动按钮和锁定按钮。
传感器组件814包括一个或多个传感器，用于为电子设备800提供各个方面的状态评估。例如，传感器组件814可以检测到电子设备800的打开/关闭状态和组件的相对定位，例如所述组件为电子设备800的显示器和小键盘，传感器组件814还可以检测电子设备800或电子设备800一个组件的位置改变，用户与电子设备800接触的存在或不存在，电子设备800方位或加速/减速和电子设备800的温度变化。传感器组件814可以包括接近传感器，被配置用来在没有任何的物理接触时检测附近物体的存在。传感器组件814还可以包括光传感器，如互补金属氧化物半导体(CMOS，Complementary  Metal-Oxide-Semiconductor)或电荷耦合器件(CCD，Charge Coupled Device，)图像传感器，配置为在成像应用中使用。在一些实施例中，该传感器组件814还可以包括加速度传感器、陀螺仪传感器、磁传感器、压力传感器或温度传感器。
通信组件816被配置为便于电子设备800和其他设备之间有线或无线方式的通信。电子设备800可以接入基于通信标准的无线网络，如无线保真(Wi-Fi，Wireless Fidelity)、第二代移动通信技术(2G，The 2nd Generation，)或第三代移动通信技术(3G，The 3nd Generation，)或它们的组合。在一个示例性实施例中，通信组件816经由广播信道接收来自外部广播管理系统的广播信号或广播相关信息。在一个示例性实施例中，所述通信组件816还包括近场通信(NFC，Near Field Communication)模块，以促进短程通信。例如，在NFC模块可基于射频识别(RFID，Radio Frequency Identification)技术，红外数据协会(IrDA，Infrared Data Association)技术，超宽带(UWB，Ultra Wide Band)技术，蓝牙(BT，Blue Tooth)技术和其他技术来实现。
在示例性实施例中，电子设备800可以被一个或多个应用专用集成电路(ASIC，Application Specific Integrated Circuit)、数字信号处理器(DSP，Digital Signal Processor)、数字信号处理设备(DSPD，Digital Signal Processing Device)、可编程逻辑器件(PLD，Programmable Logic Device)、现场可编程门阵列(FPGA，Field Programmable Gate Array)、控制器、微控制器、微处理器或其他电子元件实现，用于执行上述方法。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器804，上述计算机程序指令可由电子设备800的处理器820执行以完成上述方法。
图13示出根据本公开实施例的一种电子设备1900的框图。例如，电子设备1900可以被提供为一服务器。参照图13，电子设备1900包括处理组件1922，可以包括一个或多个处理器，以及由存储器1932所代表的存储器资源，用于存储可由处理组件1922的执行的指令，例如应用程序。存储器1932中存储的应用程序可以包括一个或一个以上的每一个对应于一组指令的模块。此外，处理组件1922被配置为执行指令，以执行上述方法。
电子设备1900还可以包括一个电源组件1926被配置为执行电子设备1900的电源管理，一个有线或无线网络接口1950被配置为将电子设备1900连接到网络，和一个I/O接口1958。电子设备1900可以操作基于存储在存储器1932的操作系统，例如Windows Server TM、Mac OS X TM、UnixTM、Linux TM、FreeBSD TM或类似系统。
在示例性实施例中，还提供了一种非易失性计算机可读存储介质，例如包括计算机程序指令的存储器1932，上述计算机程序指令可由电子设备1900的处理组件1922执行以完成上述方法。
本公开实施例可以是系统、方法和/或计算机程序产品。计算机程序产品可以包括计算机可读存储介质，其上载有用于使处理器实现本公开实施例的各个方面的计算机可读程序指令。
计算机可读存储介质可以是可以保持和存储由指令执行设备使用的指令的有形设备。计算机可读存储介质例如可以是但不限于电存储设备、磁存储设备、光存储设备、电磁存储设备、半导体存储设备或者上述的任意合适的组合。计算机可读存储介质可以包括：便携式计算机盘、硬盘、随机存取存储器(RAM，Random Access Memory)、只读存储器、可擦式可编程只读存储器(EPROM或闪存)、静态随机存取存储器(SRAM)、便携式压缩盘只读存储器(CD-ROM，Compact Disc Read-Only Memory)、数字多功能盘(DVD，Digital Video Disc)、记忆棒、软盘、机械编码设备、例如其上存储有指令的打孔卡或凹槽内凸起结构、以及上述的任意合适的组合。这里所使用的计算机可读存储介质不被解释为瞬时信号本身，诸如无线电波或者其他自由传播的电磁波、通过波导或其他传输媒介传播的电磁波(例如，通过光纤电缆的光脉冲)、或者通过电线传输的电信号。
这里所描述的计算机可读程序指令可以从计算机可读存储介质下载到各个计算/处理设备，或者通过网络、例如因特网、局域网、广域网和/或无线网下载到外部计算机或外部存储设备。网络可以包括铜传输电缆、光纤传输、无线传输、路由器、防火墙、交换机、网关计算机和/或边缘服务器。每个计算/处理设备中的网络适配卡或者网络接口从网络接收计算机可读程序指令，并转发该计算机可读程序指令，以供存储在各个计算/处理设备中的计算机可读存储介质中。
用于执行本公开操作的计算机程序指令可以是汇编指令、指令集架构(ISA，Industry Standard Architecture)指令、机器指令、机器相关指令、伪代码、固件指令、状态设置数据、或者以一种或多种编程语言的任意组合编写的源代码或目标代码，所述编程语言包括面向对象的编程语言—诸如Smalltalk、C++等，以及常规的过程式编程语言例如C语言或类似的编程语言。计算机可读程序指令可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程 计算机的情形中，远程计算机可以通过任意种类的网络—包括局域网(LAN，Local Area Network)或广域网(WAN，Wide Area Network)连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。在一些实施例中，通过利用计算机可读程序指令的状态信息来个性化定制电子电路，例如可编程逻辑电路、现场可编程门阵列或可编程逻辑阵列，该电子电路可以执行计算机可读程序指令，从而实现本公开实施例的各个方面。
这里参照根据本公开实施例的方法、装置(系统)和计算机程序产品的流程图和/或框图描述了本公开实施例的各个方面。应当理解，流程图和/或框图的每个方框以及流程图和/或框图中各方框的组合，都可以由计算机可读程序指令实现。
这些计算机可读程序指令可以提供给通用计算机、专用计算机或其它可编程数据处理装置的处理器，从而生产出一种机器，使得这些指令在通过计算机或其它可编程数据处理装置的处理器执行时，产生了实现流程图和/或框图中的一个或多个方框中规定的功能/动作的装置。也可以把这些计算机可读程序指令存储在计算机可读存储介质中，这些指令使得计算机、可编程数据处理装置和/或其他设备以特定方式工作，从而，存储有指令的计算机可读介质则包括一个制造品，其包括实现流程图和/或框图中的一个或多个方框中规定的功能/动作的各个方面的指令。
也可以把计算机可读程序指令加载到计算机、其它可编程数据处理装置、或其它设备上，使得在计算机、其它可编程数据处理装置或其它设备上执行一系列操作步骤，以产生计算机实现的过程，从而使得在计算机、其它可编程数据处理装置、或其它设备上执行的指令实现流程图和/或框图中的一个或多个方框中规定的功能/动作。
附图中的流程图和框图显示了根据本公开实施例的多个实施例的系统、方法和计算机程序产品的可能实现的体系架构、功能和操作。在这点上，流程图或框图中的每个方框可以代表一个模块、程序段或指令的一部分，所述模块、程序段或指令的一部分包含一个或多个用于实现规定的逻辑功能的可执行指令。在有些作为替换的实现中，方框中所标注的功能也可以以不同于附图中所标注的顺序发生。例如，两个连续的方框实际上可以基本并行地执行，它们有时也可以按相反的顺序执行，这依所涉及的功能而定。也要注意的是，框图和/或流程图中的每个方框、以及框图和/或流程图中的方框的组合，可以用执行规定的功能或动作的专用的基于硬件的系统来实现，或者可以用专用硬件与计算机指令的组合来实现。
该计算机程序产品可以通过硬件、软件或其结合的方式实现。在一个可选实施例中，所述计算机程序产品可以体现为计算机存储介质，在另一个可选实施例中，计算机程序产品具体体现为软件产品，例如软件开发包(SDK，Software Development Kit)等等。
以上已经描述了本公开的各实施例，上述说明是示例性的，并非穷尽性的，并且也不限于所披露的各实施例。在不偏离所说明的各实施例的范围和精神的情况下，对于本技术领域的普通技术人员来说许多修改和变更都是显而易见的。本文中所用术语的选择，旨在最好地解释各实施例的原理、实际应用或对市场中的技术的改进，或者使本技术领域的其它普通技术人员能理解本文披露的各实施例。
工业实用性
本公开实施例涉及一种图像处理方法及装置、电子设备、存储介质和程序产品。所述方法包括：获取目标图像的目标关键点；对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；获取所述目标对象的处理类型；根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。通过上述过程，可以提升图像处理的精度和准确性。

Claims (27)

  1. 一种图像处理方法，包括：获取目标图像的目标关键点；对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；获取所述目标对象的处理类型；根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。
  2. 根据权利要求1所述的方法，其中，所述获取目标图像的目标关键点，包括：
    对所述目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。
  3. 根据权利要求1所述的方法，其中，所述获取目标图像的目标关键点，包括：
    对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图；
    获取至少两个所述热度图对应的所述目标关键点的坐标以及置信度。
  4. 根据权利要求1所述的方法，其中，所述获取目标图像的目标关键点，包括：
    对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的第一热度图和第二热度图，其中，所述第一热度图对应的所述目标关键点的响应范围大于所述第二热度图对应的所述目标关键点的响应范围；从所述第一热度图中确定第一关键点，根据所述第一热度图，得到所述第一关键点的第一置信度；根据所述第一置信度，从所述第二热度图中确定第二关键点，结合所述第二热度图，得到所述第二关键点的第二置信度；根据所述第二置信度，将所述第一关键点或所述第二关键点确定为所述目标图像中的目标关键点，并得到与所述目标关键点对应的置信度。
  5. 根据权利要求4所述的方法，其中，所述根据所述第一置信度，从所述第二热度图中确定第二关键点，结合所述第二热度图，得到所述第二关键点的第二置信度，包括：
    在所述第一置信度大于第一置信度阈值的情况下，根据所述第一关键点在所述第一热度图中的位置确定响应区域，从所述第二热度图的所述响应区域内中确定第二关键点，根据所述第二热度图的所述响应区域，得到所述第二关键点的第二置信度；
    在所述第一置信度不大于所述第一置信度阈值的情况下，从所述第二热度图中确定第二关键点，根据所述第二热度图，得到所述第二关键点的第二置信度。
  6. 根据权利要求4或5所述的方法，其中，所述根据所述第二置信度，将所述第一关键点或所述第二关键点确定为所述目标图像中的目标关键点，并得到与所述目标关键点对应的置信度，包括：
    在所述第二置信度大于第二置信度阈值的情况下，将所述第二关键点确定为所述目标图像中的目标关键点，将所述第二置信度作为与所述目标关键点对应的置信度；
    在所述第二置信度不大于所述第二置信度阈值的情况下，将所述第一关键点确定为所述目标图像中的目标关键点，将所述第一置信度作为与所述目标关键点对应的置信度。
  7. 根据权利要求2至6中任意一项所述的方法，其中，所述对所述目标图像进行关键点检测，包括：将所述目标图像输入至第一神经网络进行关键点检测。
  8. 根据权利要求7所述的方法，其中，所述第一神经网络通过包含目标关键点位置标注的训练图像进行训练，所述训练包括：根据所述目标关键点位置标注，生成与所述目标关键点位置对应的目标热度图；将所述训练图像输入至所述第一神经网络，得到所述第一神经网络输出的热度图；根据所述输出的热度图与所述目标热度图，确定所述第一神经网络的损失函数；根据所述损失函数，调整所述第一神经网络的至少一个参数。
  9. 根据权利要求1至8中任意一项所述的方法，其中，所述获取所述目标对象的处理类型，包括：根据所述目标关键点，确定所述目标对象的处理类型。
  10. 根据权利要求1至9中任意一项所述的方法，其中，所述目标关键点为至少两个，所述至少一个处理参数包括第一处理参数和第二处理参数，所述根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数，包括：
    根据所述处理类型，确定所述目标对象待获取的第一处理参数和第二处理参数；
    根据至少两个所述目标关键点以及所述分割结果，获取所述第一处理参数；
    根据至少三个所述目标关键点，结合所述第一处理参数，获取所述第二处理参数。
  11. 根据权利要求1至10中任意一项所述的方法，其中，所述目标图像包括经过预处理的图像，所述预处理包括图像标准化和/或图像增强。
  12. 根据权利要求1至11中任意一项所述的方法，其中，所述目标对象包括胫骨对象；所述处理类型包括：内侧闭合式、外侧闭合式、内侧开放式或外侧开放式；所述至少一个处理参数包括进刀点、合页点、目标力线、处理角度以及处理距离中的一个或多个。
  13. 一种图像处理装置，包括：目标关键点获取模块，配置为获取目标图像的目标关键点；分割模块，配置为对所述目标图像中的目标对象进行分割，得到所述目标对象的分割结果；处理类型获取模块，配置为获取所述目标对象的处理类型；处理参数确定模块，配置为根据所述目标关键点、所述分割结果以及所述处理类型，确定所述目标对象的至少一个处理参数。
  14. 根据权利要求13所述的装置，其中，所述目标关键点获取模块，配置为对所述目标图像进行关键点检测，得到至少两个包含置信度的目标关键点。
  15. 根据权利要求13所述的装置，其中，所述目标关键点获取模块，配置为对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的热度图；获取至少两个所述热度图对应的所述目标关键点的坐标以及置信度。
  16. 根据权利要求13所述的装置，其中，所述目标关键点获取模块，配置为对所述目标图像进行关键点检测，得到至少两个目标关键点分别对应的第一热度图和第二热度图，其中，所述第一热度图对应的所述目标关键点的响应范围大于所述第二热度图对应的所述目标关键点的响应范围；从所述第一热度图中确定第一关键点；根据所述第一热度图，得到所述第一关键点的第一置信度；根据所述第一置信度，从所述第二热度图中确定第二关键点；结合所述第二热度图，得到所述第二关键点的第二置信度；根据所述第二置信度，将所述第一关键点或所述第二关键点确定为所述目标图像中的目标关键点，并得到与所述目标关键点对应的置信度。
  17. 根据权利要求16所述的装置，其中，所述目标关键点获取模块，还配置为在所述第一置信度大于第一置信度阈值的情况下，根据所述第一关键点在所述第一热度图中的位置确定响应区域，从所述第二热度图的所述响应区域内中确定第二关键点，根据所述第二热度图的所述响应区域，得到所述第二关键点的第二置信度；在所述第一置信度不大于所述第一置信度阈值的情况下，从所述第二热度图中确定第二关键点，根据所述第二热度图，得到所述第二关键点的第二置信度。
  18. 根据权利要求16或17所述的装置，其中，所述目标关键点获取模块，还配置为在所述第二置信度大于第二置信度阈值的情况下，将所述第二关键点确定为所述目标图像中的目标关键点，将所述第二置信度作为与所述目标关键点对应的置信度；在所述第二置信度不大于所述第二置信度阈值的情况下，将所述第一关键点确定为所述目标图像中的目标关键点，将所述第一置信度作为与所述目标关键点对应的置信度。
  19. 根据权利要求14至18任意一项所述的装置，其中，所述目标关键点获取模块，还配置为将所述目标图像输入至第一神经网络进行关键点检测。
  20. 根据权利要求19所述的装置，其中，所述第一神经网络通过包含目标关键点位置标注的训练图像进行训练，所述装置还包括：网络训练模块，配置为根据所述目标关键点位置标注，生成与所述目标关键点位置对应的目标热度图；将所述训练图像输入至所述第一神经网络，得到所述第一神经网络输出的热度图；根据所述输出的热度图与所述目标热度图，确定所述第一神经网络的损失函数；根据所述损失函数，调整所述第一神经网络的至少一个参数。
  21. 根据权利要求13至20任意一项所述的装置，其中，所述处理类型获取模块，配置为根据所述目标关键点，确定所述目标对象的处理类型。
  22. 根据权利要求13至21中任意一项所述的装置，其中，所述目标关键点为至少两个，所述至少一个处理参数包括第一处理参数和第二处理参数，所述处理参数确定模块，配置为根据所述处理类型，确定所述目标对象待获取的第一处理参数和第二处理参数；根据至少两个所述目标关键点以及所述分割结果，获取所述第一处理参数；根据至少三个所述目标关键点，结合所述第一处理参数，获取所述第二处理参数。
  23. 根据权利要求13至22中任意一项所述的装置，其中，所述目标图像包括经过预处理的图像，所述预处理包括图像标准化和/或图像增强。
  24. 根据权利要求13至23中任意一项所述的装置，其中，所述目标对象包括胫骨对象；所述处理类型包括：内侧闭合式、外侧闭合式、内侧开放式或外侧开放式；所述至少一个处理参数包括进刀点、合页点、目标力线、处理角度以及处理距离中的一个或多个。
  25. 一种电子设备，包括：处理器；配置为存储所述处理器可执行指令的存储器；其中，所述处理器被配置为调用所述存储器存储的指令，以执行权利要求1至12中任意一项所述的方法。
  26. 一种计算机可读存储介质，所述存储介质上存储有计算机程序指令，其中，所述计算机程序指令被处理器执行时实现权利要求1至12中任意一项所述的方法。
  27. 一种计算机程序产品，包括计算机可读代码，在所述计算机可读代码在电子设备中运行的情况下，所述电子设备中的处理器执行如权利要求1至12中任意一项所述的方法。
PCT/CN2020/138438 2020-07-07 2020-12-22 图像处理方法及装置、电子设备、存储介质和程序产品 WO2022007342A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
KR1020217042439A KR20220013404A (ko) 2020-07-07 2020-12-22 이미지 처리 방법 및 장치, 전자 기기, 저장 매체 및 프로그램 제품
JP2021576877A JP2022542780A (ja) 2020-07-07 2020-12-22 画像処理方法及び装置、電子機器、記憶媒体並びにプログラム製品

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010646714.5 2020-07-07
CN202010646714.5A CN111768399A (zh) 2020-07-07 2020-07-07 图像处理方法及装置、电子设备和存储介质

Publications (1)

Publication Number Publication Date
WO2022007342A1 true WO2022007342A1 (zh) 2022-01-13

Family

ID=72724799

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/138438 WO2022007342A1 (zh) 2020-07-07 2020-12-22 图像处理方法及装置、电子设备、存储介质和程序产品

Country Status (5)

Country Link
JP (1) JP2022542780A (zh)
KR (1) KR20220013404A (zh)
CN (1) CN111768399A (zh)
TW (1) TW202203153A (zh)
WO (1) WO2022007342A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102395664B1 (ko) * 2022-01-26 2022-05-09 주식회사 몰팩바이오 Hsv 색 공간과 clahe를 이용한 병리 슬라이드 이미지 색감 표준화 방법 및 장치
CN115187821A (zh) * 2022-07-05 2022-10-14 阿波罗智能技术(北京)有限公司 验证模型转换前后正确性的方法、相关装置及程序产品
CN115225710A (zh) * 2022-06-17 2022-10-21 中国电信股份有限公司 数据包的传输方法、装置、电子设备及存储介质

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111768399A (zh) * 2020-07-07 2020-10-13 上海商汤智能科技有限公司 图像处理方法及装置、电子设备和存储介质
CN112348892A (zh) * 2020-10-29 2021-02-09 上海商汤智能科技有限公司 点定位方法及相关装置、设备
CN115131301B (zh) * 2022-06-15 2023-04-18 北京长木谷医疗科技有限公司 基于深度学习的智能识别骨关节炎的方法及系统
CN115018834B (zh) * 2022-08-08 2022-10-25 山东光岳九州半导体科技有限公司 一种半导体晶片图像对准方法
CN117058149B (zh) * 2023-10-12 2024-01-02 中南大学 一种用于训练识别骨关节炎的医学影像测量模型的方法

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108205655A (zh) * 2017-11-07 2018-06-26 北京市商汤科技开发有限公司 一种关键点预测方法、装置、电子设备及存储介质
US20180330506A1 (en) * 2017-05-09 2018-11-15 Heartflow, Inc. Systems and methods for anatomic structure segmentation in image analysis
CN110533639A (zh) * 2019-08-02 2019-12-03 杭州依图医疗技术有限公司 一种关键点定位方法及装置
CN111768399A (zh) * 2020-07-07 2020-10-13 上海商汤智能科技有限公司 图像处理方法及装置、电子设备和存储介质
CN111768400A (zh) * 2020-07-07 2020-10-13 上海商汤智能科技有限公司 图像处理方法及装置、电子设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20180330506A1 (en) * 2017-05-09 2018-11-15 Heartflow, Inc. Systems and methods for anatomic structure segmentation in image analysis
CN108205655A (zh) * 2017-11-07 2018-06-26 北京市商汤科技开发有限公司 一种关键点预测方法、装置、电子设备及存储介质
CN110533639A (zh) * 2019-08-02 2019-12-03 杭州依图医疗技术有限公司 一种关键点定位方法及装置
CN111768399A (zh) * 2020-07-07 2020-10-13 上海商汤智能科技有限公司 图像处理方法及装置、电子设备和存储介质
CN111768400A (zh) * 2020-07-07 2020-10-13 上海商汤智能科技有限公司 图像处理方法及装置、电子设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
ZHANG YUAN, GU WEN-HUA: "Artificial intelligence in knee surgery:status and prospect", JOURNAL OF TRAUMATIC SURGERY, vol. 22, no. 2, 1 January 2020 (2020-01-01), pages 81 - 86, XP055885322, ISSN: 1009-4237, DOI: 10.3969j.issn.1009-4237.2020.02.001 *

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102395664B1 (ko) * 2022-01-26 2022-05-09 주식회사 몰팩바이오 Hsv 색 공간과 clahe를 이용한 병리 슬라이드 이미지 색감 표준화 방법 및 장치
CN115225710A (zh) * 2022-06-17 2022-10-21 中国电信股份有限公司 数据包的传输方法、装置、电子设备及存储介质
CN115225710B (zh) * 2022-06-17 2024-06-07 中国电信股份有限公司 数据包的传输方法、装置、电子设备及存储介质
CN115187821A (zh) * 2022-07-05 2022-10-14 阿波罗智能技术(北京)有限公司 验证模型转换前后正确性的方法、相关装置及程序产品
CN115187821B (zh) * 2022-07-05 2024-03-22 阿波罗智能技术(北京)有限公司 验证模型转换前后正确性的方法、相关装置及程序产品

Also Published As

Publication number Publication date
CN111768399A (zh) 2020-10-13
TW202203153A (zh) 2022-01-16
JP2022542780A (ja) 2022-10-07
KR20220013404A (ko) 2022-02-04

Similar Documents

Publication Publication Date Title
WO2022007342A1 (zh) 图像处理方法及装置、电子设备、存储介质和程序产品
WO2021051965A1 (zh) 图像处理方法及装置、电子设备、存储介质和计算机程序
WO2021147257A1 (zh) 网络训练、图像处理方法及装置、电子设备和存储介质
US12002212B2 (en) Image segmentation method and apparatus, computer device, and storage medium
WO2022151755A1 (zh) 目标检测方法及装置、电子设备、存储介质、计算机程序产品和计算机程序
WO2021135145A1 (zh) 图像分割方法及装置、电子设备和存储介质
WO2021057174A1 (zh) 图像处理方法及装置、电子设备、存储介质和计算机程序
CN111768400A (zh) 图像处理方法及装置、电子设备和存储介质
US20220058821A1 (en) Medical image processing method, apparatus, and device, medium, and endoscope
WO2021259391A2 (zh) 图像处理方法及装置、电子设备和存储介质
CN113222038B (zh) 基于核磁图像的乳腺病灶分类和定位方法及装置
WO2021082517A1 (zh) 神经网络训练及图像分割方法、装置、设备、介质和程序
TWI767614B (zh) 圖像處理方法及電子設備、電腦可讀儲存介質
US20210118551A1 (en) Device to enhance and present medical image using corrective mechanism
WO2022088729A1 (zh) 点定位方法及相关装置、设备、介质及计算机程序
US20210407637A1 (en) Method to display lesion readings result
US20220301718A1 (en) System, Device, and Method of Determining Anisomelia or Leg Length Discrepancy (LLD) of a Subject by Using Image Analysis and Machine Learning
WO2022125930A1 (en) Self-supervised machine learning for medical image analysis
JP2022548453A (ja) 画像分割方法及び装置、電子デバイス並びに記憶媒体
Sanjid et al. From pixels to pathology: A novel dual-pathway multi-scale hierarchical upsampling network for mri-based prostate zonal segmentation
Kim et al. Enhanced deep learning model enables accurate alignment measurement across diverse institutional imaging protocols
Feng et al. Research and application of tongue and face diagnosis based on deep learning
CN113553460B (zh) 影像检索方法及装置、电子设备和存储介质
CN111178297A (zh) 一种图像处理方法、装置、电子设备及介质
Hong et al. Automated cephalometric landmark detection using deep reinforcement learning

Legal Events

Date Code Title Description
ENP Entry into the national phase

Ref document number: 2021576877

Country of ref document: JP

Kind code of ref document: A

Ref document number: 20217042439

Country of ref document: KR

Kind code of ref document: A

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20943919

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20943919

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 10.07.2023)

122 Ep: pct application non-entry in european phase

Ref document number: 20943919

Country of ref document: EP

Kind code of ref document: A1