WO2022205928A1 - 骨赘识别方法、装置、电子设备及存储介质 - Google Patents

骨赘识别方法、装置、电子设备及存储介质 Download PDF

Info

Publication number
WO2022205928A1
WO2022205928A1 PCT/CN2021/130471 CN2021130471W WO2022205928A1 WO 2022205928 A1 WO2022205928 A1 WO 2022205928A1 CN 2021130471 W CN2021130471 W CN 2021130471W WO 2022205928 A1 WO2022205928 A1 WO 2022205928A1
Authority
WO
WIPO (PCT)
Prior art keywords
area
region
osteophyte
segmentation
feature
Prior art date
Application number
PCT/CN2021/130471
Other languages
English (en)
French (fr)
Inventor
张逸凌
刘星宇
Original Assignee
北京长木谷医疗科技有限公司
张逸凌
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京长木谷医疗科技有限公司, 张逸凌 filed Critical 北京长木谷医疗科技有限公司
Publication of WO2022205928A1 publication Critical patent/WO2022205928A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
    • G06F18/2415—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G06F18/243—Classification techniques relating to the number of classes
    • G06F18/2431—Multiple classes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G06V10/267—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion by performing operations on regions, e.g. growing, shrinking or watersheds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/03—Recognition of patterns in medical or anatomical images
    • G06V2201/033—Recognition of patterns in medical or anatomical images of skeletal patterns

Definitions

  • the present invention relates to the technical field of image processing, in particular to a method, device, electronic device and storage medium for identifying osteophytes.
  • osteophytes In preoperative planning, osteophytes have a great impact on the positioning of key landmarks such as mechanical axis, joint line, anterior and posterior femoral axis, and AP axis. Wrongly judging the shape and position of osteophytes will lead to deviations in the positioning landmarks, thereby affecting the knee joint. function, stability, range of motion and easy postoperative pain.
  • doctors Before clinical total knee arthroplasty, doctors need to plan and make decisions based on bone images of the lower extremity without osteophytes. Doctors are better at medical knowledge, and image processing is time-consuming and labor-intensive for doctors.
  • embodiments of the present invention provide an osteophyte identification method, apparatus, electronic device, and storage medium, so as to intelligently identify osteophytes in medical images.
  • an embodiment of the present invention provides an osteophyte identification method, including: acquiring a medical image; inputting the medical image into a trained first segmentation model to obtain the femoral region in the medical image, at least one of the tibia area, the fibula area, and the patella area; input at least one of the femur area, the tibia area, the fibula area or the patella area into the trained second segmentation model to obtain the corresponding femoral bone Osteophytes, tibial osteophytes, fibular osteophytes, or patellar osteophytes.
  • obtaining at least one of the femur region, tibia region, fibula region, and patella region in the medical image includes: inputting the medical image into a neural network model to obtain the femoral region, tibia region, and patella region. Coarse segmentation prediction results of the region, fibula region and patella region;
  • the rough segmentation prediction result is optimized to obtain at least one of the optimized femur area, tibia area, fibula area, and patella area.
  • the method for identifying osteophytes by the trained second segmentation model includes: performing rough segmentation of the osteophyte feature map on the original image of any one of the femur area, tibia area, fibula area or patella area through the backbone network, obtaining a coarse segmentation feature map; based on the original region image and the coarse segmentation feature map, using the global correction unit included in the second segmentation model to perform global correction on the coarse segmentation feature map to obtain a global correction map; Extracting the osteophyte features included in the original image of the region and the global correction map to obtain an osteophyte feature map; using the local correction unit included in the second segmentation model to modify the osteophyte feature map, Obtain osteophyte segmentation results.
  • optimizing the coarse segmentation prediction result includes: selecting feature points whose confidence levels meet preset requirements from the coarse segmentation prediction results; classifying the features of the feature points, and classifying them according to the classification results. Update the coarse segmentation prediction result of the feature point.
  • classifying the features of the feature points includes: performing feature extraction on the feature points to obtain target features; predicting the target features based on a classifier to obtain whether the feature points belong to the femoral region, the tibia, or not. area, fibula area, or patella area.
  • the global correction unit includes three layers of global correction modules with the same structure, wherein each global correction module includes a feature fusion module and a residual network module connected in sequence;
  • the feature map of the preset size is used as the feature fusion map in the second layer;
  • the feature map of the second preset size output by the global correction module of the second layer is used as the feature fusion map of the third layer;
  • the output of the global correction module of the third layer as the global correction map.
  • the first segmentation model is a unet neural network model including the Pointrend algorithm.
  • the second segmentation model is a cascadePSP neural network model.
  • the backbone network in the cascadePSP neural network model is the unet network.
  • the method before inputting the medical image into the trained first segmentation model, the method further includes: acquiring a first image data set, the first image data set includes at least one medical image containing osteophytes, the The medical image has at least the label of the femur area, the tibia area, the fibula area, or the patella area; using the first image data set to train the first segmentation model to obtain the trained first segmentation model; Before inputting at least one of the femoral region, the tibial region, the fibula region, or the patellar region into the trained second segmentation model, further comprising: acquiring a second image dataset, the second image dataset Contains a plurality of femur images, tibial images, fibula images, and patella images, the femoral images, tibial images, fibula images, and patella images with at least corresponding femoral osteophytes, tibial osteophytes, fibular osteophytes, or patella The label of the osteophyte; the
  • an embodiment of the present invention provides an apparatus for identifying osteophytes, including: an acquisition module configured to acquire a medical image; a first processing module configured to input the medical image into a trained first In the segmentation model, at least one of the femur area, tibia area, fibula area, and patella area in the medical image is obtained; the second processing module is configured to divide the femur area, tibia area, fibula area, or At least one of the patella regions is input into the trained second segmentation model, resulting in the corresponding femoral osteophyte, tibial osteophyte, fibular osteophyte, or patellar osteophyte.
  • an embodiment of the present invention provides an electronic device, including a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor passes The computer instructions are executed, thereby executing the osteophyte identification method described in the first aspect or any one of the implementation manners of the first aspect.
  • an embodiment of the present invention provides a computer-readable storage medium, where the computer-readable storage medium stores computer instructions, and the computer instructions are used to cause the computer to execute the first aspect or any one of the first aspects.
  • FIG. 1 is a schematic flowchart of an osteophyte identification method according to Embodiment 1 of the present invention
  • FIG. 2 is a schematic diagram of a processing flow of a medical image in a first segmentation model
  • FIG. 3 is a schematic diagram of the network structure of the first segmentation model and the second segmentation model in Embodiment 1 of the present invention
  • FIG. 4 is a schematic structural diagram of an apparatus for identifying osteophytes according to Embodiment 2 of the present invention.
  • Embodiment 1 of the present invention provides an osteophyte identification method.
  • FIG. 1 is a schematic flowchart of a method for identifying osteophytes according to Embodiment 1 of the present invention. As shown in Figure 1, the method for identifying osteophytes in Embodiment 1 of the present invention includes the following steps:
  • the medical image is a medical image of a lower extremity bone, such as a lower extremity bone image.
  • S102 Input the medical image into the trained first segmentation model to obtain at least one of the femur area, the tibia area, the fibula area, and the patella area in the medical image.
  • the first segmentation model may include a unet neural network model including the Pointrend algorithm, so that image segmentation with high quality and high pixels can be achieved.
  • Figure 2 is a schematic diagram of the processing flow of the medical image in the first segmentation model. As shown in Figure 2, the medical image can be first input into the unet backbone network to obtain the coarse segmentation prediction result, and then the coarse segmentation prediction result can be processed through Pointrend Get the fine segmentation prediction result.
  • the medical image is input into the trained first segmentation model, and at least one of the femur area, tibia area, fibula area, and patella area in the medical image is obtained
  • the first step includes: inputting the medical image into a neural network model to obtain rough segmentation prediction results of the femur area, tibia area, fibula area, and patella area; optimizing the rough segmentation prediction results to obtain an optimized Femoral area, tibia area, fibula area, and patella area.
  • the Pointrend algorithm may be used to optimize the rough segmentation prediction result to obtain the optimized femur area, tibia area, fibula area, and patella area.
  • optimizing the coarse segmentation prediction result includes: selecting feature points whose confidence levels meet preset requirements in the coarse segmentation prediction result; The feature is classified, and the coarse segmentation prediction result of the feature point is updated according to the classification result.
  • the Pointrend algorithm can be used to optimize the coarse segmentation prediction result, and the following technical solution can be adopted: selecting feature points whose confidence levels meet preset requirements in the coarse segmentation prediction result; The feature of the feature point is acquired, the feature of the feature point is classified, and the coarse segmentation prediction result of the feature point is updated according to the classification result.
  • the feature points whose confidence levels meet the preset requirements may be feature points whose confidence levels are about 0.5.
  • step S102 may adopt the following method:
  • the segmentation neural network model pointrend+unet is established, as shown in Figure 2: First, the unet network is used as the backbone network to perform rough segmentation. The first stage uses 4 downsampling to learn the deep features of the image, and then performs 4 upsampling to extract the features.
  • each downsampling layer includes 2 convolution layers and 1 pooling layer, the convolution kernel size of the convolution layer is 3*3, and the size of the convolution kernel in the pooling layer is 2*2, the number of convolution kernels in each convolutional layer is 128, 256, 512; each upsampling includes 1 upsampling layer and 2 convolutional layers, of which the convolution kernel of the convolutional layer The size is 3*2, the size of the convolution kernel in the upsampling layer is 2*2, and the number of convolution kernels in each upsampling layer is 512, 256, 128.
  • the first step is to select a series of potential feature points to prepare for the next judgment.
  • the selection is based on the points with a classification confidence close to 0.5 in the coarse segmentation prediction result (in the classification task).
  • the confidence of a point will be close to 0 or 1, and the confidence of a point near 0.5 means that the network is very uncertain about the classification of this point), usually such points are points close to the edge of the object.
  • classifying the features of the feature points includes: performing feature extraction on the feature points to obtain target features; and predicting the target features based on a classifier to obtain the target features. Whether the feature point belongs to the femoral area, tibial area, fibula area, or patella area
  • feature extraction is performed on the feature points selected in the first step, and the unet neural network model comes with a feature extractor, so only the selected feature points need to be placed in the feature extractor.
  • the features of the corresponding positions can be selected.
  • the features of these feature points are calculated by bilinear interpolation; the third step: use a small classifier such as MLP to determine which category the point belongs to, which is actually equivalent to Use a 1*1 convolution to predict, but not for points with confidence close to 1 or 0. In this way, we can classify all uncertain pixels one by one, thereby improving the accuracy of segmentation.
  • the high-pixel mask can be quickly calculated by using the unet neural network model including the Pointrend algorithm.
  • the method before the medical image is input into the trained first segmentation model, the method further includes:
  • the first image data set includes at least one medical image containing osteophytes, and the medical image at least has labels of the femur area, the tibia area, the fibula area, and the patella area;
  • the first segmentation model is trained by using the first image data set to obtain the trained first segmentation model.
  • using the image data set to train the first segmentation model may adopt the following technical solutions:
  • a dataset of CT medical images of osteophytes in patients with knee disease was obtained and manually annotated with regions of the femur, tibia, fibula, and patella as our database. According to the ratio of 6:2:2, it is divided into training set, validation set, and test set; the DICOM (Digital Imaging and Communications in Medicine) data of the two-dimensional cross-section (image) of CT medical images is converted into For pictures in JPG format, convert the manually labeled label file into a picture in png format, and save it as the input of the neural network.
  • DICOM Digital Imaging and Communications in Medicine
  • Establish a segmentation neural network model pointrend+unet firstly use the unet neural network as the backbone network, the first stage uses 4 downsampling to learn the deep features of the image, and then performs 4 upsampling to re-store the feature map into the image, where each Each downsampling layer includes 2 convolution layers and 1 pooling layer, the convolution kernel size of the convolution layer is 3*3, the size of the convolution kernel in the pooling layer is 2*2, and each convolution layer has a size of 3*3.
  • the number of convolution kernels in is 128, 256, 512; each upsampling includes 1 upsampling layer and 2 convolutional layers, where the convolution kernel size of the convolutional layer is 3*2, and the upsampling layer The size of the convolution kernels in is 2*2, and the number of convolution kernels in each upsampling layer is 512, 256, 128. There is a dropout layer after the last upsampling, and the dropout rate is set to 0.7. All convolutional layers are followed by activation functions as relu functions.
  • pointrend to accurately segment the results, select a set of points with a confidence level of 0.5, and extract the features of the selected points.
  • the features of these feature points can be calculated by bilinear interpolation Bilinear, and then use a small classifier to determine which category the point belongs to.
  • the background pixel value of the data label is set to 0, the femur/tibia/fibula/patella are set to 1, the training batch_size (batch size, that is, the number of samples for one training) is set to 6, and the learning rate is set to 1e-4 , the optimizer uses the Adam optimizer, the loss function used is DICE loss, the original image of the training set and the femur/tibia/fibula/patella are sent to the network for training, and the training batch is adjusted according to the change of the loss function during the training process. size, and finally get the rough segmentation result of each part.
  • bilinear interpolation is used to upsample the prediction results of the previous step, and then the N most uncertain points are selected in this denser feature map, such as points with a probability close to 0.5, and then the N points are calculated. features of the points, and predict their classification using the classifier MLP, this process is repeated until the required size is upsampled.
  • a simple multi-layer perceptron is used for pointwise prediction, since the MLP predicts the segmentation label of each point, it can be trained using the loss in the Unet coarse segmentation task.
  • S103 Input at least one of the femur area, tibia area, fibula area, and patella area into the trained second segmentation model to obtain the corresponding femoral osteophyte, tibial osteophyte, fibular osteophyte, or patella bone superfluous.
  • pointrend+unet can only divide a rough area, but the segmentation of small objects and edges is very rough, so it is necessary to further accurately identify osteophytes.
  • the method further includes: acquiring a second image data set, the second image data set includes a plurality of femur images, Tibia image, fibula image, patella image, the femur image, tibia image, fibula image, and patella image have at least corresponding labels of femoral osteophyte, tibial osteophyte, fibular osteophyte, and patellar osteophyte;
  • the second segmentation model is trained on the two image data sets to obtain the trained second segmentation model.
  • the training of the second segmentation model can be carried out by the following method: obtain a CT medical image dataset with osteophytes, manually label the femoral osteophytes and tibial osteophytes, and use them as our database. According to the ratio of 6:2:2, it is divided into training set and test set; save the femoral area and tibia area output by the above pointrend network as a JPG image as an image, convert the label file into a png format image as a label, and save it as a neural input to the network.
  • the second segmentation model is a cascadePSP neural network model.
  • the cascadePSP neural network model can produce high-quality and very high-resolution segmentations, which work well for edge segmentation of small objects.
  • the backbone network in the cascadePSP neural network model is the unet network.
  • the method for identifying osteophytes by the trained second segmentation model includes: using the backbone network to create an original image of any one of the femur area, tibia area, fibula area, or patella area Perform coarse segmentation of the osteophyte feature map to obtain a coarse segmentation feature map; based on the original image of the region and the coarse segmentation feature map, use the global correction unit included in the second segmentation model to globally perform the coarse segmentation feature map.
  • the original image of any one of the femur area, tibia area, fibula area or patella area may be roughly segmented by the backbone network to obtain the rough segmentation of the osteophyte area features.
  • Feature map based on the coarse segmentation feature map, determine the 1/4 size of the coarse segmentation feature map and the 1/8 size of the coarse segmentation feature map; based on bilinear interpolation upsampling, make the 1/4 size of the coarse segmentation feature map, 1
  • the size of the coarse segmentation feature map of size /8 becomes the first coarse segmentation feature map and the second coarse segmentation map of the original size of the region;
  • the corrected feature map is input to the global correction unit of the second segmentation model for global correction, the corrected feature map with the same size as the original region image, the feature map after 1/4 of the size of the original region image, and 1/ 8.
  • the feature map containing the osteophyte is input into the local correction unit of the second segmentation model, and the osteophyte segmentation result is output.
  • FIG. 3 is a schematic diagram of the network structure of the first segmentation model and the second segmentation model in Embodiment 1 of the present invention.
  • a segmentation neural network model cascadepsp is first established, and the unet network is used as the backbone network. image, label) for rough segmentation to obtain a rough segmentation result mask.
  • the global modification module includes three modification modules, and the operation of one modification module includes the following steps: accepting the original image, mask, 1/4 of the mask, and 1/8 of the mask, and converting 1/4 and 1/8 of the
  • the mask is upsampled by bilinear interpolation to become a mask of the same size as the original image, and finally four vectors are obtained: the original image, mask, mask4, and mask8.
  • the resnet network ie, RM in Figure 3
  • the global correction unit includes three layers of global correction modules with the same structure, wherein each global correction module includes a feature fusion module and a residual network module connected in sequence;
  • the feature map of the first preset size output by the correction module is used as the feature fusion map in the second layer;
  • the feature map of the second preset size output by the global correction module of the second layer is used as the feature fusion map in the third layer;
  • the output of the layer global correction module is used as the global correction map.
  • the above operation is called a correction module, and it is copied three times to become a global correction module.
  • Each correction module will produce three outputs, the first correction module only uses the output of 1/8 of the mask, and then upsamples it by a factor of 2 and itself as the input of the next correction module, and so on, until the third The final output mask of the layer, a quarter of the size of the mask, is used as the input of the local correction module.
  • the local correction unit includes two layers of local correction modules with the same structure, wherein each local correction module includes a sequentially connected feature fusion module and a residual network module; the first layer of local correction modules The third preset size feature map output by the correction module is used as the feature fusion map in the second layer; the output of the second layer local correction module is used as the osteophyte segmentation result
  • the local correction module (ie, the local step in FIG. 2 ) is similar to the global correction module, and is also obtained by stacking two correction modules. However, the local correction module does not take a whole original image as input, but cuts the large image into small images for input separately (extracts the part containing osteophytes from the large image), and obtains eight parts of the original image size after processing by the resnet network.
  • the residual network extracts a preset stride feature image from the input, and transfers the extracted feature image to spatial pyramid pooling, where the loss function is based on the stride size Different, the type of loss function is different.
  • Different loss functions are suitable for different strides, because coarse refinement focuses on global structure and ignores local details, while precise refinement achieves pixel-level accuracy by relying on local cues.
  • a medical image is obtained, and the medical image is input into the trained first segmentation model to obtain the femur region and tibia region in the medical image, and the femur region and tibia region in the medical image are obtained.
  • the region and tibial region are input into the trained second segmentation model to obtain femoral osteophytes and tibial osteophytes. That is to say, the use of the first segmentation model and the second segmentation model can intelligently identify osteophytes, saving a lot of money for orthopedic surgeons. Time, but also guidance and help for junior doctors.
  • the osteophyte identification method of Embodiment 1 of the present invention can be reused only after completing model training, quickly and accurately identify osteophytes in lower extremity bone images, and help doctors in surgical planning. It is easy to operate, has high accuracy, and meets the individual differences of patients. At the same time, by improving the basis of preoperative planning, guiding surgical planning and prosthesis selection, improving the accuracy of subsequent surgery, increasing the service life of postoperative prostheses, reducing postoperative complications, and improving patients’ postoperative life Quality matters.
  • Embodiment 2 of the present invention provides an apparatus for identifying osteophytes.
  • FIG. 4 is a schematic structural diagram of an apparatus for identifying osteophytes according to Embodiment 2 of the present invention. As shown in FIG. 4 , the apparatus for identifying osteophytes according to Embodiment 2 of the present invention includes an acquisition module 20 , a first processing module 22 and a second processing module 24 .
  • the acquisition module 20 is configured to acquire medical images.
  • the first processing module 22 is configured to input the medical image into the trained first segmentation model to obtain at least one of the femur area, the tibia area, the fibula area, and the patella area in the medical image
  • the second processing module 24 is used to input at least one of the femur region, tibia region, fibula region or the patella region into the second segmentation model after training, obtains corresponding femoral osteophyte, tibial osteophyte , fibular osteophyte, or patellar osteophyte.
  • the embodiment of the present invention also provides an electronic device, the electronic device may include a processor and a memory, wherein the processor and the memory may be connected by a bus or in other ways.
  • the processor may be a central processing unit (Central Processing Unit, CPU).
  • the processor may also be other general-purpose processors, digital signal processors (Digital Signal Processors, DSPs), application specific integrated circuits (Application Specific Integrated Circuits, ASICs), Field-Programmable Gate Arrays (Field-Programmable Gate Arrays, FPGAs) or other Chips such as programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above types of chips.
  • the memory can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions/ modules (eg, acquisition module 20, first processing module 22, and second processing module 24 shown in FIG. 4).
  • the processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, the osteophyte identification method in the above method embodiments is implemented.
  • the memory may include a storage program area and a storage data area, wherein the storage program area may store an operating system and an application program required by at least one function; the storage data area may store data created by the processor, and the like. Additionally, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid state storage device. In some embodiments, the memory may optionally include memory located remotely from the processor, such remote memory being connectable to the processor via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
  • the one or more modules are stored in the memory, and when executed by the processor, execute the osteophyte identification method in the embodiments shown in FIG. 1 to FIG. 3 .
  • the storage medium may be a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a flash memory (Flash Memory), a hard disk (Hard) Disk Drive, abbreviation: HDD) or solid-state drive (Solid-State Drive, SSD), etc.; the storage medium may also include a combination of the above-mentioned types of memories.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • General Engineering & Computer Science (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biophysics (AREA)
  • Health & Medical Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Probability & Statistics with Applications (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Multimedia (AREA)
  • Prostheses (AREA)
  • Apparatus For Radiation Diagnosis (AREA)
  • Image Analysis (AREA)

Abstract

一种骨赘识别方法、装置、电子设备及存储介质，其中骨赘识别方法包括：获取医学图像（S101）；将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域和/或胫骨区域和/或腓骨区域和/或髌骨区域（S102）；将所述股骨区域和/或胫骨区域和/或腓骨区域和/或髌骨区域输入到经过训练的第二分割模型中，得到股骨骨赘和/或胫骨骨赘和/或腓骨骨赘和/或髌骨骨赘（S103）。由此可以利用第一分割模型和第二分割模型可以迅速准确智能识别骨赘，帮助医生进行手术规划，易于操作、准确度高、满足患者个体差异的优点，同时通过完善术前规划的依据资料，指导手术规划和假体选择，提高后续手术的准确率，为骨科医生节约大量的时间。

Description

骨赘识别方法、装置、电子设备及存储介质
相关申请的交叉引用
本申请要求于2021年3月29日提交中国专利局，申请号为202110335659.2，发明名称为“骨赘识别方法、装置、电子设备及存储介质”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本发明涉及图像处理技术领域，具体涉及一种骨赘识别方法、装置、电子设备及存储介质。
背景技术
在术前规划中，骨赘对于机械轴、关节线、股骨前后轴、AP轴等关键标志的定位具有较大影响，错误的判断骨赘形貌、位置会导致定位标志偏差，从而影响膝关节功能、稳定性、运动范围且易造成术后疼痛。在临床全膝关节置换术前，医生需要依据不含骨赘的下肢骨影像资料进行手术的规划与决策，医生更为擅长的是医学知识，图像处理对于医生而言可谓耗时耗力。经验丰富的医生能够提出需求进行指导，协助其他技术人员完美消除下肢骨影像的骨赘，但是他们却苦于自己动手进行图像处理消除下肢骨影像的骨赘部分，而经验欠缺的医生尚且不能凭借医学经验进行指导，完美消除下肢骨影像的骨赘部分，自己操作则更为困难。
相关技术中，对于图像中骨赘的识别仍然受限于人工识别的方式，识别精度低、识别效率低。
发明内容
有鉴于此，本发明实施例提供了一种骨赘识别方法、装置、电子设备及存储介质，以智能识别医学图像中的骨赘。
根据第一方面，本发明实施例提供了一种骨赘识别方法，包括：获取医学图像；将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一；将所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘。
可选地，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一，包括：将所述医学图像输入到神经网络模型中，得到所述股骨区域、胫骨区域、腓骨区域以及髌骨区域的粗分割预测结果；
对所述粗分割预测结果进行优化，得到优化后的股骨区域、胫骨区域、腓骨区域、以及髌骨 区域中的至少之一。
可选地，经过训练的第二分割模型识别骨赘的方法包括：通过主干网络对所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的任一个区域原图进行骨赘特征图粗分割，得到粗分割特征图；基于所述区域原图、所述粗分割特征图，利用所述第二分割模型包括的全局修正单元，对所述粗分割特征图进行全局修正，得到全局修正图；对所述区域原图、以及所述全局修正图所包含的骨赘特征进行提取，得到骨赘特征图；利用所述第二分割模型包括的局部修正单元，对所述骨赘特征图进行修正，得到骨赘分割结果。
可选地，对所述粗分割预测结果进行优化，包括：在所述粗分割预测结果中选取出置信度符合预设要求的特征点；对所述特征点的特征进行分类，并根据分类结果更新所述特征点的粗分割预测结果。
可选地，对所述特征点的特征进行分类包括：对所述特征点进行特征提取，得到目标特征；基于分类器对所述目标特征进行预测，得到所述特征点是否属于股骨区域、胫骨区域、腓骨区域、或髌骨区域。
可选地，所述全局修正单元包括3层结构相同的全局修正模块，其中，每个全局修正模块包括顺次连接的特征融合模块和残差网络模块；第一层全局修正模块输出的第一预设大小的特征图作为第二层中特征融合用图；第二层全局修正模块输出的第二预设大小的特征图作为第三层中特征融合用图；第三层全局修正模块的输出作为所述全局修正图。
可选地，第一分割模型为包含Pointrend算法的unet神经网络模型。
可选地，所述第二分割模型为cascadePSP神经网络模型。
可选地，所述cascadePSP神经网络模型中的主干网络为unet网络。
可选地，在将所述医学图像输入到经过训练的第一分割模型之前，还包括：获取第一图像数据集，所述第一图像数据集中至少包含一个含有骨赘的医学图像，所述医学图像中至少带有股骨区域、胫骨区域、腓骨区域、或髌骨区域的标签；利用所述第一图像数据集对所述第一分割模型进行训练，得到所述经过训练的第一分割模型；在将所述股骨区域、胫骨区域、腓骨区域、或髌骨区域中的至少之一输入到经过训练的第二分割模型中之前，还包括：获取第二图像数据集，所述第二图像数据集中包含多个股骨图像、胫骨图像、腓骨图像、以及髌骨图像，所述股骨图像、胫骨图像、腓骨图像、以及髌骨图像中至少带有对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘的标签；利用所述第二图像数据集对所述第二分割模型进行训练，得到所述经过训练的第二 分割模型。
根据第二方面，本发明实施例提供了一种骨赘识别装置，包括：获取模块，被配置为获取医学图像；第一处理模块，被配置为将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一；第二处理模块，被配置为将所述股骨区域、胫骨区域、腓骨区域、或髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘或髌骨骨赘。
根据第三方面，本发明实施例提供了一种电子设备，包括存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行第一方面或者第一方面的任意一种实施方式中所述的骨赘识别方法。
根据第四方面，本发明实施例提供一种计算机可读存储介质，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行第一方面或者第一方面的任意一种实施方式中所述的骨赘识别方法。
附图说明
通过参考附图会更加清楚的理解本发明的特征和优点，附图是示意性的而不应理解为对本发明进行任何限制，在附图中：
图1为本发明实施例1骨赘识别方法的流程示意图；
图2为医学图像在第一分割模型的处理流程示意图；
图3为本发明实施例1第一分割模型和第二分割模型的网络结构示意图；
图4为本发明实施例2骨赘识别装置的结构示意图。
具体实施方式
为使本发明实施例的目的、技术方案和优点更加清楚，下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。
实施例1
本发明实施例1提供了一种骨赘识别方法。图1为本发明实施例1骨赘识别方法的流程示意图。如图1所示，本发明实施例1的骨赘识别方法包括以下步骤：
S101：获取医学图像。
作为本实施一种可选的实现方式，所述医学图像为下肢骨的医学图像，例如下肢骨影像。
S102：将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一。
在本发明实施例1中，所述第一分割模型可以包括包含Pointrend算法的unet神经网络模型，从而可以实现高质量高像素的图像分割。图2为医学图像在第一分割模型的处理流程示意图，如图2所示，可以首先将医学图像输入到unet主干网络中，得到粗分割预测结果，然后可通过Pointrend对粗分割预测结果进行处理得到精细分割预测结果。
作为本实施一种可选的实现方式，将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一包括：将所述医学图像输入到神经网络模型中，得到所述股骨区域、胫骨区域、腓骨区域、以及髌骨区域的粗分割预测结果；对所述粗分割预测结果进行优化，得到优化后的股骨区域、胫骨区域、腓骨区域、以及髌骨区域。
在本可选的实现方式中，可以利用Pointrend算法对所述粗分割预测结果进行优化，得到优化后的股骨区域、胫骨区域、腓骨区域、以及髌骨区域。
作为本实施例一种可选的实现方式，对所述粗分割预测结果进行优化，包括：在所述粗分割预测结果中选取出置信度符合预设要求的特征点；对所述特征点的特征进行分类，并根据分类结果更新所述特征点的粗分割预测结果。
在本可选的实现方式中，可以利用所述Pointrend算法对所述粗分割预测结果进行优化可以采用如下技术方案：在所述粗分割预测结果中选取出置信度符合预设要求的特征点；获取所述特征点的特征，所述特征点的特征进行分类，并根据分类结果更新所述特征点的粗分割预测结果。在本发明实施例1中，置信度符合预设要求的特征点可以为置信度为0.5左右的特征点。
示例的，步骤S102可以采用如下方法：
建立分割神经网络模型pointrend+unet，见图2：首先利用unet网络作为主干网络，对其进行粗分割，第一阶段使用4次下采样学习图像的深层特征，然后进行4次上采样以将特征图重新存储到图像中，其中每个下采样层中包括2个卷积层和1个池化层，卷积层的卷积核大小为3*3，池化层中的卷积核大小为2*2，每个卷积层中的卷积核的个数为128，256，512；每个上采样中包括1个上采样层和2个卷积层，其中卷积层的卷积核大小为3*2，上采样层中的卷积核大小为 2*2，每个上采样层中的卷积核个数为512，256，128。最后一次上采样结束后设有一个的dropout层，droupout率设置为0.7。所有的卷积层后面都设有激活函数为relu函数。最终获得股骨、胫骨、腓骨、髌骨的粗分割预测结果，它们的结果均为0-1之间的预测概率值(图像中构成mask的是一堆像素点，其中每个点都对应一个概率值)。
然后使用pointrend精确分割结果，第一步：挑选出一系列潜在的特征点来为接下来的判断做准备，在此挑选的依据是粗分割预测结果中分类置信度接近0.5的点(在分类任务中一个点的置信度会趋近于0或者1，置信度在0.5附近则代表网络对这个点的分类很不确定)，通常这类点都是接近物体边缘的点。
作为本实施例一种可选的实现方式，对所述特征点的特征进行分类包括：对所述特征点进行特征提取，得到目标特征；基于分类器对所述目标特征进行预测，得到所述特征点是否属于股骨区域、胫骨区域、腓骨区域、或髌骨区域
在本可选的实现方式中，对第一步挑选出的特征点进行特征提取，而unet神经网络模型自带特征提取器(feature extractor)，因此只需要将所选特征点在特征提取器中相应位置的特征选取出来即可，具体的，这些特征点的特征通过双线性插值Bilinear计算；第三步：使用一个小型的分类器例如MLP去判断这个点属于哪个类别，这其实等价于用一个1*1的卷积来预测，但是对于置信度接近于1或者0的点并不计算。这样我们就可以对所有不确定的像素点逐个进行分类，从而提高分割的精准度。
传统方法中要实现高像素的实例分割，需要对像素进行逐一计算，必然会带来大算力的问题，因此就需要权衡算力和高像素Mask之间的关系。而本发明实施例利用包含Pointrend算法的unet神经网络模型可以快速的计算出高像素mask。
作为本实施一种可选的实现方式，在将所述医学图像输入到经过训练的第一分割模型之前，还包括：
获取第一图像数据集，所述第一图像数据集中至少包含一个含有骨赘的医学图像，所述医学图像中至少带有股骨区域、胫骨区域、腓骨区域、髌骨区域的标签；
利用所述第一图像数据集对所述第一分割模型进行训练，得到所述经过训练的第一分割模型。
示例的，利用所述图像数据集对所述第一分割模型进行训练可以采用如下技术方案：
获取患有膝关节疾病患者的骨赘的CT医学图像数据集，将其进行手动标注股骨、胫骨、腓骨、髌骨区域，将其作为我们的数据库。按照6:2:2的比例划分为训练集、验证集、测试集；将 CT医学图像的二维横断面(image)的DICOM(Digital Imaging and Communications in Medicine，医学数字成像和通信)数据转换成JPG格式的图片，将人工标注的label文件转换成png格式的图片，保存后作为神经网络的输入。
建立分割神经网络模型pointrend+unet：首先利用unet神经网络作为主干网络，第一阶段使用4次下采样学习图像的深层特征，然后进行4次上采样以将特征图重新存储到图像中，其中每个下采样层中包括2个卷积层和1个池化层，卷积层的卷积核大小为3*3，池化层中的卷积核大小为2*2，每个卷积层中的卷积核的个数为128，256，512；每个上采样中包括1个上采样层和2个卷积层，其中卷积层的卷积核大小为3*2，上采样层中的卷积核大小为2*2，每个上采样层中的卷积核个数为512，256，128。最后一次上采样结束后设有一个的dropout层，droupout率设置为0.7。所有的卷积层后面都设有激活函数为relu函数。
然后使用pointrend精确分割结果，选择置信度为0.5的一组点，提取被选择出来的点的特征，具体的，这些特征点的特征可以通过双线性插值Bilinear计算，然后使用一个小型的分类器去判断这个点属于哪个类别。
模型训练过程中，数据标签的背景像素值设置为0，股骨/胫骨/腓骨/髌骨分别为1，训练的batch_size(批量大小，即一次训练的样本数)为6，学习率设置为1e-4，优化器使用Adam优化器，使用的损失函数为DICE loss，将训练集原图和股骨/胫骨/腓骨/髌骨分别送入网络进行训练，根据训练过程中损失函数的变化，调整训练批次的大小，最终得到各个部分的粗分割结果。进入pointrend模块后，先会使用双线性插值上采样前一步分割预测结果，然后在这个更密集的特征图中选择N个最不确定的点，比如概率接近0.5的点，然后计算这N个点的特征，并且利用分类器MLP预测它们的分类，这个过程一直被重复，直到上采样到需要的大小。对于每个选定点的逐点特征表示，使用简单的多层感知器进行逐点预测，因为MLP预测的是各点的分割label，所以可以使用Unet粗分割任务中的loss来训练。
S103：将所述股骨区域、胫骨区域、腓骨区域、髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘。
在本实施例中，pointrend+unet只能分出大概的区域，但是对细小的物体和边缘分割是非常粗糙的，因此需要对骨赘进行进一步精准地识别。
在将所述股骨区域、胫骨区域、腓骨区域、髌骨区域输入到经过训练的第二分割模型中之前，还包括：获取第二图像数据集，所述第二图像数据集中包含多个股骨图像、胫骨图像、腓骨图像、 髌骨图像，所述股骨图像、胫骨图像、腓骨图像、髌骨图像中至少带有对应的股骨骨赘、胫骨骨赘、腓骨骨赘、髌骨骨赘的标签；利用所述第二图像数据集对所述第二分割模型进行训练，得到所述经过训练的第二分割模型。
第二分割模型的训练可以采用如下方法：获取带有骨赘的CT医学图像数据集，将其进行手动标注股骨骨赘、胫骨骨赘部分，将其作为我们的数据库。按照6：2：2的比例划分为训练集、测试集；将上面pointrend网络输出的股骨区域和胫骨区域保存为JPG的图片作为image，标注文件转换成png格式的图片作为label，保存后作为神经网络的输入。
作为本实施一种可选的实现方式，所述第二分割模型为cascadePSP神经网络模型。
在本可选的实现方式中，cascadePSP神经网络模型可以生产高质量和非常高分辨率的分割，对小物体的边缘分割效果良好。
作为本实施例一种可选的实现方式，所述cascadePSP神经网络模型中的主干网络为unet网络。
作为本实施例一种可选的实现方式，经过训练的第二分割模型识别骨赘的方法包括：通过主干网络对所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的任一个区域原图进行骨赘特征图粗分割，得到粗分割特征图；基于所述区域原图、所述粗分割特征图，利用所述第二分割模型包括的全局修正单元，对所述粗分割特征图进行全局修正，得到全局修正图；对所述区域原图、以及所述全局修正图所包含的骨赘特征进行提取，得到骨赘特征图；利用所述第二分割模型包括的局部修正单元，对所述骨赘特征图进行修正，得到骨赘分割结果。
在本可选的实现方式中，参考图3，可以首先通过主干网络对所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的任一个区域原图进行骨赘区域特征粗分割，得到粗分割特征图；而后基于粗分割特征图确定1/4大小的粗分割特征图、1/8大小的粗分割特征图；基于双线性插值上采样，使1/4大小的粗分割特征图、1/8大小的粗分割特征图的大小变为区域原图大小的第一粗分割特征图、以及第二粗分割图；之后将区域原图、粗分割特征图、第一粗分割图、第二粗分割特征图输入至第二分割模型的全局修正单元中进行全局修正后，输出与区域原图大小相同的修正后特征图、1/4区域原图大小的特修正后的征图、1/8区域原图大小的特修正后的征图；最后可分别从区域原图、修正后1/4大小的粗分割特征图、修正后的1/8大小的粗分割特征图中提取包含骨赘的特征图；将包含骨赘的特征图输入至第二分割模型的局部修正单元中，输出骨赘分割结果。
继续参考图3，为本发明实施例1第一分割模型和第二分割模型的网络结构示意图，如图3 所示，首先建立分割神经网络模型cascadepsp，首先利用unet网络作为主干网络，对图片(image、label)进行粗分割处理，得到粗分割结果mask。具体的，全局修改模块包括三个修正模块，其中一个修正模块的操作包括以下步骤：接受image原图、mask、mask的1/4、mask的1/8，将1/4、1/8的mask进行双线性插值上采样变成与原图同样大小的mask，最终得到四个向量：原图、mask、mask4、mask8，把他们进行concat操作之后输入resnet网络(即图3中的RM)中，以resnet-50为骨干，利用分割神经网络模型cascadepsp从输入中提取stride＝8的feature map(特征图像)，传入[1,2,3,6]空间金字塔池化，捕获全局上下文。输出三个不同大小的经过修正的mask，即原图大小(图3中的image)，原图大小的1/4(图3中的S4)，原图大小的1/8(图3中的S1)，在图3中W表示图片的长，H表示图片的宽。
作为本实施例一种可选的实现方式，全局修正单元包括3层结构相同的全局修正模块，其中，每个全局修正模块包括顺次连接的特征融合模块和残差网络模块；第一层全局修正模块输出的第一预设大小的特征图作为第二层中特征融合用图；第二层全局修正模块输出的第二预设大小的特征图作为第三层中特征融合用图；第三层全局修正模块的输出作为所述全局修正图。
在本可选的实现方式中，上述操作称之为一个修正模块，将其复制三份，变成一个全局修正模块。每一个修正模块会产生三个输出，第一个修正模块只利用1/8的mask的输出，然后将它上采样2倍和它本身作为下一次修正模块的输入，以此类推，直到第三层的最终输出mask、mask的四分之一大小作为局部修正模块的输入。
作为本实施例一种可选的实现方式，局部修正单元包括2层结构相同的局部修正模块，其中，每个局部修正模块包括顺次连接的特征融合模块和残差网络模块；第一层局部修正模块输出的第三预设大小的特征图作为第二层中特征融合用图；第二层局部修正模块的输出作为骨赘分割结果
在本可选的实现方式中，局部修正模块(即图2中的局部步骤)的类似于全局修正模块，也是两个修正模块堆叠得到的。但是局部修正模块并不是将一整原图作为输入，而是将大图裁剪为小图分别输入(从大图中提取包含有骨赘的部分)，经过resnet网络处理后得到原图大小的八分之一(即图3中的OS8)，原图大小的四分之一(即图3中的OS4)和原图(即图3中的OS1)，得到最终优化的结果mask(即OS1)，最后的分割结果为股骨骨赘mask和胫骨骨赘mask。
作为本实施例一种可选的实现方式，所述残差网络从输入中提取预设步幅特征图像，并将提取的特征图像传入空间金字塔池化，其中，损失函数基于步幅的大小不同，损失函数的类型不同。
Loss函数(损失函数)的选用为：对于stride＝8的输出使用交叉熵损失函数，对于stride＝1 的输出，使用L1+L2损失函数，对于stride＝4的输出：使用交叉熵+mean(L1+L2)损失函数。不同的损失函数适用于不同的步幅，因为粗略的refinement集中在全局结构上，而忽略了局部细节，而精确的refinement通过依赖局部线索来实现像素级精度。为了进一步提高分割边界的精度，在stride＝1的输出上采用了分割梯度。分割梯度由3*3的平均滤波器+sobel算子进行估计。
本发明实施例1提供的骨赘识别方法，通过获取医学图像，将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域，将所述股骨区域、胫骨区域输入到经过训练的第二分割模型中，得到股骨骨赘、胫骨骨赘，也就是说，利用第一分割模型和第二分割模型可以智能识别骨赘，为骨科医生节约大量的时间，同时也为资历尚浅的医生指导帮助。而且本发明实施例1的骨赘识别方法，只需完成模型训练便可以重复利用，迅速准确识别下肢骨影像中的骨赘，帮助医生进行手术规划，易于操作、准确度高、满足患者个体差异的优点，同时通过完善术前规划的依据资料，指导手术规划和假体选择，提高后续手术的准确率，对增加患者术后假体的使用寿命，降低术后并发症，提高患者术后生活质量有重要作用。
实施例2
与本发明实施例1相对应，本发明实施例2提供了一种骨赘识别装置。图4为本发明实施例2骨赘识别装置的结构示意图。如图4所示，本发明实施例2的骨赘识别装置包括获取模块20、第一处理模块22和第二处理模块24。
获取模块20，被配置为用于获取医学图像。第一处理模块22，被配置为于将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一；第二处理模块24，用于将所述股骨区域、胫骨区域、腓骨区域、或髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘。
上述骨赘识别装置具体细节可以对应参阅图1至图3所示的实施例中对应的相关描述和效果进行理解，此处不再赘述。
实施例3
本发明实施例还提供了一种电子设备，该电子设备可以包括处理器和存储器，其中处理器和存储器可以通过总线或者其他方式连接。
处理器可以为中央处理器(Central Processing Unit，CPU)。处理器还可以为其他通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific  Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其他可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等芯片，或者上述各类芯片的组合。
存储器作为一种非暂态计算机可读存储介质，可用于存储非暂态软件程序、非暂态计算机可执行程序以及模块，如本发明实施例中的车载显示装置按键屏蔽方法对应的程序指令/模块(例如，图4所示的获取模块20、第一处理模块22和第二处理模块24)。处理器通过运行存储在存储器中的非暂态软件程序、指令以及模块，从而执行处理器的各种功能应用以及数据处理，即实现上述方法实施例中的骨赘识别方法。
存储器可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储处理器所创建的数据等。此外，存储器可以包括高速随机存取存储器，还可以包括非暂态存储器，例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施例中，存储器可选包括相对于处理器远程设置的存储器，这些远程存储器可以通过网络连接至处理器。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
所述一个或者多个模块存储在所述存储器中，当被所述处理器执行时，执行如图1至图3所示实施例中的骨赘识别方法。
上述电子设备具体细节可以对应参阅图1至图3所示的实施例中对应的相关描述和效果进行理解，此处不再赘述。
本领域技术人员可以理解，实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，所述的程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，所述存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)、随机存储记忆体(Random Access Memory，RAM)、快闪存储器(Flash Memory)、硬盘(Hard Disk Drive，缩写：HDD)或固态硬盘(Solid-State Drive，SSD)等；所述存储介质还可以包括上述种类的存储器的组合。
虽然结合附图描述了本发明的实施例，但是本领域技术人员可以在不脱离本发明的精神和范围的情况下作出各种修改和变型，这样的修改和变型均落入由所附权利要求所限定的范围之内。

Claims (15)

  1. 一种骨赘识别方法，包括：
    获取医学图像；
    将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一；
    将所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘。
  2. 根据权利要求1所述的方法，其中，将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一，包括：
    将所述医学图像输入到神经网络模型中，得到所述股骨区域、胫骨区域、腓骨区域以及髌骨区域的粗分割预测结果；
    对所述粗分割预测结果进行优化，得到优化后的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一。
  3. 根据权利要求1所述的方法，其中，经过训练的第二分割模型识别骨赘的方法，包括：
    通过主干网络对所述股骨区域、胫骨区域、腓骨区域或髌骨区域中的任一个区域原图进行骨赘特征图粗分割，得到粗分割特征图；
    基于所述区域原图、所述粗分割特征图，利用所述第二分割模型包括的全局修正单元，对所述粗分割特征图进行全局修正，得到全局修正图；
    对所述区域原图、以及所述全局修正图所包含的骨赘特征进行提取，得到骨赘特征图；
    利用所述第二分割模型包括的局部修正单元，对所述骨赘特征图进行修正，得到骨赘分割结果。
  4. 根据权利要求2所述的方法，其中，对所述粗分割预测结果进行优化，包括：
    在所述粗分割预测结果中选取出置信度符合预设要求的特征点；
    对所述特征点的特征进行分类，并根据分类结果更新所述特征点的粗分割预测结果。
  5. 根据权利要求4所述的方法，其中，对所述特征点的特征进行分类，包括：
    对所述特征点进行特征提取，得到目标特征；
    基于分类器对所述目标特征进行预测，得到所述特征点是否属于股骨区域、胫骨区域、腓骨区域、或髌骨区域。
  6. 根据权利要求3所述的方法，其中，所述全局修正单元包括3层结构相同的全局修正模块，其中，每个全局修正模块包括顺次连接的特征融合模块和残差网络模块；第一层全局修正模块输出的第一预设大小的特征图作为第二层中特征融合用图；第二层全局修正模块输出的第二预设大小的特征图作为第三层中特征融合用图；第三层全局修正模块的输出作为所述全局修正图。
  7. 根据权利要求3所述的方法，其中，局部修正单元包括2层结构相同的局部修正模块，其中，每个局部修正模块包括顺次连接的特征融合模块和残差网络模块；第一层局部修正模块输出的第三预设大小的特征图作为第二层中特征融合用图；第二层局部修正模块的输出作为骨赘分割结果。
  8. 根据权利要求6或7所述的方法，其中，所述残差网络从输入中提取预设步幅特征图像，并将提取的特征图像传入空间金字塔池化，其中，损失函数基于步幅的大小不同，损失函数的类型不同。
  9. 根据权利要求1所述的方法，其中：所述第一分割模型为包含Pointrend算法的unet神经网络模型。
  10. 根据权利要求1所述的方法，其中，所述第二分割模型为cascadePSP神经网络模型。
  11. 根据权利要求10所述的方法，其中，所述cascadePSP神经网络模型中的主干网络为unet网络。
  12. 根据权利要求1所述的方法，其中，在将所述医学图像输入到经过训练的第一分割模型之前，还包括：
    获取第一图像数据集，所述第一图像数据集中至少包含一个含有骨赘的医学图像，所述医学图像中至少带有股骨区域、胫骨区域、腓骨区域、或髌骨区域的标签；
    利用所述第一图像数据集对所述第一分割模型进行训练，得到所述经过训练的第一分割模型；
    在将所述股骨区域、胫骨区域、腓骨区域、或髌骨区域中的至少之一输入到经过训练的第二分割模型中之前，还包括：
    获取第二图像数据集，所述第二图像数据集中包含多个股骨图像、胫骨图像、腓骨图像、以及髌骨图像，所述股骨图像、胫骨图像、腓骨图像、以及髌骨图像中至少带有对应的股骨骨赘、胫骨骨赘、腓骨骨赘、或髌骨骨赘的标签；
    利用所述第二图像数据集对所述第二分割模型进行训练，得到所述经过训练的第二分割模型。
  13. 一种骨赘识别装置，其中，包括：
    获取模块，被配置为获取医学图像；
    第一处理模块，被配置为将所述医学图像输入到经过训练的第一分割模型中，得到所述医学图像中的股骨区域、胫骨区域、腓骨区域、以及髌骨区域中的至少之一；
    第二处理模块，被配置为将所述股骨区域、胫骨区域、腓骨区域、或髌骨区域中的至少之一输入到经过训练的第二分割模型中，得到对应的股骨骨赘、胫骨骨赘、腓骨骨赘或髌骨骨赘。
  14. 一种电子设备，其中，包括：
    存储器和处理器，所述存储器和所述处理器之间互相通信连接，所述存储器中存储有计算机指令，所述处理器通过执行所述计算机指令，从而执行权利要求1至12中任一项所述的骨赘识别方法。
  15. 一种计算机可读存储介质，其中，所述计算机可读存储介质存储有计算机指令，所述计算机指令用于使所述计算机执行权利要求1至12中任一项所述的骨赘识别方法。
PCT/CN2021/130471 2021-03-29 2021-11-12 骨赘识别方法、装置、电子设备及存储介质 WO2022205928A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110335659.2A CN113076987B (zh) 2021-03-29 2021-03-29 骨赘识别方法、装置、电子设备及存储介质
CN202110335659.2 2021-03-29

Publications (1)

Publication Number Publication Date
WO2022205928A1 true WO2022205928A1 (zh) 2022-10-06

Family

ID=76611552

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/130471 WO2022205928A1 (zh) 2021-03-29 2021-11-12 骨赘识别方法、装置、电子设备及存储介质

Country Status (2)

Country Link
CN (1) CN113076987B (zh)
WO (1) WO2022205928A1 (zh)

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114170128B (zh) * 2020-08-21 2023-05-30 张逸凌 基于深度学习的骨骼分割方法和系统
CN113076987B (zh) * 2021-03-29 2022-05-20 北京长木谷医疗科技有限公司 骨赘识别方法、装置、电子设备及存储介质
CN113689402B (zh) * 2021-08-24 2022-04-12 北京长木谷医疗科技有限公司 基于深度学习的股骨髓腔形态识别方法、装置及存储介质
CN113870261B (zh) * 2021-12-01 2022-05-13 杭州柳叶刀机器人有限公司 用神经网络识别力线的方法与系统、存储介质及电子设备
CN114511548A (zh) * 2022-02-16 2022-05-17 云南白药集团医药电子商务有限公司 基于深度学习的多阶段多层次骨赘检测方法
CN116543221B (zh) * 2023-05-12 2024-03-19 北京长木谷医疗科技股份有限公司 关节病变智能检测方法、装置、设备及可读存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110689551A (zh) * 2019-10-14 2020-01-14 慧影医疗科技（北京）有限公司 四肢骨分割方法、装置、电子设备及可读存储介质
CN111340760A (zh) * 2020-02-17 2020-06-26 中国人民解放军国防科技大学 基于多任务两级卷积神经网络的膝关节定位方法
US20210004956A1 (en) * 2018-03-12 2021-01-07 Persimio Ltd. Automated bone segmentation in images
CN113076987A (zh) * 2021-03-29 2021-07-06 北京长木谷医疗科技有限公司 骨赘识别方法、装置、电子设备及存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110097967A (zh) * 2018-09-13 2019-08-06 龙岩学院 基于固有时间尺度分解、相空间重构和神经网络的膝骨性关节炎诊断系统
CN110969622B (zh) * 2020-02-28 2020-07-24 南京安科医疗科技有限公司 用于辅助肺炎诊断的图像处理方法和系统
CN111640093A (zh) * 2020-05-20 2020-09-08 上海联影智能医疗科技有限公司 医学图像的质控方法和计算机可读存储介质
CN111539956B (zh) * 2020-07-07 2020-12-29 南京安科医疗科技有限公司 基于脑部辅助图像的脑出血自动检测方法及电子介质

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210004956A1 (en) * 2018-03-12 2021-01-07 Persimio Ltd. Automated bone segmentation in images
CN110689551A (zh) * 2019-10-14 2020-01-14 慧影医疗科技（北京）有限公司 四肢骨分割方法、装置、电子设备及可读存储介质
CN111340760A (zh) * 2020-02-17 2020-06-26 中国人民解放军国防科技大学 基于多任务两级卷积神经网络的膝关节定位方法
CN113076987A (zh) * 2021-03-29 2021-07-06 北京长木谷医疗科技有限公司 骨赘识别方法、装置、电子设备及存储介质

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
CHENG HO KEI; CHUNG JIHOON; TAI YU-WING; TANG CHI-KEUNG: "CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 13 June 2020 (2020-06-13), pages 8887 - 8896, XP033805360, DOI: 10.1109/CVPR42600.2020.00891 *
KIRILLOV ALEXANDER; WU YUXIN; HE KAIMING; GIRSHICK ROSS: "PointRend: Image Segmentation As Rendering", 2020 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR), IEEE, 13 June 2020 (2020-06-13), pages 9796 - 9805, XP033804684, DOI: 10.1109/CVPR42600.2020.00982 *

Also Published As

Publication number Publication date
CN113076987B (zh) 2022-05-20
CN113076987A (zh) 2021-07-06

Similar Documents

Publication Publication Date Title
WO2022205928A1 (zh) 骨赘识别方法、装置、电子设备及存储介质
TWI755717B (zh) 圖像處理方法及裝置、電子設備和計算機可讀儲存介質
WO2023024882A1 (zh) 基于深度学习的股骨髓腔形态识别方法、装置及存储介质
Tang et al. CT image enhancement using stacked generative adversarial networks and transfer learning for lesion segmentation improvement
JP7083037B2 (ja) 学習装置及び学習方法
JP2023530023A (ja) 深層学習に基づく脊椎ｍｒｉ映像キーポイント検出方法
WO2023024883A1 (zh) 基于深度强化学习的股骨柄放置方法、装置及电子设备
Giordano et al. Epiphysis and metaphysis extraction and classification by adaptive thresholding and DoG filtering for automated skeletal bone age analysis
CN110751179B (zh) 超声设备
WO2023045231A1 (zh) 一种解耦分治的面神经分割方法和装置
CN113298786B (zh) 图像识别及模型训练的方法、真臼位置识别的方法及装置
WO2020022027A1 (ja) 学習装置及び学習方法
KR102228087B1 (ko) 의료영상에서의 특정연골 분할 방법 및 장치
US20230169666A1 (en) Methods and systems for image registration
WO2023151199A1 (zh) 基于交叉注意力机制的骨折影像精细识别网络构建方法
CN112085696B (zh) 医学图像分割网络模型的训练方法、分割方法及相关设备
CN110210544A (zh) 图像分类方法、计算机设备和存储介质
Goutham et al. Automatic localization of landmarks in cephalometric images via modified U-Net
WO2020106393A2 (en) Skeletal maturity determination using radiographs of portions of a hand
US20230207106A1 (en) Image segmentation for sets of objects
CN114897703A (zh) 一种骨科全长x线片自动拼接系统、方法、计算机设备及存储介质
KR101690268B1 (ko) 조직 씨앗을 이용한 의료영상 반자동 영역화 장치 및 방법
CN113920128A (zh) 膝关节股骨胫骨分割方法和装置
CN113129278A (zh) 基于非局部分离注意力机制的X-Ray图片股骨干骨折检测方法
Sun et al. Semi-supervised breast cancer pathology image segmentation based on fine-grained classification guidance

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21934564

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21934564

Country of ref document: EP

Kind code of ref document: A1